JP7744523B2 - Display Control System - Google Patents

Description

One aspect of the present invention relates to a display control system that controls the display of virtual objects.

Technologies for presenting virtual objects to users, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) (hereinafter referred to as "XR technologies"), are known. For example, Patent Literature 1 discloses a mechanism for providing augmented reality or mixed reality to users by overlaying virtual objects on a real-space image displayed on a display device.

JP 2014-203382 A

In the above-mentioned XR technology, it is necessary to display virtual objects as quickly as possible in order to improve the comfort of the user's virtual experience. For example, one system that applies the above-mentioned XR technology is a system that downloads and displays virtual objects, which are 3D objects created by scanning objects existing in real space, to a user's device. However, such virtual objects based on real objects generally have large file sizes, which can take a long time to download and can result in a long time until the virtual object is displayed on the user's device's display. On the other hand, if the file size of a virtual object is reduced by methods such as file reduction to shorten the download time, another problem can arise: the virtual object cannot be delivered to the user at its original quality (resolution), thereby degrading the quality of the user's virtual experience.

Therefore, one aspect of the present invention aims to provide a display control system that can shorten the time it takes for a virtual object to be displayed on a user terminal while maintaining the quality of the user's virtual experience.

A display control system according to one aspect of the present invention controls the display of virtual objects displayed on a display unit of a user terminal. The display control system includes a file storage unit that stores, for a first virtual object, a first file having a first file size and a second file having a second file size larger than the first file size, and a display control unit that controls the display of the first virtual object on the user terminal. The display control unit performs the following operations: a first process of transmitting the first file to the user terminal in priority to the second file, thereby displaying the first virtual object based on the first file on the display unit; a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file; and a third process of switching the first virtual object based on the first file displayed on the display unit to a first virtual object based on the second file when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal has been completed.

In a display control system according to one aspect of the present disclosure, a first file having a small file size (first file size) is first preferentially transmitted to a user terminal. This configuration shortens the time it takes for a first virtual object to be displayed on the display unit of the user terminal. Furthermore, after the first file is transmitted, or in parallel with the transmission of the first file, a second file is transmitted to the user terminal. Then, after the user terminal receives the second file, the first virtual object based on the first file displayed on the display unit is replaced with a first virtual object based on the second file. This configuration allows the user to later be presented with a first virtual object based on the second file, which has a larger file size and higher quality than the first file, thereby maintaining the quality of the user's virtual experience.

According to one aspect of the present invention, a display control system can be provided that can shorten the time it takes for a virtual object to be displayed on a user terminal while maintaining the quality of the user's virtual experience.

FIG. 1 is a diagram illustrating an example of a functional configuration of a system including a server and a user terminal, which is a display control system according to an embodiment. FIG. 10 illustrates an example of a file generation process. FIG. 10 is a diagram illustrating an example of display control of a virtual object. FIG. 10 is a sequence diagram illustrating an example of the operation of the server and the user terminal. FIG. 2 illustrates an example of the hardware configuration of a server and a user terminal.

One embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Note that in the description of the drawings, identical or equivalent elements will be given the same reference numerals, and duplicate explanations will be omitted.

FIG. 1 illustrates an example of a system including a server 10, which is a display control system according to an embodiment, and a user terminal 20. The server 10 is a device that controls the display of virtual objects displayed on the display unit 24 of the user terminal 20. For example, the server 10 may provide the user with an AR or MR experience by displaying virtual objects together with a real-world image on the display unit 24 of the user terminal 20. Alternatively, the server 10 may provide the user with a VR experience by displaying virtual objects together with a virtual background (virtual-world image) on the display unit 24 of the user terminal 20. The user terminal 20 is a computer terminal used by the user. The type of user terminal 20 is not limited. For example, the user terminal 20 may be a personal computer such as a desktop PC or laptop PC, a terminal device such as a smartphone or tablet held by the user, or a wearable device attached to a part of the user's body, such as the head. Examples of wearable devices include head-mounted displays such as goggles, glasses (eyeglasses), and hats.

The server 10 and the user terminal 20 are connected to each other via a wired or wireless communication network so that they can communicate data with each other. The server 10 and the user terminal 20 may also be connected via one or more other devices. For example, if the user terminal 20 is a head-mounted display, each of the server 10 and the user terminal 20 may be configured to be able to communicate with another device, such as the user's smartphone. In this case, the server 10 is configured to be able to communicate with the user terminal 20 via the other device.

As shown in FIG. 1, the server 10 (display control system) has a receiving unit 11 (terminal information acquisition unit), a file generation unit 12, a file storage unit 13, and a display control unit 14. The user terminal 20 also has an imaging unit 21, a transmitting unit 22, a receiving unit 23, and a display unit 24.

First, the functional components of the user terminal 20 will be described. The imaging unit 21 has the function of capturing an image of the real space around (e.g., in front of) the user terminal 20 and acquiring a real space image (image data). The imaging unit 21 is configured, for example, by a camera. The real space image may be a still image or a moving image (video). When an AR experience or MR experience is provided to the user, the real space image may be displayed on the display unit 24. Furthermore, in order to generate a virtual object, which is a 3D object, by the file generation unit 12 described below, the image data may be composed of multiple image data obtained by capturing images of an object (real object) existing in real space from multiple different directions.

The transmission unit 22 transmits the imaging data acquired by the imaging unit 21 to the server 10. In addition, in this embodiment, the transmission unit 22 transmits terminal information regarding the attributes of the user terminal 20 to the server 10. Examples of the attributes of the user terminal 20 include application type, communication speed, processing performance, etc.

"Application type" is information indicating the type of application used to display the virtual object transmitted from the server 10. The application type can be identified, for example, by a combination of the type of user terminal 20 and the type of application running on the user terminal 20. Examples of application types include dedicated applications running on a PC, web browsers running on a PC, web browsers running on a smartphone, and applications built into head-mounted displays such as smart glasses. The transmission unit 22 transmits terminal information indicating the application type to the server 10, for example, when starting execution of a service that displays a virtual object transmitted from the server 10 (for example, during a login operation).

The "communication speed" is the downstream speed (download speed) when downloading data from the server 10 to the user terminal 20. For example, if the user terminal 20 is a terminal that moves with the user (e.g., a head-mounted display, smartphone, etc.) and the server 10 and the user terminal 20 communicate wirelessly, the communication speed may change because the strength of the radio waves varies depending on the location of the user terminal 20. The transmission unit 22, for example, periodically transmits terminal information indicating the communication speed to the server 10.

"Processing performance" is information indicating the specifications of the processor, memory, etc. equipped in the user terminal 20, processor usage rate, memory usage rate, etc. For example, terminal information indicating processing performance at a certain point in time is information indicating the amount of computing resources (processor, memory, etc.) available on the user terminal 20 at that point in time. The transmission unit 22, for example, periodically transmits terminal information indicating processing performance to the server 10.

The receiving unit 23 receives information (the first file, second file, etc. described below) from the server 10 for displaying a virtual object on the display unit 24.

The display unit 24 displays a virtual object in a display area 24a provided on the display unit 24 based on instructions from the display control unit 14 of the server 10. Specifically, the display unit 24 displays a virtual object in the display area 24a based on information for displaying the virtual object (first file, second file, etc.) received from the server 10. If the user terminal 20 is a PC, the display unit 24 may be configured as a display monitor. If the user terminal 20 is a mobile terminal such as a smartphone, the display unit 24 may be configured as a touch panel display. If the user terminal 20 is a head-mounted display, the display unit 24 may be configured as a display placed in front of both eyes of the user.

Next, we will explain each functional configuration of the server 10. The receiving unit 11 receives various data transmitted by the user terminal 20 (transmitting unit 22). In this embodiment, the receiving unit 11 receives imaging data acquired by the imaging unit 21. The receiving unit 11 also functions as a terminal information acquiring unit that acquires the above-mentioned terminal information (application type, communication speed, processing performance, etc.).

The file generation unit 12 acquires the imaging data received by the receiving unit 11. If the imaging data includes a real object that exists in real space, the file generation unit 12 generates a first file and a second file of a virtual object corresponding to the real object based on the imaging data. The file generation unit 12 generates a file representing a virtual object corresponding to the real object included as a subject in the imaging data, for example, by performing at least one of image recognition (2D image recognition), photogrammetry, and 3D scanning on the imaging data. Note that the method for generating a virtual object by the file generation unit 12 is not limited to the method based on imaging data of the actual object as described above. For example, the file generation unit 12 may generate a file representing a virtual object by modeling, etc. In other words, the virtual objects used in this embodiment are not limited to objects based on real objects, but may also be objects created by creators such as CG designers.

An example of the file generation process performed by the file generation unit 12 will be described with reference to Figure 2. Figure 2 shows an example in which a rabbit kept as a pet by a user is captured as a real object on the user terminal 20 (imaging unit 21). In this example, the file generation unit 12 generates three types of files 50A, 50B, and 50C as data representing a virtual object (first virtual object) corresponding to this real object. Files 50A, 50B, and 50C are all data used to display a virtual object on the display unit 24 of the user terminal 20, and may include drawing information for drawing the virtual object in the display area 24a. In this embodiment, the virtual object is represented as a 3D object.

File 50A is data that has not been compressed (reduced) from the image data and retains the original quality (resolution). In other words, file 50A is an original file generated based on image data of a real object. File 50B is data obtained by compressing file 50A by 30%. In other words, the file size (file capacity) of file 50B is 70% of the file size of file 50A. File 50C is data obtained by compressing file 50A by 60%. In other words, the file size of file 50C is 40% of the file size of file 50A.

The quality of files 50B and 50C is lower than that of file 50A due to the data compression. As an example, as shown in FIG. 2, the surface shape of the virtual object (rabbit) represented by file 50B is coarser than the surface shape of the virtual object represented by the original file 50A. Furthermore, the surface shape of the virtual object represented by file 50C is even coarser than the surface shape of the virtual object represented by file 50B. In other words, files 50B and 50C have lower quality (resolution) than the original file 50A by the amount that their file sizes are smaller than the original file size.

In this embodiment, as an example, the file generation unit 12 generates two types of compressed files (files 50B and 50C), but the file generation unit 12 may generate only one type of compressed file (one specific compression ratio), or may generate three or more types of compressed files (three or more different compression ratios). For example, the file generation unit 12 may determine the compression ratio of the compressed file to be generated depending on the file size of the original file 50A. For example, if the file size of the original file 50A is not very large, the file size can be sufficiently small without increasing the compression ratio very much. In this case, the file generation unit 12 may generate only file 50B.

The file memory unit 13 acquires and stores, for a first virtual object, a first file having a first file size and a second file having a second file size larger than the first file size.

In this embodiment, as an example, the first virtual object is a virtual object corresponding to the real object (rabbit) shown in FIG. 2. That is, in this embodiment, the first virtual object is a virtual object generated based on image data of the real object. In the example of FIG. 2, file 50C corresponds to a first file having a first file size. Furthermore, each of files 50A and 50B corresponds to a second file having a second file size larger than the first file size. That is, in the example of FIG. 2, the file storage unit 13 stores multiple (here, two) second files (files 50A and 50B) having different file sizes. Note that when the file generation unit 12 generates only files 50A and 50B, file 50B functions as the first file and file 50A functions as the second file.

In this embodiment, the file storage unit 13 stores files 50A, 50B (second files) and file 50C (first file) generated by the file generation unit 12 in association with the virtual objects corresponding to the real objects. That is, the file storage unit 13 stores (registers) the first file and second file corresponding to each virtual object.

The file storage unit 13 manages multiple virtual objects (i.e., multiple virtual objects each corresponding to a different real object) using, for example, unique identification information (object IDs). For example, when virtual objects are operated on the web (i.e., when displayed by a web browser on the user terminal 20, etc.), the file storage unit 13 may assign an identifier such as a unique URI (Uniform Resource Identifier) to the file representing each virtual object.

Furthermore, the first file and second file (in this embodiment, three files 50A, 50B, and 50C) for the same virtual object can be managed as follows.

In the first example, files 50A, 50B, and 50C are managed by file name. For example, the file storage unit 13 manages (stores) file 50A (the original file) as "usagi.3d," file 50B (a file compressed by 30%) as "usagi_a.3d," and file 50C (a file compressed by 60%) as "usagi_b.3d." In this example, "usagi" is character string information that uniquely identifies a virtual object (here, as an example, a virtual object corresponding to the real object (rabbit) shown in Figure 2). ".3d" is an example of an identifier assigned to each file (data). The "_a" and "_b" appended after "usagi" are information corresponding to the compression rate relative to the original file. According to the first example, it is possible to identify the virtual object and the compression rate from the file name.

In the second example, files 50A, 50B, and 50C are managed by embedding information about the compression rate (the compression rate of the original file is 0) in the header of each file. In this case, the file storage unit 13 may assign a common file name (e.g., "usagi.3d") to files 50A, 50B, and 50C by storing files 50A, 50B, and 50C in different locations.

In a third example, multiple files 50A, 50B, and 50C corresponding to the same virtual object are managed by a management table. For example, multiple files 50A, 50B, and 50C corresponding to the same virtual object are associated with each other in the management table by a common key that indicates the virtual object.

When it becomes necessary to send data (files) of a certain virtual object to the user terminal 20 (for example, when a request to display a virtual object is received from the user terminal 20), any of the methods in the first to third examples above makes it possible to access the first file and second file corresponding to the virtual object from among multiple files stored in the file storage unit 13. Note that the file management method is not limited to the first to third examples above, and other methods may also be used.

The display control unit 14 controls the display of a first virtual object on the user terminal 20. For example, when it becomes necessary to display a first virtual object on the display unit 24 of the user terminal 20, the display control unit 14 performs the above-mentioned display control by controlling the process of providing files (first file, second file) corresponding to the first virtual object to the user terminal 20. Specifically, the display control unit 14 performs the first process, second process, and third process described below. An example of the process of the display control unit 14 will be described below with reference to Figure 3.

The first process is a process of displaying a first virtual object based on the first file on the display unit 24 by transmitting the first file (file 50C in the example of FIG. 2) to the user terminal 20 with priority over the second file (files 50A and 50B in the example of FIG. 2). That is, the display control unit 14 provides the user terminal 20 with file 50C, which has a reduced file size compared to the original file, with priority. According to the above configuration, as shown in the upper part of FIG. 3, the time from when a download request is made from the user terminal 20 to the server 10 to when the first virtual object based on file 50C is displayed on the display unit 24 (display area 24a) of the user terminal 20 can be shortened compared to when the original file (file 50A) is transmitted to the user terminal 20.

The second process is a process of transmitting a second file (file 50A or file 50B) to the user terminal 20 after transmitting a first file (file 50C) to the user terminal 20, or in parallel with transmitting the first file to the user terminal 20. In the second process, the display control unit 14 may execute either a first control pattern in which transmission of the second file to the user terminal 20 is started after transmitting the first file, or a second control pattern in which transmission of the second file to the user terminal 20 is performed in parallel with transmitting the first file to the user terminal 20.

Because the file size of the second file is larger than the file size of the first file, even in the second control pattern, the process of sending the first file completes before the process of sending the second file. However, because the process of sending the second file is executed in parallel with the process of sending the first file in the second control pattern, the time at which the process of sending the second file completes in the second control pattern is earlier than the time at which the process of sending the second file completes in the first control pattern. In other words, the second control pattern is configured to start the process of sending the second file before the process of sending the first file completes, thereby enabling the process of sending the second file to complete earlier than the first control pattern, which starts the process of sending the second file after the process of sending the first file completes. On the other hand, the time at which the process of sending the first file completes in the second control pattern is later than the time at which the process of sending the first file completes in the first control pattern. In other words, while the second control pattern allows the second file to be sent more quickly than the first control pattern, the time required to send the first file is longer than with the first control pattern because the second file is also sent while the first file is being sent (i.e., the communication resources are shared between the first file and the second file).

Therefore, from the perspective of minimizing the time until the first virtual object is displayed on the display unit 24 of the user terminal 20, the first control pattern, which downloads the first file and the second file sequentially, is more advantageous than the second control pattern. On the other hand, from the perspective of shortening the waiting time until the first virtual object is displayed (the waiting time experienced by the user) and providing the user with high-quality data (the second file) as quickly as possible, the second control pattern, which downloads the first file and the second file in parallel, is more advantageous than the first control pattern.

Therefore, the display control unit 14 may be configured to be able to switch between the first control pattern and the second control pattern. For example, the display control unit 14 may determine whether to execute the first control pattern or the second control pattern based on a user setting. That is, the display control unit 14 may allow the user to select whether to execute the first control pattern or the second control pattern. Alternatively, the display control unit 14 may determine whether to execute the first control pattern or the second control pattern based on the terminal information (e.g., communication speed) described above. For example, the display control unit 14 may determine to execute the second control pattern when it is expected that the download of the first file will be completed in a sufficiently short time even if the first file and the second file are downloaded in parallel (e.g., when the communication speed is equal to or greater than a predetermined threshold). Note that the download of the first file is expected to be completed in a sufficiently short time even when the user terminal 20 is equipped with a configuration for performing parallel processing at high speed (e.g., multithreading, etc.) or a configuration supporting divided downloads. Therefore, the display control unit 14 may determine whether to execute the first control pattern or the second control pattern depending on the communication method implemented in the user terminal 20. According to the above configuration, it is possible to appropriately switch between the first control pattern and the second control pattern described above depending on the use case.

In addition, in the second process, the display control unit 14 may determine the second file to be sent to the user terminal 20 from among multiple second files (in this embodiment, files 50A and 50B) based on the terminal information. For example, the display control unit 14 may determine the second file to be provided to the user terminal 20 according to the attributes of the user terminal 20 described above (application type, communication speed, processing performance, etc.).

For example, if an application running on the user terminal 20 cannot smoothly display file 50A with original quality (i.e., if file 50A has a quality that exceeds the range that can be processed by the application), or if the user terminal 20 does not have the processing performance (e.g., processor, memory, etc.) required to perform display processing using file 50A, then even if file 50A is sent to the user terminal 20, the user terminal 20 will not be able to perform display processing that makes the most of the quality of file 50A. Therefore, the display control unit 14 may determine, based on terminal information including information regarding at least one of the application type and processing performance of the user terminal 20, as the second file to be provided to the user terminal 20, a second file of quality that does not exceed the maximum quality at which display processing is possible on the user terminal 20.

For example, in the example of Figure 2, if file 50A has a quality higher than the maximum quality at which display processing can be performed on the user terminal 20, and file 50B has a quality lower than the maximum quality, the display control unit 14 may select file 50B as the second file to be sent to the user terminal 20.

Furthermore, even if the user terminal 20 has sufficient specifications (e.g., processor, memory, etc.) to perform display processing using file 50A, if the communication speed of the user terminal 20 is slow, it will take a very long time for the download of file 50A to be completed. Therefore, the display control unit 14 may determine, for example, based on terminal information including information about the communication speed of the user terminal 20, as the second file to be provided to the user terminal 20, a second file whose download is expected to be completed within a predetermined threshold time. For example, in the example of Figure 2, if the estimated download time for file 50A estimated from the communication speed of the user terminal 20 exceeds the threshold time and the estimated download time for file 50B is equal to or less than the threshold time, the display control unit 14 may select file 50B as the second file to be sent to the user terminal 20.

According to the above configuration, when multiple second files (files 50A and 50B in the example of Figure 2) having different file sizes (i.e., different qualities) are stored in the file storage unit 13, it is possible to provide (transmit) a second file with appropriate quality to the user terminal 20 based on the attributes of the user terminal 20. Note that in the example of Figure 3, file 50B is determined as the second file to be transmitted to the user terminal 20 by the above-described determination process.

The third process is a process of switching the first virtual object based on file 50C displayed on the display unit 24 to a first virtual object based on file 50B when a first virtual object based on a first file (file 50C) is displayed on the display unit 24 (display area 24a) and transmission of a second file (here, as an example, file 50B) to the user terminal 20 has been completed (see arrow P1 in Figure 3).

Switching from file 50C to file 50B can be performed, for example, by changing the identifier designated as the display target on the user terminal 20 (here, the identifier corresponding to file 50C (e.g., the URI, object ID, etc. described above)) to the identifier corresponding to file 50B. Furthermore, information such as the angle of view, position, and distance (depth) of the first virtual object based on file 50C that was displayed on the display unit 24 immediately before the switch may be configured to be carried over after the switch. In this case, it is possible to smoothly switch from file 50C to file 50B while maintaining the way the first virtual object appears to the user.

Furthermore, when a first virtual object based on the second file (file 50B) is displayed on the display unit 24 (display area 24a) (see the lower part of Figure 3), the display control unit 14 may execute display control to switch back the first virtual object based on file 50B displayed on the display unit 24 to a first virtual object based on the first file (file 50C) depending on the occurrence of a processing delay in the user terminal 20. For example, the display control unit 14 may determine whether a processing delay is occurring in the user terminal 20 based on information including information related to the processing performance of the user terminal 20 (e.g., the terminal information described above). For example, the display control unit 14 may determine that a processing delay is occurring when the processor usage rate of the user terminal 20 is equal to or greater than a threshold. Then, when the display control unit 14 determines that a processing delay is occurring, it may cause the user terminal 20 to execute a process of switching back from the second file (file 50B) to the first file (file 50C) (see arrow P2 in Figure 3). Because file 50C has a smaller file size than file 50B, the amount of calculation and memory usage required for the display process (drawing process) is smaller than those for file 50B. Therefore, according to the above display control, by executing the switchback process when a processing delay occurs, the processing delay can be eliminated.

Furthermore, when a first virtual object based on the second file (file 50B) is displayed on the display unit 24 (display area 24a) and multiple virtual objects including the first virtual object are displayed on the display unit 24, the display control unit 14 may execute display control to switch back the first virtual object based on file 50B displayed on the display unit 24 to the first virtual object based on the first file (file 50C) in accordance with the number of virtual objects displayed on the display unit 24. For example, the display control unit 14 may acquire information indicating the number of virtual objects displayed on the display unit 24. For example, the display control unit 14 may acquire such information from the user terminal 20, or may determine the number of virtual objects displayed on the display unit 24 based on a history of transmission of virtual object files to the user terminal 20. Then, for example, when the display control unit 14 determines that the number of displayed virtual objects is equal to or greater than a predetermined threshold, the display control unit 14 may cause the user terminal 20 to execute a process of switching back from the second file (file 50B) to the first file (file 50C) (see arrow P2 in FIG. 3). According to the above display control, by performing a revert process in a situation where a processing delay is likely to have occurred (or is likely to occur in the future) due to a large number of displayed virtual objects, it is possible to eliminate or avoid the processing delay.

Next, with reference to Figure 4, an example of the operation of the server 10 and the user terminal 20 will be described. In step S1, the imaging unit 21 captures an image of a real object. In step S2, the transmission unit 22 transmits the imaging data acquired in step S1 to the server 10. The transmission unit 22 also transmits terminal information (information including, as an example in this embodiment, application type, communication speed, processing performance, etc.) to the server 10. The imaging data and terminal information are received by the receiving unit 11 of the server 10.

In step S3, the file generation unit 12 generates file 50A (second file), file 50B (second file), and file 50C (first file) based on the imaging data (see Figure 2).

In step S4, the file memory unit 13 stores the files 50A, 50B, and 50C generated by the file generation unit 12.

Next, for example, the server 10 receives a request to display the first virtual object from the user terminal 20, which triggers the execution of steps S5 and S6 (first processing). That is, in step S5, the display control unit 14 transmits the first file (file 50C in the example of FIG. 3) to the user terminal 20 with priority over the second file (file 50B in the example of FIG. 3). As a result, in step S6, the display unit 24 displays the first virtual object based on the first file (file 50C) (see the upper part of FIG. 3).

Furthermore, in step S7, the display control unit 14 executes a second process. That is, after transmitting the first file (file 50C), or in parallel with transmitting the first file, the display control unit 14 transmits a second file (file 50B) to the user terminal 20. Because the file size of the second file (second file size) is larger than the file size of the first file (first file size), the second file is transmitted to the user terminal 20 after step S6. That is, the time at which reception of the second file at the user terminal 20 is completed is later than step S6.

Next, in step S8, the display control unit 14 executes a third process. That is, when a first virtual object based on a first file (file 50C) is displayed on the display unit 24 (display area 24a) and transmission of a second file (file 50B) to the user terminal 20 is complete, the display control unit 14 switches the first virtual object based on file 50C displayed on the display unit 24 to a first virtual object based on file 50B (see arrow P1 in FIG. 3). For example, the display control unit 14 can cause the user terminal 20 to execute the switching process described above by transmitting the second file together with instruction information to the user terminal 20 instructing the user terminal 20 to "switch from the first file to the second file after receiving the second file."

Next, in step S9, the display control unit 14 determines whether to perform display control to switch back the first virtual object based on file 50B displayed on the display unit 24 to the first virtual object based on the first file (file 50C). This determination process can be performed periodically or irregularly based on terminal information received from the user terminal 20 (e.g., information related to processing performance), information indicating the number of virtual objects displayed on the display unit 24, etc.

For example, as described above, if the display control unit 14 determines that a processing delay is occurring in the user terminal 20, or if it determines that the number of virtual objects displayed on the display unit 24 is equal to or greater than a threshold, it instructs the user terminal 20 to perform a revert process (step S10). As a result, in step S11, a process of reverting from the second file (file 50B) to the first file (file 50C) is performed (see arrow P2 in FIG. 3). Note that after the revert process is performed (i.e., after step S11), the display control unit 14 may perform the process of step S8 again if it determines that the reason for the revert (e.g., the aforementioned processing delay, the number of displayed virtual objects being equal to or greater than a threshold) has been resolved.

In the server 10 described above, the first file (file 50C) having a small file size (first file size) is first transmitted to the user terminal 20 with priority. This configuration shortens the waiting time until the first virtual object is displayed on the display unit 24 of the user terminal 20. After the first file is transmitted, or in parallel with the transmission of the first file, the second file (file 50B) is transmitted to the user terminal 20. After the user terminal 20 receives the second file, the first virtual object based on the first file displayed on the display unit 24 is replaced with the first virtual object based on the second file (see arrow P1 in Figure 3). This configuration allows the user to later be presented with a first virtual object based on the second file, which has a larger file size and higher quality than the first file, thereby maintaining the quality of the user's virtual experience.

In addition, in this embodiment, the file generation unit 12 generates a first file and a second file of a virtual object corresponding to a real object based on the imaging data, and the file storage unit 13 stores the first file and the second file generated by the file generation unit 12 in association with the virtual object. According to the above configuration, when a virtual object corresponding to a real object is generated, the first file and the second file corresponding to the virtual object can be automatically generated and stored.

In the above embodiment, the display control system is configured by the server 10, but the display control system may be configured by the server 10 alone, or by the server 10 and the user terminal 20. That is, some of the functions of the server 10 described above may be executed by the user terminal 20. For example, in the above embodiment, the display control unit 14 of the server 10 controlled the display content of the display unit 24 of the user terminal 20, but the display control unit 14 may be provided in the user terminal 20 rather than the server 10. Also, some of the functions of the server 10 described above may be omitted. For example, in the above embodiment, the server 10 (file generation unit 12) generated a file of a virtual object, but the file of a virtual object may be generated by a device other than the server 10.

The display control system disclosed herein (server 10 in the above embodiment) has the following configuration.

[1] A display control system that controls the display of a virtual object displayed on a display unit of a user terminal,
a file storage unit that stores, for a first virtual object that is one of the virtual objects, a first file having a first file size and a second file having a second file size larger than the first file size;
a display control unit that controls display of the first virtual object on the user terminal,
The display control unit
a first process of transmitting the first file to the user terminal with priority over the second file, thereby displaying the first virtual object based on the first file on the display unit;
a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file;
and executing a third process of switching the first virtual object based on the first file displayed on the display unit to the first virtual object based on the second file when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal has been completed.
Display control system.

[2] A file generation unit acquires imaging data obtained by imaging a real object that is an object existing in real space, and generates the first file and the second file of the virtual object corresponding to the real object based on the imaging data,
the file storage unit stores the first file and the second file generated by the file generation unit in association with the virtual object corresponding to the real object;
The display control system according to [1].

[3] When the first virtual object based on the second file is displayed on the display unit, the display control unit switches the first virtual object based on the second file displayed on the display unit back to the first virtual object based on the first file according to an occurrence status of a processing delay of the user terminal.
The display control system according to [1] or [2].

[4] When the first virtual object based on the second file is displayed on the display unit and a plurality of virtual objects including the first virtual object are displayed on the display unit, the display control unit switches back the first virtual object based on the second file displayed on the display unit to the first virtual object based on the first file, depending on the number of the virtual objects displayed on the display unit.
A display control system according to any one of [1] to [3].

[5] The display control unit is configured to be able to switch between a first control pattern in which transmission of the second file to the user terminal is started after transmission of the first file, and a second control pattern in which transmission of the second file to the user terminal is performed in parallel with transmission of the first file, in the second process.
A display control system according to any one of [1] to [4].

[6] Further comprising a terminal information acquisition unit that acquires terminal information indicating attributes of the user terminal,
the file storage unit stores a plurality of the second files having different file sizes;
the display control unit determines, based on the terminal information, the second file to be transmitted to the user terminal from among the plurality of second files;
A display control system according to any one of [1] to [5].

Furthermore, the block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and/or software. Furthermore, there are no particular limitations on the method of realizing each functional block. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, using wires, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or multiple devices.

Functions include, but are not limited to, judging, determining, calculating, computing, processing, deriving, investigating, searching, verifying, receiving, transmitting, outputting, accessing, resolving, selecting, choosing, establishing, comparing, assuming, expecting, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning.

For example, the server 10 in one embodiment of the present disclosure may function as a computer that performs the display control method of the present disclosure. Figure 5 is a diagram showing an example of the hardware configuration of the server 10 in one embodiment of the present disclosure. The server 10 may be physically configured as a computer device including a processor 1001, memory 1002, storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc. The user terminal 20 may also have a hardware configuration similar to that of the server 10.

In the following description, the term "device" can be interpreted as a circuit, device, unit, etc. The hardware configuration of server 10 may be configured to include one or more of the devices shown in Figure 5, or may be configured to exclude some of the devices.

Each function of the server 10 is realized by loading specified software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control unit, an arithmetic unit, registers, etc.

The processor 1001 also reads programs (program code), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes in accordance with these. The program used is a program that causes a computer to execute at least some of the operations described in the above-described embodiments. For example, each functional unit of the server 10 (e.g., the display control unit 14, etc.) may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be made for other functional blocks. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted from a network via a telecommunications line.

Memory 1002 is a computer-readable recording medium and may be composed of, for example, at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory (primary storage device), etc. Memory 1002 can store executable programs (program code), software modules, etc. for implementing a display control method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium and may be composed of, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, network controller, network card, or communication module.

The input device 1005 is an input device (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, speaker, LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one structure (e.g., a touch panel).

Furthermore, each device such as the processor 1001 and memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

Furthermore, the server 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA), and some or all of the functional blocks may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

Although the present embodiment has been described in detail above, it will be clear to those skilled in the art that the present embodiment is not limited to the embodiments described herein. The present embodiment can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

The order of the procedures, sequences, flowcharts, etc. of each aspect/embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

Input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

The determination may be made based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched between depending on the implementation. Furthermore, notification of specified information (e.g., notification that "X is true") is not limited to being done explicitly, but may also be done implicitly (e.g., not notifying the specified information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), these wired and/or wireless technologies are included within the definition of transmission media.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or corresponding other information.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the mathematical formulas and the like that use these parameters may differ from those explicitly disclosed in this disclosure. The various information elements may be identified by any suitable names, and the various names assigned to these various information elements are not intended to be limiting in any way.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the noun following these articles being plural.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

10...Server (display control system), 11...Receiving unit, 12...File generation unit, 13...File storage unit, 14...Display control unit, 20...User terminal, 24...Display unit, 24a...Display area, 50A...File (second file), 50B...File (second file), 50C...File (first file).

Claims (5)

A display control system that controls the display of a virtual object displayed on a display unit of a user terminal,
a file storage unit that stores, for a first virtual object that is one of the virtual objects, a first file having a first file size and a second file having a second file size larger than the first file size;
a display control unit that controls display of the first virtual object on the user terminal,
The display control unit
a first process of transmitting the first file to the user terminal with priority over the second file, thereby displaying the first virtual object based on the first file on the display unit;
a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file;
a third process of switching the first virtual object based on the first file displayed on the display unit to the first virtual object based on the second file when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal has been completed ;
The display control unit is configured to be able to switch, in the second process, between a first control pattern in which transmission of the second file to the user terminal is started after transmission of the first file, and a second control pattern in which transmission of the second file to the user terminal is performed in parallel with transmission of the first file, and to determine to execute the second control pattern when a communication speed of the user terminal is equal to or higher than a predetermined threshold.
Display control system. a file generation unit that acquires imaging data obtained by imaging a real object that is an object existing in a real space, and generates the first file and the second file of the virtual object corresponding to the real object based on the imaging data,
the file storage unit stores the first file and the second file generated by the file generation unit in association with the virtual object corresponding to the real object;
The display control system according to claim 1 . the display control unit, when the first virtual object based on the second file is displayed on the display unit, switches the first virtual object based on the second file displayed on the display unit back to the first virtual object based on the first file in accordance with an occurrence status of a processing delay of the user terminal;
The display control system according to claim 1 . when the first virtual object based on the second file is displayed on the display unit and a plurality of virtual objects including the first virtual object are displayed on the display unit, the display control unit switches back the first virtual object based on the second file displayed on the display unit to the first virtual object based on the first file, in accordance with the number of the virtual objects displayed on the display unit;
The display control system according to claim 1 . a terminal information acquisition unit that acquires terminal information indicating attributes of the user terminal;
the file storage unit stores a plurality of the second files having different file sizes;
the display control unit determines, based on the terminal information, the second file to be transmitted to the user terminal from among the plurality of second files;
The display control system according to claim 1 .