JP7634853B2 - Communication System - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act, Handout, Panasonic Corporation, February 25, 2021

Article 30, paragraph 2 of the Patent Act applies [Publications, etc.] SPORTS CHANGE MAKERS pre-event, Panasonic Center Tokyo, March 9, 2021

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://www. 1-10. com/project/smart-solution, August 19, 2021

Article 30, paragraph 2 of the Patent Act applies [Publications, etc.] Distributed materials, Panasonic Corporation, August 12, 2021

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://www. panasonic. com/global/olympic/ja/sportschangemakers. html, https://channel. panasonic. com/jp/contents/34773/, https://channel. panasonic. com/jp/contents/34774/, August 23, 2021

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://www. 1-10. com/news, https://www. 1-10. com/wpdata/wp-content/uploads/2021/10/Smart-Digital-Field_20211012_. pdf, https://youtu. be/6A1Yj4UeDBM, October 12, 2021

Article 30, paragraph 2 of the Patent Act applies [Publications, etc.] https://www. 1-10. com/news, https://www. 1-10. com/wpdata/wp-content/uploads/2022/01/SUSTAINABLE-CONNECT2022_PressRelease_220121. pdf, https://www. 1-10. com/wpdata/wp-content/uploads/2022/01/Follow-up SUSTAINABLE-CONNECT2022_Release_220125. pdf, https://eventregist. com/e/D20Bs5IwE3TG, January 21 and 25, 2020

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://youtu. be/5j3YVAK5Mq0, February 1, 2020

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://www. 1-10. com/project/qurios, https://www. 1-10. com/wpdata/wp-content/uploads/2022/07/%E3%83%95%E3%82%9A%E3%83%AC%E3%82%B9%E3% 83%AA%E3%83%AA%E3%83%BC%E3%82%B9_QURIOS%E4%BA%8B%E6%A5%AD_%E7%AC%AC1%E5%BC%BE_. pdf, https://prtimes. jp/main/html/rd/p/000000129.000016942. html, July 11, 2020

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://www. 1-10. com/wpdata/wp-content/uploads/2022/11/%E3%83%95%E3%82%9A%E3%83%AC%E3%8 2%B9%E3%83%AA%E3%83%AA%E3%83%BC%E3%82%B9_%E7%9F%B3%E8%A6%8B%E9%8A%80%E5 %B1%B1%E3%83%A1%E3%82%BF%E3%83%8F%E3%82%99%E3%83%BC%E3%82%B9%E3%83%95%E 3%82%9A%E3%83%AD%E3%82%B7%E3%82%99%E3%82%A7%E3%82%AF%E3%83%88_20221116. pdf, https://prtimes. jp/main/html/rd/p/000000142.000016942. html, November 16, 2020

Article 30, paragraph 2 of the Patent Act applies [Publications, etc.] November 26, 2022, Iwami Ginzan Thatched-roof house "Hinsha" (Omori-cho, Ota City, Shimane Prefecture)

Article 30, paragraph 2 of the Patent Act applies. [Publications, etc.] Tokyo Convention & Visitors Bureau, TOKYO MICE Technology Introduction Guidelines, page 18, January 27, 2023

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] https://prtimes. jp/main/html/rd/p/000000147.000016942. html, February 1, 2020

Article 30, Paragraph 2 of the Patent Act applies [Publications, etc.] Metaverse Japan Summit 2023, February 2, 2020

Article 30, paragraph 2 of the Patent Law applies [Publications, etc.] Business Guide List (Appendix)

The present invention relates to a technology used to link an event that takes place in real space with a simulated event that takes place in a virtual space to imitate that event.

In recent years, with the spread of high-speed Internet networks and advances in digital technology, systems have come into use that can hold various types of events in virtual spaces set up on the Internet. For example, there is a system that can hold simulated events in virtual spaces that mimic events such as live music concerts and costume contests that take place in real space. Patent Document 1 also describes holding online classes in a virtual space that has a room that mimics a classroom. In these systems, users participate in the simulated event by making their own avatars, which they have set up in advance, appear in the virtual space. Participants in the simulated event can have the same experience as participating in an event held in real space by moving their avatars within the virtual space.

JP 2013-88878 A

Using a system like the one described above, it is possible to hold a simulated event in a virtual space that imitates an event (real event) taking place in real space so that the events proceed simultaneously. However, with conventional systems, the real event taking place in real space and the simulated event taking place in virtual space proceed independently of each other, meaning that participants in these events are unable to feel a sense of unity as if they are taking part in a common event.

The problem that this invention aims to solve is to provide technology that allows participants in an event held in real space and participants in a simulated event held in virtual space to feel a sense of unity.

The communication system according to the present invention, which has been developed to solve the above problems, comprises:
a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit ;
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a second terminal used by the second user and equipped with an input unit and a display unit;
a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal;
and a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information, thereby placing an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on the display unit of the second terminal.

The communication system according to the present invention is used when an event (real event) is held in an event space (real event space) provided in a real space, and a simulated event corresponding to the real event is held in an event space (virtual event space) provided in a virtual space. In this system, a first user who participates in the real event in the real event space uses a first terminal equipped with an image acquisition unit and a display unit that acquire an image of the real event space captured from a viewpoint corresponding to the position of the first user. Also, a second user who participates in the simulated event in the virtual event space uses a second terminal equipped with an input unit and a display unit. Note that the image captured from a viewpoint corresponding to the position of the first user is an image captured from the viewpoint of the first user if the image acquisition unit is provided in a device such as MR glasses worn by the first user on the head, and is an image captured from a shooting unit of the tablet terminal if the image acquisition unit is provided in a portable device such as a tablet terminal used by the first user.

While the first user is participating in the real event, information on the position of the first terminal used by the first user moving within the real event space is acquired by the first terminal position acquisition unit . The information on the position of the first terminal may include information on the orientation of the first terminal. The second user can make an avatar appear in the virtual event space and move freely within it by operating an input unit of the second terminal. While the second user is participating in the simulated event, information on the position of the second user's avatar is acquired by the second user position acquisition unit.

The first display control unit converts the position of the second user's avatar into a position in the real event space based on the corresponding position information stored in advance in the storage unit, and displays an image in which the second user's avatar is placed at that position on the display unit of the first terminal. Therefore, the first user can view an image in which the second user's avatar is present in the real event space. Furthermore, the second display control unit converts the position of the first terminal into a position in the virtual space based on the corresponding position information, places the first user's avatar at that position, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the second user's avatar on the display unit of the second terminal. Therefore, the second user can view an image of the virtual event space including the avatar of the first user in the real event space. Note that the viewpoint corresponding to the position of the second user's avatar is not limited to the viewpoint of the avatar itself, but may be a viewpoint that captures the virtual event space including the avatar from behind the avatar. Furthermore, such multiple viewpoints may be switchable.

In this way, in the communication system according to the present invention, a first user participating in a real event in a real event space and a second user participating in a simulated event in a virtual event space can display each other's avatars on their respective devices, allowing participants in the real event taking place in the real space and participants in the simulated event taking place in the virtual space to feel a sense of unity.

The communication system according to the present invention comprises:
the storage unit further stores reference position information in the real event space,
The first terminal position acquisition unit may acquire information about the position of the first terminal by comparing an image of the real event space acquired by the image acquisition unit with the reference position information.

In the communication system of the above aspect, the image captured by the image acquisition unit of the first terminal is compared with reference position information in the real event space, thereby making it possible to acquire information on the position and orientation of the first terminal with high accuracy.

For example, MR (Mixed Reality) glasses can be used for the first terminal. MR glasses are glasses-type digital devices. There are two types of MR glasses: one that displays an image such as a 3D hologram on the lens of a glasses-type device worn by the user and allows the user to view it together with the real space directly captured by the user's own eyes (optical see-through type), and one that recognizes and images the real event space through a sensor mounted on the device, superimposes an image such as a 3D hologram on the image, and displays it on the display unit (video see-through type). When using video see-through MR glasses as the first terminal, for example, the reference position information included in the image of the real event space acquired through the sensor of the MR glasses may be collated with a previously prepared image of the real event space (for example, reference position information of the real event space). When using optical see-through MR glasses as the first terminal, for example, a configuration can be adopted in which short-distance communication is performed between sensors previously arranged at multiple positions in the real event space, the position of the first terminal is identified from the response time, and the orientation is identified by an acceleration sensor. MR glasses are worn in the same way as general glasses, so both hands are free. Tablet devices are also examples of video see-through devices.

In the communication system according to the present invention,
the storage unit further stores data of a plurality of emotion images, each of which is an image expressing an emotion of a user;
moreover,
an emotion image selection receiving unit that receives an operation to select one of the emotion images in one of the first terminal and the second terminal,
and an emotion image display unit that displays the selected emotion image together with an avatar of a user of the terminal whose operation has been accepted by the emotion image selection accepting unit on the other display unit of the first terminal and the second terminal.

In addition, in the communication system according to the present invention,
the storage unit further stores data of emotional actions, each of which is an avatar's action expressing an emotion of the user;
moreover,
an emotional action selection receiving unit that receives an operation to select one of the emotional actions in one of the first terminal and the second terminal;
and an emotional action executing unit that displays, on the other display unit of the first terminal and the second terminal, an image in which an avatar of a user of the terminal whose operation has been accepted by the emotional action selection accepting unit is performing the selected emotional action.

The above two aspects of the communication system allow for rich emotional expression between the first and second users.

By using the communication system according to the present invention, participants in an event taking place in real space and participants in a simulated event taking place in virtual space can feel a sense of unity.

1 is a schematic diagram showing the overall configuration of an embodiment of a communication system according to the present invention; FIG. 2 is a diagram showing the main configuration of a server device according to the present embodiment. FIG. 2 is a diagram showing the main configuration of a first terminal in the present embodiment. FIG. 2 is a diagram showing the main configuration of a second terminal in the present embodiment. 4 is an example of a display on a display unit of a first terminal according to the present embodiment. 13 shows an example of a display on a display unit of a second terminal according to the present embodiment. 11 is a display example showing the movements of an avatar on the display unit of the first terminal of the present embodiment. 11 is a display example showing the movement of an avatar on the display unit of the second terminal of the present embodiment. 13 is an example of an expanded display of an explanatory text on the display unit of the first terminal of the present embodiment. 13 is a display example in which an explanatory text is displayed in an expanded form on the display unit of the second terminal of the present embodiment. 13 is a display example in which video is synchronously displayed on the second terminal of the embodiment; 13 shows another display example of the display unit of the first terminal of the present embodiment. 13 is an example of a chat screen displayed on a display unit of a first terminal according to the present embodiment. 13 is another display example of the display unit of the second terminal of the present embodiment. 13 is an example of a chat screen displayed on a display unit of the first terminal or the second terminal of the present embodiment. 13 is an example of a participant list screen displayed on a display unit of the first terminal or the second terminal in the present embodiment. 5A to 5C are diagrams illustrating a manner in which an image acquired by a first terminal is displayed on a display unit of a second terminal in the present embodiment. 13 is an example of a state in which an image acquired by the first terminal is displayed full screen on the display unit of the second terminal in this embodiment. 13 is an example of a state in which an image acquired by the first terminal is displayed on a small screen on the display unit of the second terminal in this embodiment. 13 is an example of a state in which an image captured by an internal camera of a first terminal is displayed on a display unit of a second terminal in this embodiment. FIG. 1 is a diagram for explaining a digital twin embodied in the communication system of this embodiment. 13 is an example in which the communication system of this embodiment is applied to an outdoor event.

One embodiment of the communication system according to the present invention will be described below with reference to the drawings. The communication system 1 of this embodiment is used when simultaneously holding a real event in a real event space and a simulated event that imitates the real event in a virtual event space.

Figure 1 shows a schematic configuration of the entire communication system 1 of this embodiment. The communication system 1 of this embodiment includes a server device 10, a first terminal 30, and a second terminal 50, which are interconnected via the Internet. The first terminal 30 is used by a user (first user) who participates in a real event in a real event space. The second terminal 50 is used by a user (second user) who participates in a simulated event in a virtual event space. Although two first terminals 30 and three second terminals 50 are shown in Figure 1, these numbers can be changed as desired.

Figure 2 shows the main configuration of the server device 10. In addition to the memory unit 11, the server device 10 has the following functional blocks: a user registration reception unit 21, a participation reception unit 22, and a user intermediation unit 23. In this embodiment, the server device 10 is provided as a cloud server. The server device 10 may be any computer device capable of communicating with the first terminal 30 and the second terminal 50, and a personal computer or the like may also be used. The server device 10 may be provided with an input unit and a display unit as necessary.

The storage unit 11 is provided with a user information storage unit 12 and an event data storage unit 13. User information is stored in the user information storage unit 12. The user information may include each user's identification information (e.g., an identification number), username and password information, contact information (email address, etc.), the type and identification information of the terminal used by each user (e.g., the terminal model number and identification number. If the user owns multiple terminals, the type and identification information for each terminal), display data of each user's avatar used to display in the virtual event space, and identification information of users who interact with each user.

The event data storage unit 13 stores image data of the space (real event space) where the real event takes place, data on a number of reference positions, display data of the space (virtual event space) where the simulated event takes place, and corresponding position information that associates positions in the real event space with positions in the virtual event space. The corresponding position information is stored as information that associates a number of reference positions provided in the real event space with positions in the virtual space that correspond to each of the reference positions. This information can be created, for example, by using an AR marker or the like at each reference position, or by using a technology called VPS (Visual Positioning Service/System), which analyzes images from a camera attached to a smartphone or tablet to identify the position of the camera in a three-dimensional map prepared in advance. For example, a position having a visual feature such as a corner of an object existing in the space (a position that is easy to identify) is selected as the reference position in the real event space. The information on the multiple reference positions provided in the real event space corresponds to the reference position information in the present invention. Note that the multiple reference positions in the real event space included in the reference position information and the multiple reference positions in the real event space used in the corresponding position information may be the same or different.

3 shows the main configuration of the first terminal 30. The first terminal 30 includes a storage unit 31 and an image acquisition unit 32. The first terminal 30 also includes a first terminal position acquisition unit 33, a first display control unit 34, an expanded display unit 35, an emotion image selection reception unit 36, and an emotion action selection reception unit 37 as functional blocks. These functional blocks are embodied by running a dedicated application preinstalled in the first terminal 30, downloading a script described later from the server device 10, and executing a program for the first terminal on a processor. The application includes data that is commonly used regardless of the content of the event (for example, data such as an emotion display button 731 and an action display button 732 for expressing emotions) and a function to read and display the data from the storage unit 31. On the other hand, the script is a program that is created for each event and performs a unique process for each event.

The script downloaded to the first terminal 30 includes a program that performs a process of identifying the position and orientation of the first terminal 30 from an image captured in the real event space using reference position information of the real event space stored in the event storage unit 13, and a process of converting a position in the virtual event space to the real event space using the corresponding position information. For example, this program can parameterize the reference position information included in the image of the real event space acquired by a sensor at a predetermined time interval, and calibrate the position and orientation of the first terminal 30 by comparing it with pre-indexed reference position information.

For example, the first terminal 30 may be a portable device such as a tablet terminal or a smartphone, or a head-mounted device such as MR glasses, and the image acquisition unit 32 may be a camera included in these devices. The first terminal 30 is also provided with an input unit 41 and a display unit 42. As described above, in this embodiment, each functional block is realized by data and programs contained in a dedicated application and downloaded scripts, but these functional blocks may also be realized by only a dedicated application or only a script.

Figure 4 shows the main components of the second terminal 50. The second terminal 50 includes a storage unit 51. The second terminal 50 also includes, as functional blocks, a second user position acquisition unit 52, a second display control unit 53, an extended display unit 54, an emotion image selection reception unit 55, and an emotion action selection reception unit 56. These functional blocks are realized by running a dedicated application pre-installed in the second terminal 50, downloading a script described below from the server device 10, and executing a program for the second terminal on the processor. This application also includes data that is commonly used regardless of the content of the event (for example, data such as icons 731 and action buttons 732 for expressing emotions), and a function for reading and displaying such data from the storage unit 51.

The script downloaded to the second terminal 50 includes a program that converts a position in the real event space into a virtual event space using the corresponding position information stored in the event storage unit 13. The script is a program that is created for each event and performs processing unique to each event.

The second terminal 50 may be, for example, a stationary device such as a personal computer, a portable device such as a tablet terminal or a smartphone, or a head-mounted device such as VR glasses. The second terminal 50 is also provided with an input unit 61 and a display unit 62. As described above, in this embodiment, each functional block is realized by the data and program contained in a dedicated application and a downloaded script, but these functional blocks may be realized only by a dedicated application or only by a script.

Next, we will explain the process for holding real events and simulated events using the communication system 1 of this embodiment.

When a real event and a simulated event are held, information (such as the event name) that identifies a set of real and simulated events (hereinafter collectively referred to as an "event set") and information on the date and time of the event are stored in the event data storage unit 13 in advance. In addition, image data of the space in which the real event is held (real event space) and display data of the space in which the simulated event is held (virtual event space) are prepared and stored in the event data storage unit 13. Furthermore, multiple positions in the real event space are set as reference positions, positions in the virtual space corresponding to each of the multiple reference positions (corresponding reference positions) are determined, corresponding position information that associates the two is created and stored in the event data storage unit 13. Furthermore, data that associates the items, etc. with the text, etc. for each part of the items, such as exhibits, and walls, etc., that are placed in the real event space and that display text, images, or videos related to the items, etc. (hereinafter referred to as "extended display targets") is also stored in the event data storage unit 13. Furthermore, the event data storage unit 13 also stores data on images and videos that are planned to be displayed on a screen or the like in the real event space during the actual event, as well as information on the display location and timing of the display.

The event data storage unit 13 also stores scripts (programs) to be executed by the first terminal 30 and the second terminal 50. The program operated by the first terminal 30 performs processing such as identifying the position and orientation of the first terminal 30 by comparing an image acquired by the image acquisition unit 32 of the first terminal 30 with data (reference position information) of a plurality of reference positions set in the real event space, and generating display data of the second user's avatar according to the position and orientation. The program operated by the second terminal 50 performs processing such as generating display data of the first user's avatar and avatars of other second users according to the position and orientation of the second user set by operation through the input unit 61 of the second terminal 50. Details of the operations in the first terminal 30 and the second terminal 50 will be described later.

Once the process of storing this necessary data in the event data storage unit 13 is complete, real events and simulated events can be held.

The participation acceptance unit 22 reads out the event sets stored in the storage unit 11 that are in a state where they can be held and accepts participation. The participation acceptance unit 22 sends an e-mail containing the address of a website containing the event sets for which participation is accepted to the user's contact information stored in the storage unit 11. The participation acceptance unit 22 also notifies the information of the event sets for which participation is accepted through an application pre-installed on a terminal owned by a person who has already been registered as a user. The user who receives the e-mail or notification accesses the website described in the received e-mail, etc., enters his/her user name and password to log in, and then checks the information of the event set in which he/she wishes to participate. Then, for the event set in which he/she wishes to participate, he/she selects whether to participate in a real event or a simulated event. If the user selects participation in a real event, a screen for downloading a script to be run on the first terminal 30 is displayed. When the script is downloaded, the participation acceptance unit 22 sets the user as the first user in the event set, sets the terminal of the user stored in the user information storage unit 12 to the first terminal 30, and stores this information in the event data storage unit 13. Furthermore, if the user selects to participate in the simulated event, a screen is displayed for downloading a script to be run on the second terminal 50. Once the script has been downloaded, the participation acceptance unit 22 sets the user as the second user in the event set, sets the terminal of the user stored in the user information storage unit 12 to the second terminal 50, and stores this information in the event data storage unit 13.

The site that accepts participation in the event set also has a screen where new users can register. When an unregistered person selects to display the screen for new user registration, the user registration acceptance unit 21 prompts the person to enter contact information (such as an email address). When the contact information is entered, the user registration acceptance unit 21 sends the contact information an email with a link to a website for user registration, and prompts the person to set a user name and password. When the user name and password are set, a new identification number is assigned to the new user and saved in the user information storage unit 12. New users can also be registered in the same manner as above by downloading a program for event participants.

A user who is registered is provided with a settings page, through which the user can change their registration information, create an avatar, enter information about the device they are using, etc.

Here, we have explained the case where registered users are allowed to participate in the event, but it is also possible to allow users to participate in the event without registering, by simply downloading a script and setting a username.

When the date and time of the event set arrives, the real event starts in the real event space. At the same time, a simulated event starts on a pre-set website. These real and simulated events are a pair of event sets that proceed synchronously, and it is also possible for a first user participating in the real event and a second user participating in the simulated event to interact with each other. In other words, one event set is executed in the real event space and the digital twin space that is paired with the real event space. A person participating as a second user participates in the simulated event by entering a user name and password to access this website. When participation in the simulated event begins, the avatar of the second user is placed at an initial position (for example, the entrance to the virtual event space). During the simulated event, the second user can move his or her avatar to any position in the virtual event space by operating the input unit.

In the first terminal 30 used by the first user participating in a real event in the real event space, the image acquisition unit 32 acquires image data of the real event space at a predetermined time interval. The image data acquired at the predetermined time interval may also include video data. The first terminal position acquisition unit 33 compares the image data acquired by the image acquisition unit 32 with data of multiple reference positions in the real event space using the previously downloaded script (program), extracts the reference positions contained in the image, and acquires information on the position and orientation in the real event space where the image was acquired (i.e., the position and orientation of the first terminal 30 in the real event space) based on the extracted reference positions. The above-mentioned VPS can be suitably used for this process. The first terminal position acquisition unit 33 transmits the position and orientation information of the first terminal 30 to the server device 10 at a predetermined time interval (or upon receiving a request from the server device 10).

The server device 10 transmits to the first terminal 30 at a predetermined time interval the information on the position and orientation of the second user's avatar and the display data of the avatar that are collected at that time. The first terminal 30 converts the position of the second user's avatar in the virtual event space to a position in the real event space based on the corresponding position information using the previously downloaded script (program). Next, the first display control unit 34 generates a 3D hologram image of the avatar and generates display data in which each avatar is given a user name. In addition, an image in which the display data including the generated 3D hologram image is superimposed on the image data of the real event space acquired by the image acquisition unit 32 is displayed on the display unit 42 (if the first terminal 30 is an optical see-through MR glass, only the 3D hologram image is displayed on the display unit 42). When there are multiple first users and the other first users appear in the image of the real event space acquired by the image acquisition unit 32, a 3D hologram image of the second user's avatar is superimposed on the image of the real event space including the other first users and displayed on the display unit 42 of the first terminal 30. In other words, on the display unit 42 of the first terminal 30, the other first users are displayed as real images and the second user is displayed as an avatar.

Figure 5 shows an example of a screen displayed on the display unit 42 of the first terminal 30. On the display unit 42 of the first terminal 30 used by the first user, an image is displayed in which an avatar 72 of a second user participating in a simulated event and their user name are superimposed on an image of a real event space including real images of other first users participating in the real event. Note that the example screen shown in Figure 5 does not include the real image of the first user (i.e., there are no other first users in the direction in which the first user is facing in the real event space), and the avatars 72 of the three second users are displayed superimposed on the background of the real space.

In the second terminal 50 used by a second user participating in a real event in the virtual event space, each time the second user moves his or her avatar via the input unit 61, the input information is transmitted to the server device 10. In the second terminal 50, each time the avatar moves, the second user position acquisition unit 52 identifies the position and orientation of the second user's avatar in the virtual event space, and sequentially transmits the information to the server device 10.

In addition, the server device 10 transmits, at a predetermined time interval, the display data of the virtual event space stored in the event data storage unit 13, along with the position and orientation of the first user, the display data of the avatar of the user, and the position and orientation information of the avatar of the other second users, and the display data of the avatar to the second terminal 50. In the second terminal 50, the position of the first terminal 30 in the real event space is converted to a position in the virtual event space based on the corresponding position information by the previously downloaded script (program). Next, the second display control unit 53 generates an image in which the avatar of the first user is placed in the virtual event space as seen from the avatar of the second user who is the user of the second terminal 50 according to the position and orientation of the first terminal 30, and further the avatars of the other second users are placed according to the positions and orientations of the other second users, and displays the image on the display unit 42. In this way, the first user and the other second users are both displayed as avatars on the display unit 62 of the second terminal 50. Here, an image of the virtual event space captured from the viewpoint of the second user's avatar is displayed, but an image capturing the virtual event space including the second user's avatar from behind the avatar (an image including the second user's own avatar) may also be displayed. Furthermore, images of the virtual event space captured from multiple viewpoints may be displayed in a switchable manner.

Figure 6 shows an example of a screen displayed on the display unit 62 of the second terminal 50. As shown in Figure 6, the display unit 62 of the second terminal 50 used by the second user displays an avatar 71 of a first user participating in the real event and an avatar 72 of another second user participating in the simulated event, along with an image of the virtual event space. Note that by changing the display settings, it is also possible to display the username of the first user and the username of the second user on the screen together.

In the example shown in FIG. 6, an auxiliary screen 73 is displayed in the lower left of the display unit 62. An emotion display button 731 for switching between displaying and hiding icons for expressing one's own emotions and an action display button 732 for switching between displaying and hiding actions are displayed on this auxiliary screen 73. The display of the auxiliary screen 73 can be turned on and off by a predetermined input operation (for example, double-tapping the lower left area of the screen of the display unit 62). FIG. 6 shows the state in which the emotion display button 731 is clicked, and as a result, icons 7311 corresponding to a plurality of emotional expressions are displayed. In this embodiment, the icons 7311 are image data, but they are not limited to image data and may be modified text data or the like.

In the screen shown in FIG. 6, the second user can express his/her own emotion by selecting one of the icons 7311. When the second user performs an operation to select one of the icons 7311, the emotion image selection receiving unit 55 receives the operation, and the information of the selected icon 7311 is transmitted to the server device 10. The server device 10 accumulates information received from the second terminal 50, and when transmitting the position and orientation of the avatar of the second user and the display data to the first terminal 30 at the above-mentioned predetermined time interval, transmits the information of the icon 7311 received from the second terminal 50 to the first terminal 30 with the information attached. Similarly, when transmitting the position and orientation of the avatar of another second user and the display data to the second terminal 50, transmits the information of the icon 7311 received from the other second terminal 50 (the second terminal 50 used by the other second user) to the other second terminal 50 with the information attached.

When the first terminal 30 receives the information of the icon 7311 together with the location information of the second user, the first display control unit 34 generates image data for displaying the icon 7311 near the avatar 72 of the user of the second terminal 50 who selected the icon 7311, and displays the image data on the display unit 42. As shown in FIG. 5, the first terminal 30 displays the icon 7311 above the avatar 72 of the second user. Similarly, when the second display control unit 53 receives the information of the icon 7311 together with the location information of another second user, the second display control unit 53 generates image data for displaying the icon near the avatar 72 of the user of the second terminal 50 who selected the icon 7311, and displays the image data on the display unit 62. As shown in FIG. 6, the icon 7311 is displayed on the second terminal 50.

In addition, in the screen shown in FIG. 6, the second user can also express his/her own emotions by clicking the action display button 732 and selecting an action displayed. When the second user performs an operation to select one of the actions, the emotion action selection receiving unit 56 receives the operation, and the information of the selected action is transmitted to the server device 10. The server device 10 accumulates information received from the second terminal 50, and when transmitting the position and orientation of the avatar of the second user and the display data to the first terminal 30 at the above-mentioned predetermined time interval, the information of the action received from the second terminal 50 is added and transmitted to the first terminal 30. Similarly, when transmitting the position and orientation of the avatar of another second user and the display data to the second terminal 50, the information of the action received from the other second terminal 50 (the second terminal 50 used by the above-mentioned other second user) is added and transmitted to the other second terminal 50.

When the first terminal 30 receives information about an action together with the location information of a second user, the first display control unit 34 generates image data for making the avatar 72 of the user of the second terminal 50 who selected the action perform an action corresponding to the action, and displays the image data on the display unit 42. On the first terminal 30, an image including the action of the avatar 721 is displayed on the display unit 42, as shown in FIG. 7. Similarly, when the second display control unit 53 receives information about an action together with the location information of another second user, it generates image data for making the avatar 721 of that second user perform the action, and displays the image data on the display unit 62, as shown in FIG. 8.

Here, an example in which the second user selects the emotion display button 731 or the action display button 732 has been described, but the first user can also select the icon 731 or the action button 732 in the same way, and the emotion image selection receiving unit 36 and the emotion action selection receiving unit 37 of the first terminal 30 function in the same way as described above. When a predetermined operation (e.g., double tapping the screen) is performed on the screen of the display example shown in FIG. 5, the auxiliary screen 73 can be displayed on the display unit 42 as in FIG. 6. The auxiliary screen 73 can be operated in the same way as the second user, and the server device 10 performs the same process as when the icon 7311 or the action is selected by the second user. However, in this embodiment, when the icon 7311 or the action is selected by the first user, only the second display control unit 53 generates image data according to the selection and displays it on the display unit 62. In other words, the image data displayed on the display unit 42 of the first terminal 30 viewed by the first user does not change before or after the selection of the action. This is because the other first users displayed on the first terminal 30 are not avatars but real images, and it is impossible to impart motion to the real images of the first users. However, since it is possible to display the icon 7311 near the first users displayed as real images, when the first user selects the icon 7311, the icon 7311 may be displayed near the real image of the user. Also, when an action is selected, an icon with a motion corresponding to the selected action may be displayed near the real image of the first user.

During the actual event, the extended display unit 35 identifies the line of sight direction (the direction captured by the image acquisition unit 32 of the first terminal 30) corresponding to the position of the first user based on the information on the position and attitude of the first terminal 30 acquired by the first terminal position acquisition unit 33. Then, when it detects that the direction of the first user's line of sight is staying on a preset extended display target 74 for a certain period of time (e.g., 5 seconds), it reads out extended display information corresponding to the extended display target 74 from the event data storage unit 13. When the extended display information is read out, the first display control unit 34 displays an image on the display unit 42 in which an extended display 741 is superimposed near the extended display target 74, as shown in FIG. 9.

Also, while the simulated event is being held, the extended display unit 54 identifies the line of sight of the second user's avatar 72 based on the position and posture information of the second user's avatar 72. Then, when it detects that the line of sight of the second user's avatar 72 is staying on a preset extended display target for a certain period of time (e.g., 5 seconds), it reads out extended display information corresponding to the extended display target from the event data storage unit 13. When the extended display information is read out, the second display control unit 53 displays on the display unit 62 an image in which an extended display 741 is superimposed near the extended display target 74, as shown in FIG. 10.

During a real event, images and the like may be projected onto a screen or the like provided in the real event space in accordance with the progress of the event that has been prepared in advance. When the projection time (time elapsed since the start of the event) and the projection location (screen, etc.) are determined in advance in this way, the data of the video and audio to be projected is stored in the event data storage unit 13. When the time elapsed since the start of the simulated event reaches the projection time, the second display control unit 53 reads out the projection data of the video and audio data stored in the event data storage unit 13, displays an image of the video projected onto the screen in the virtual event space on the display unit 62, and outputs audio based on the audio data. This allows a first user participating in the real event in the real event space (left figure) and a second user participating in the simulated event in the virtual event space (right figure) to simultaneously view the same video and other video images, as shown in FIG. 11.

As described above, while a moving image is being projected onto a screen or the like and sound is being output, on the second terminal 50, the second user can increase the volume of the moving image by moving their avatar 72 closer to the screen, and decrease the volume of the moving image by moving their avatar 72 away from the screen.

In the communication system 1 of this embodiment, text chat (open chat and private chat) and voice calls can be performed between a first user and a second user, or between second users, and images and videos can be distributed from a first user to a second user. These are described below. In the following operations, similar to the operations described above, the first display control unit 34 controls the screen display of the display unit 42 of the first terminal 30, and the second display control unit 53 controls the screen display of the display unit 62 of the second terminal 50, but to avoid redundant explanation, descriptions of the first display control unit 34 and the second display control unit 53 are omitted.

Open chat is a chat in which both first users who are participants in the real event and second users who are participants in the simulated event can post and view messages. Private chat includes TORIMAKI chat and direct chat. Private chat will be described later.

Figure 12 shows an example of a screen displayed on the display unit 42 of the first terminal 30 used by the first user. At the bottom of the screen, a chat button 81, a start distribution button 82, and a participant button 83 are provided. In addition, in this example screen, an auxiliary screen 73 including an emotion display button 731 (emoticon) for calling up the above-mentioned icon 7311 and an action display button (emo action) 732 is displayed in the upper left of the screen.

When the chat button 81 is pressed, a chat screen 811 as shown in FIG. 13 is displayed. An OPEN button 812 and a TORIMAKI button 813 are arranged on the chat screen 811. Pressing the OPEN button 812 displays the open chat screen, enabling posting and viewing. Pressing the TORIMAKI button 813 displays the TORIMAKI chat screen. Furthermore, pressing the chat button 81 while the chat screen 811 is displayed (FIG. 13) switches the chat screen to hidden (FIG. 12).

When the second user performs a predetermined operation to select one of the avatars 71 of the first user displayed on the display unit 62 of the second terminal 50 (e.g., double-tapping the avatar 71), a "Become TORIMAKI" button 92 is displayed near the avatar 71 of the first user (at the top in FIG. 14), as shown in FIG. 14.

When the second user presses the "Become TORIMAKI" button 92 on the display unit 62, the user intermediation unit 23 in the server device 10 identifies the first user corresponding to the avatar 71 whose button was pressed. Then, the first terminal 30 of the first user is notified that there has been a TORIMAKI request from the second user, and the server device 10 is inquired as to whether or not to accept the request. When the first user accepts the TORIMAKI request, the information is sent to the server device 10. When the server device 10 receives the information that the first user has accepted the second user to become TORIMAKI, the user intermediation unit 23 links the first terminal 30 with the second terminal 50 owned by the second user who has been accepted as TORIMAKI for the first user who is the owner of the first terminal 30, and registers the second user as TORIMAKI for the first user (setting the connection of the second user to the first user). In addition, a chat room is opened for the two users to have private chats (TORIMAKI chats). This establishes a link between the first terminal 30 and the second terminal 50 owned by the second user who is authorized as a TORIMAKI of the first user who is the owner of the first terminal 30. In this embodiment, one or more second users can simultaneously become TORIMAKI for one first user.

When the first user presses the TORIMAKI button 813 arranged on the chat screen 811 shown in FIG. 13, a screen is displayed for conducting a private chat between the first user and the second user who is the first user's TORIMAKI. Also, when the second user presses the TORIMAKI button 912 on the chat screen 91 after being approved as a TORIMAKI (see FIG. 15), a screen is displayed for conducting a private chat between the first user, the first user who is the TORIMAKI, and other TORIMAKIs (i.e., the first user and all TORIMAKIs of the first user).

This chat screen 91 is provided with an OPEN button 911, a TORIMAKI button 912, and a DIRECT button 913. FIG. 15 shows an example in which the TORIMAKI button 912 is pressed and the TORIMAKI chat screen is displayed. Only the first user and the second user who is authorized as the TORIMAKI of the first user can post and view messages in this TORIMAKI chat. In the TORIMAKI chat, every time text information is input from the first user or the second user, the text information is transmitted to each other's terminal via the user intermediation unit 23 of the server device 10 and displayed on the respective display units 42 and 62. Furthermore, when the first user accepts a TORIMAKI request from the second user, in addition to the private chat described above, it is also possible to have a voice call between the two users. This voice call may be either between all TORIMAKIs or between only one of the TORIMAKIs. Moreover, it may be possible to switch between them.

In addition, the first user participating in the real event and the second user participating in the simulated event can be confirmed from the participant list display 93 (enlarged view is shown again in FIG. 16) displayed in the lower right of the screen shown in FIG. 14. By pressing either of the chat buttons 933 displayed next to the first user name 931 or the second user name 932 in this participant list display 93, a one-to-one direct chat with the user is started. By pressing the DIRECT button 913 on the chat screen 91 shown in FIG. 15, the direct chat screen is displayed. During the direct chat, one-to-one voice calls with the user are also possible. By pressing the button 934 displayed next to the first user, a TORIMAKI request is sent to the first user in the same way as when the "Become TORIMAKI" button 92 is pressed.

Although direct chat is not displayed in the example screen of the display unit 42 of the first terminal 30 shown in FIG. 13, direct chat and voice call can be performed on the first terminal 30 by specifying either the first user or the second user. When the participant button 83 displayed at the bottom left of the screen examples of FIG. 12 and FIG. 13 is pressed, a list of the first user and the second user is displayed as in FIG. 16. Direct chat and voice call can be started by selecting one user from this list and pressing the chat button. Note that since first users participating in a real event in real space can talk to each other on the spot, a configuration may be adopted in which direct chat between first users is not performed. However, in consideration of cases where a conversation is to be held about matters that should be kept secret from other first users, in this embodiment, direct chat between first users is also possible.

The distribution start button 82 shown in Figures 12 and 13 is a button that executes a process in which the first user distributes images (including videos) acquired by the image acquisition unit 32 of the first terminal 50 that the first user uses to the second user who is the user's TORIMAKI. For example, the second user who is the TORIMAKI can request the first user to take pictures of a place of interest in real space through chat or voice call, and the first user who receives the request can distribute images and videos taken of the requested place on his/her first terminal 30 to the second user who is the user's TORIMAKI in real time (see Figure 17). At this time, the first user can orally explain the images and videos being captured, and the audio data can also be distributed to the TORIMAKI at the same time.

Image, video, and audio data distributed from the first terminal 30 is sent via the user intermediary unit 23 of the server device 10 to the second terminal 50 owned by the second user who is a TORIMAKI of the first user who is the owner of the first terminal 30. On the second terminal 50, images and videos captured by the image acquisition unit 32 of the first terminal 30 are displayed full screen on the display unit 62 (see FIG. 18). This allows the second user to share in real time images and videos of the real event space captured by the image acquisition unit 32 of the first terminal 30 used by the first user who is now a TORIMAKI. The size of the small screen on the display unit 62 of the second terminal 50 can be changed as appropriate.

However, images and videos delivered by a first user in response to a request from a second user may not necessarily be of interest to all TORIMAKI. Therefore, in the full-screen display state shown in FIG. 18, on the second terminal 50, the display of images and videos delivered from the first terminal 30 can be ended by pressing the EXIT button 941, or the images and videos delivered from the first terminal 30 can be displayed on the small screen 95 by pressing the reduce button 942 (see FIG. 19).

Figures 17 to 19 show an example in which the first user distributes images and videos of a real event space taken by the first terminal 30 to his/her TORIMAKI, but by changing the settings of the image acquisition unit 32 of the first terminal 30 (changing to a setting that uses the inner camera), the first user can also distribute his/her own image to the TORIMAKI. This allows the first user to distribute his/her own image by talking to his/her TORIMAKI, etc. Figure 20 shows an example of a screen displayed on the display unit 62 of the second terminal 50 when the first user distributes his/her own image using the inner camera. Note that when the first user distributes his/her own image using the inner camera, the image acquired by the inner camera may be displayed full screen on the display unit 42 of the first terminal 30, or the image acquired by the inner camera may be displayed in a small screen on top of the image acquired by the outer camera.

The user mediation unit 23 also writes mutual identification information into the user information of the first user and the second user approved by the first user as TORIMAKI, and stores the information in the user information storage unit 12. As a result, each user's user information is recorded as a user who has interactions as TORIMAKI. A list of users who have interactions can be confirmed, for example, in a user information setting unit that each user can access using a username and password. Alternatively, when the real event or simulated event ends, or when the first user or the second user leaves the event, permission to record each other as a user who has interacted as TORIMAKI is requested, and only when permission is obtained from both users, the user may be recorded as an interacting user. In addition, a configuration can be adopted in which multiple stages are set for interactions between users, such as disclosing all user information to each other, disclosing only usernames and contact information, and disclosing only usernames, and information at the level approved by each user is disclosed.

By adopting the above configuration, the communication system 1 of this embodiment can provide an environment (digital twin environment) that is identical to the real space (upper part of FIG. 21) in the virtual space (lower part of FIG. 21). In addition, by utilizing the above TORIMAKI system in such a digital twin environment, images and videos captured in the real space can be shared in real time with a second user participating in the virtual space.

Conventionally, there are things that allow users to have various experiences in virtual spaces that mimic real space, but when recreating real space in a virtual space, a virtual space is often created that omits some of the information in the real space from the perspective of the amount of data and the load on image processing. As a result, it is often not possible to experience the same realism in a virtual space as in a real space. In contrast, the above TORIMAKI system allows a second user who becomes the TORIMAKI of a first user to share realistic images and videos taken by a first user in real space. Note that while the above examples are all indoor real spaces, it is also possible to provide a virtual space that is a digital twin of an outdoor real space, as shown in FIG. 22.

The communication system 1 of the present embodiment can be used in various situations. For example, if the first user is a celebrity or an influencer, the fan can become the TORIMAKI of the first user and follow the activities of the first user. Alternatively, for a place where entry is prohibited to those other than those involved, a person involved in the place can become the TORIMAKI and request the delivery of images of places (backyards) and scenes (animal birth scenes) that cannot usually be seen and explanations of the place. Examples of such places include amusement parks, zoos, aquariums, factories, and airport backyards, and places where important cultural properties are installed in temples and art museums. Furthermore, a clerk at a shop such as an apparel brand can become the first user, and a person interested in the shop can become the TORIMAKI and request an introduction of the shop's products, etc. Furthermore, a person with a disability or a recluse who has difficulty going out can become a tour guide or a resident TORIMAKI in a place of interest to them and request the delivery of images and explanations of the place of interest. It can also be used to provide an educational setting for people living in remote areas.

As described above, in the communication system 1 of this embodiment, not only is a real event in the real event space and a simulated event in the virtual event space performed simultaneously in parallel, but the position and orientation of the first terminal 30 are identified based on the reference position information stored in advance in the event data storage unit 13, and the position information of the real event space and the virtual event space are mutually converted based on the corresponding position information. This allows the avatar 71 of the first user participating in the event in the real event space to be displayed in the virtual event space on the display unit 62 of the second terminal 50. In addition, a three-dimensional hologram image of the avatar 72 of the second user can also be displayed on the display unit 42 of the first terminal 30. In addition, since the moving image projected in the real event can be simultaneously projected in the simulated event in the virtual event space, the first user and the second user can simultaneously watch the same moving image.

In the communication system 1 of this embodiment, the first user and the second user can appeal to other users by selecting the emotion display button 731 that expresses their own emotion and the action display button 732. In addition, the first user and the second user, or the second users can communicate with each other through private chat.

Furthermore, in the communication system 1 of this embodiment, a second user participating in a simulated event in a virtual space can share realistic images and videos in the real space by becoming a TORIMAKI of a first user participating in a real event taking place in the real space.

The above embodiment is merely an example and can be modified as appropriate in accordance with the spirit of the present invention.

In the above embodiment, the server device 10 has an event data storage unit 13 in the storage unit 11, and various data required for holding an event is stored therein. However, some or all of the data stored in the event data storage unit 13 may be stored in the storage unit 31 of the first terminal 30 or the storage unit 51 of the second terminal 50. In addition, in the above embodiment, the first terminal 30 has a first terminal position acquisition unit 33, the second terminal 50 has a second user position acquisition unit 52, and the first display control unit 34 in the first terminal 30 generates data to be displayed on the display unit 42, and the second display control unit 53 in the second terminal 50 generates data to be displayed on the display unit 62. However, it is also possible to adopt a configuration in which some or all of the functional blocks including these are provided in the server device 10.

In the above embodiment, the first terminal position acquisition unit 33 identifies the position and orientation of the first terminal 30 by comparing the image data acquired by the image acquisition unit 32 of the first terminal 30 with the data of multiple reference positions in the real event space stored in the event data storage unit 13, but other methods can also be used. For example, sensors can be placed in advance at multiple positions in the real event space, and the position of the first terminal 30 can be identified by performing short-range communication with the first terminal 30 to obtain the response time for each sensor, and the orientation of the first terminal can be identified by an acceleration sensor that the first terminal 30 has.

In the above embodiment, video data with a predetermined projection time and projection location is used, but it is possible that an unplanned video is projected before the event begins. In that case, for example, image data of the real event space can be generated by integrating image data acquired by the image acquisition units 32 of multiple first terminals 30 at appropriate time intervals, and the image data of the real event space stored in the event data storage unit 13 can be compared to detect differences between the two, and projection data can be generated from image data of the location where the difference was detected. Alternatively, multiple image acquisition units can be arranged in advance in the real event space, and when it is detected that an unplanned video is being projected based on changes in the images acquired by each image acquisition unit, projection data can be generated from data from the image acquisition unit that detected the video.

In the above embodiment, the avatar 71 of the first user and the avatar 72 of the second user are all displayed in the virtual event space. However, if many second users participate in a simulated event held in the virtual event space, it may be impossible to display all of the avatars 72 of the second users in the virtual event space. In that case, multiple virtual event spaces may be provided, the second users may be divided into multiple groups, and the avatars 72 may be displayed in the virtual event spaces provided for each group. In that case, only the avatar 72 of the second user in one of the virtual event spaces may be displayed on the display unit 42 of the first terminal 30. The first user may be allowed to select which virtual space's avatar is to be displayed on the display unit 42 of the first terminal 30. Alternatively, only the avatar 72 of the second user who has interactions with the first user may be displayed on the display unit 42 of the first terminal 30.

In addition, when enjoying an event as a TORIMAKI of a first user who is a celebrity, the second user who is the TORIMAKI often does not have a particular interest in other first users or second users, and is likely to feel annoyed by the display in the virtual space of avatars 71 of first users other than the first user who is the TORIMAKI and avatars 72 of other second users. In such a case, it is preferable to adopt a configuration in which the second user can perform a predetermined operation to switch between a display mode that displays only the avatar 71 of the first user who is the TORIMAKI, or a display mode that displays only the avatar 71 of the first user who is the TORIMAKI and the avatar 72 of the second user who is the TORIMAKI.

In the TORIMAKI system of the above embodiment, one or more second users become TORIMAKI for one first user, but multiple second users may become TORIMAKI for multiple first users to form a TORIMAKI group. In that case, it is recommended that the display unit 62 of the second terminal 50 is provided with separate screens for displaying images and videos simultaneously distributed by multiple first users.

In the above embodiment, one virtual event space is associated with one real event space, but multiple different virtual event spaces may be associated with one real event space. For example, virtual event spaces provided by multiple different businesses may be associated with one real event, and virtual spaces created by different businesses may be linked via an API (Application Programming Interface) to enable users to move between multiple virtual event spaces provided in correspondence with one real event space. Alternatively, an E-commerce (Electric Commerce) function may be set in the virtual space corresponding to the real event space, so that more experiences can be made in the virtual event space than in the real event space. In the above embodiment, a virtual event space is modeled after a real event space, but the virtual event space does not necessarily have to model after the real event space as long as it corresponds to the real event space and the positions in the space are associated with each other. For example, a virtual event space that is an appropriate deformation of the real event space can be used.

[Aspects]
It will be apparent to those skilled in the art that the above-described exemplary embodiments are illustrative of the following aspects.

(Section 1)
A communication system according to one aspect of the present invention comprises:
a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit ;
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a second terminal used by the second user and equipped with an input unit and a display unit;
a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal;
and a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information, thereby placing an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on the display unit of the second terminal.

(Article 11)
A program for a communication system according to another aspect of the present invention includes:
a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit;
a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit,
The first terminal,
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
A program characterized by operating as a first display control unit that converts the position of the second user's avatar acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the second user's avatar is placed at the converted position on the display unit of the first terminal.

The communication system according to paragraph 1 and the communication system embodied by executing the program according to paragraph 11 are used when holding an event (real event) in an event space (real event space) set up in real space, and holding a simulated event corresponding to the real event space in an event space (virtual event space) set up in a virtual space. In this system, a first user who participates in the real event in the real event space uses a first terminal equipped with an image acquisition unit and a display unit that acquires an image of the real event space captured from a viewpoint corresponding to the position of the first user. Also, a second user who participates in the simulated event in the virtual event space uses a second terminal equipped with an input unit and a display unit. Note that the image captured from a viewpoint corresponding to the position of the first user is an image captured from the viewpoint of the first user if the image acquisition unit is provided in a device such as MR glasses worn by the first user on the head, and is an image captured from the imaging unit of the tablet terminal if the image acquisition unit is provided in a portable device such as a tablet terminal used by the first user.

While the first user is participating in the real event, information on the position of the first terminal used by the first user moving within the real event space is acquired by a first terminal position information acquisition unit. The information on the position of the first terminal may include information on the orientation of the first terminal. The second user can make an avatar appear in the virtual event space and move it freely within it by operating an input unit of the second terminal. While the second user is participating in the simulated event, information on the position of the second user's avatar is acquired by a second user position acquisition unit.

The first display control unit converts the position of the second user's avatar into a position in the real event space based on the corresponding position information stored in advance in the storage unit, and displays an image in which the second user's avatar is placed at that position on the display unit of the first terminal. Therefore, the first user can view an image in which the second user's avatar is present in the real event space. Furthermore, the second display control unit converts the position of the first terminal into a position in the virtual space based on the corresponding position information, places the first user's avatar at that position, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the second user's avatar on the display unit of the second terminal. Therefore, the second user can view an image of the virtual event space including the avatar of the first user in the real event space. Note that the viewpoint corresponding to the position of the second user's avatar is not limited to the viewpoint of the avatar itself, but may be a viewpoint that captures the virtual event space including the avatar from behind the avatar. Furthermore, such multiple viewpoints may be switchable.

In this way, in the above communication system, a first user participating in a real event in a real event space and a second user participating in a simulated event in a virtual event space can display each other's avatars on their respective devices, allowing participants in the real event taking place in the real space and participants in the simulated event taking place in the virtual space to feel a sense of unity.

(Section 2)
A communication system according to the second aspect of the present invention is a communication system according to the first aspect of the present invention,
the storage unit further stores image data of the real event space,
The first terminal position acquisition unit acquires information about the position of the first terminal by comparing an image of the real event space acquired by the image acquisition unit with the image data.

In the communication system according to paragraph 2, the image captured by the image capture unit of the first terminal is compared with reference position information in the real event space, thereby making it possible to obtain information on the position and orientation of the first terminal with high accuracy.

(Section 3)
A communication system according to the third aspect of the present invention is a communication system according to the first or second aspect of the present invention,
the storage unit further stores data of a plurality of emotion images, each of which is an image expressing an emotion of a user;
moreover,
an emotion image selection receiving unit that receives an operation to select one of the emotion images in one of the first terminal and the second terminal;
an emotion image display unit that displays the selected emotion image together with an avatar of a user of the terminal whose operation has been accepted by the emotion image selection accepting unit, on the other display unit of the first terminal or the second terminal.

(Section 4)
A communication system according to paragraph 4 is a communication system according to any one of paragraphs 1 to 3,
the storage unit further stores data of emotional actions, each of which is an avatar's action expressing an emotion of the user;
moreover,
an emotional action selection receiving unit that receives an operation to select one of the emotional actions in one of the first terminal and the second terminal;
and an emotional action performing unit that displays, on the other display unit of the first terminal and the second terminal, an image in which an avatar of a user of the terminal whose operation has been accepted by the emotional action selection accepting unit is performing the selected emotional action.

The communication systems according to paragraphs 3 and 4 allow a wide range of emotional expressions between the first and second users.

(Section 5)
A communication system according to paragraph 5 is a communication system according to any one of paragraphs 1 to 4,
The storage unit further stores information on augmented display objects to be arranged in each of the real event space and the virtual space, and augmented display content associated with the augmented display objects;
moreover,
an extended display unit that, when it is determined based on position information acquired by the first terminal position acquisition unit that the gaze of the user of the first terminal is staying on the extended display target for a predetermined period of time, generates image data for displaying the extended display content together with the extended display target on a display unit of the first terminal, and, when it is determined based on position information acquired by the second user position acquisition unit that the gaze of the avatar of the second user is staying on the extended display target for a predetermined period of time, displays the extended display content together with the extended display target on a display unit of the second terminal.

In the communication system according to paragraph 5, detailed explanations of exhibits, etc. at real events and simulated events can be provided only to first and second users who are interested in the exhibits, etc. In addition, since the exhibits, etc. are not displayed to users who are not interested in the exhibits, such users do not feel bothered by unnecessary displays.

(Section 6)
A communication system according to paragraph 6 is a communication system according to any one of paragraphs 1 to 5,
the storage unit further stores data of a moving image to be projected in the real event space, a projection location of the moving image, and information on a timing of projecting the moving image;
The second display control unit causes a display unit of the second terminal to display, at the timing, an image obtained by projecting the moving image at a position in the virtual event space corresponding to the projection location.

In the communication system according to paragraph 6, video images can be viewed in real time in the virtual event space simultaneously with the real event space.

(Section 7)
A communication system according to paragraph 7 is a communication system according to any one of paragraphs 1 to 6,
When there are a plurality of the second terminals, the plurality of second terminals are divided into a plurality of groups, and a virtual event space is set for each group;
The first terminal display control unit causes only avatars of users of the second terminal who are classified into any one of the plurality of groups to be displayed on a display unit of the first terminal .

In the communication system according to paragraph 7, even if a large number of second users participate in a simulated event, the number of avatars displayed on the display unit of the first terminal and the display unit of the second terminal are limited and arranged in the virtual space, so that the first user and the second user do not feel bothered by the display of a large number of avatars.

(Section 8)
A communication system according to claim 8 is a communication system according to any one of claims 1 to 7,
moreover,
The device is provided with a user intermediation unit that receives a request from the second terminal to specify one of the first users and to connect with the first user, queries a first terminal owned by the first user as to whether or not the first user agrees to connect with a second user who is an owner of a second terminal, and sets up a link between the first terminal and the second terminal when an operation to grant permission is performed from the first terminal.
(Section 8)
In the communication system according to paragraph 8, a private connection can be formed between a second user who issues a request and a first user who accepts the request.

(Section 9)
A communication system according to claim 9, in the communication system according to claim 8,
The user intermediation unit further displays a chat screen that can be written on and viewed only from the first terminal and the second terminal for which the linkage is set up, on a display unit of the first terminal used by the first user and on a display unit of the second terminal used by the second user.

In the communication system according to paragraph 9, only the first and second users who have formed a connection can have private interactions through chats that can be written and viewed.

(Article 10)
A communication system according to claim 10, in the communication system according to claim 8 or 9,
The user interface unit displays the image acquired by the first terminal on a display unit of a second terminal that is set up to cooperate with the first terminal.

In the communication system according to paragraph 10, a second user who participates in a simulated event and forms a private connection with a first user can enjoy images captured by the first user in the real event space in real time.

1...communication system 10...server device 11...storage unit 12...user information storage unit 13...event data storage unit 21...user registration acceptance unit 22...participation acceptance unit 23...user mediation unit 30...first terminal 31...storage unit 32...image acquisition unit 33...first terminal position acquisition unit 34...first display control unit 35...expanded display unit 36...emotion image selection acceptance unit 37...emotion action selection acceptance unit 41...input unit 42...display unit 50...second terminal 51...storage unit 52...second user position acquisition unit 53...second display control unit 54...expanded display unit 55...emotion image selection acceptance unit 56...emotion action selection acceptance unit 61...input unit 62...display unit 71, 711...first user's avatar 72, 721...first 2. User avatar 73... Auxiliary screen 731... Emotion display button 7311... Icon 732... Action display button 74... Expanded display target 741... Expanded display 81... Chat button 811... Chat screen 812... OPEN button 813... TORIMAKI button 82... Start distribution button 83... Participant button 91... Chat screen 911... OPEN button 912... TORIMAKI button 913... DIRECT button 92... "Become TORIMAKI" button 93... Participant list display 931... First user name 932... Second user name 933... Chat button 934... Button 941... EXIT button 942... Reduce button 95... Small screen

Claims (11)

a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a second terminal used by the second user and equipped with an input unit and a display unit;
a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal;
a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information to place an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on a display unit of the second terminal ;
When there are a plurality of the second terminals, the plurality of second terminals are divided into a plurality of groups, and a virtual event space is set for each group;
The first display control unit causes only avatars of the users of the second terminal who are classified into any one of the plurality of groups to be displayed on the display unit of the first terminal.
A communication system characterized by : a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a second terminal used by the second user and equipped with an input unit and a display unit;
a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal;
a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information to place an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on a display unit of the second terminal ;
a user intermediation unit that receives a request from the second terminal to specify one of the first users and to establish a connection with the first user, queries a first terminal owned by the first user as to whether or not the first user agrees to establish a connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of approval is performed from the first terminal;
Equipped with
When an operation for permitting a connection with a plurality of the second users is performed on the first terminal, the user intermediation unit sets up a link between the first terminal and the plurality of the second terminals corresponding to the plurality of second users, and displays a chat screen that can be written and viewed only from the first terminal and the plurality of second terminals for which the link has been set up on a display unit of the first terminal used by the first user and on display units of the plurality of second terminals.
A communication system characterized by : a storage unit that stores display data for displaying in a virtual event space a simulated event corresponding to a real event held in a real event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, display data of an avatar of a first user participating in the real event in the real event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
a first terminal used by the first user, the first terminal including an image acquisition unit that acquires an image of the first user and the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a second terminal used by the second user and equipped with an input unit and a display unit;
a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the position being set by an operation via the input unit;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal;
a second display control unit that converts the position of the first terminal acquired by the first terminal position acquisition unit into a position in the virtual event space based on the corresponding position information to place an avatar of the first user, and displays an image of the simulated event captured from a viewpoint corresponding to the position of the avatar of the second user on a display unit of the second terminal ;
a user intermediation unit that receives a request from the second terminal to specify one of the first users and to establish a connection with the first user, queries a first terminal owned by the first user as to whether or not the first user agrees to establish a connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of approval is performed from the first terminal;
Equipped with
The user intermediation unit displays an image of the first user acquired by the first terminal on a display unit of a second terminal that is set up to cooperate with the first terminal.
A communication system characterized by : the storage unit further stores reference position information in the real event space,
The communication system according to any one of claims 1 to 3, characterized in that the first terminal position acquisition unit acquires information on the position of the first terminal by comparing an image of the real event space acquired by the image acquisition unit with the reference position information. the storage unit further stores data of a plurality of emotion images, each of which is an image expressing an emotion of a user;
moreover,
an emotion image selection receiving unit that receives an operation to select one of the emotion images in one of the first terminal and the second terminal;
and an emotion image display unit that displays the selected emotion image together with an avatar of a user of a terminal whose operation has been accepted by the emotion image selection accepting unit, on the other display unit of the first terminal and the second terminal. the storage unit further stores data of emotional actions, each of which is an avatar's action expressing an emotion of the user;
moreover,
an emotional action selection receiving unit that receives an operation to select one of the emotional actions in one of the first terminal and the second terminal;
and an emotional action performing unit that displays, on the other display unit of the first terminal and the second terminal, an image in which an avatar of a user of a terminal whose operation has been accepted by the emotional action selection accepting unit has performed the selected emotional action. the storage unit further stores information on augmented display objects to be arranged in each of the real event space and the virtual event space, and augmented display content associated with the augmented display objects;
moreover,
a display unit that generates image data for displaying the extended display content together with the extended display target on a display unit of the first terminal when it is determined based on position information acquired by the first terminal position acquisition unit that the gaze of the user of the first terminal is staying on the extended display target for a predetermined period of time, and displays the extended display content together with the extended display target on a display unit of the second terminal when it is determined based on position information acquired by the second user position acquisition unit that the gaze of the avatar of the second user is staying on the extended display target for a predetermined period of time. the storage unit further stores data of a moving image to be projected in the real event space, a projection location of the moving image, and information on a timing of projecting the moving image;
The communication system according to any one of claims 1 to 3, characterized in that the second display control unit, at the timing, displays an image of the moving image projected at a position corresponding to the projection location in the virtual event space on the display unit of the second terminal. a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit;
a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit,
The first terminal,
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is placed at the converted position on a display unit of the first terminal ;
When there are a plurality of the second terminals, the plurality of second terminals are divided into a plurality of groups, and a virtual event space is set for each group;
The first display control unit causes only avatars of the users of the second terminal who are classified into any one of the plurality of groups to be displayed on the display unit of the first terminal.
A program characterized by : a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit;
a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit,
The first terminal,
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal ;
a user intermediation unit that accepts a request from the second terminal to specify one of the first users and to establish a connection with the first user, inquires of a first terminal owned by the first user as to whether or not the first terminal accepts the connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of acceptance is performed from the first terminal.
Operate as
When an operation for permitting a connection with a plurality of the second users is performed on the first terminal, the user intermediation unit sets up a link between the first terminal and the plurality of the second terminals corresponding to the plurality of second users, and displays a chat screen that can be written and viewed only from the first terminal and the plurality of second terminals for which the link has been set up on a display unit of the first terminal used by the first user and on display units of the plurality of second terminals.
A program characterized by : a first terminal used by a first user who is a participant of a real event held in a real event space, the first terminal including an image acquisition unit that acquires an image of the real event space and the first user captured from a viewpoint corresponding to a position of the first user, and a display unit that displays the image acquired by the image acquisition unit;
a second terminal used by a second user who is a participant of a simulated event corresponding to the real event held in a virtual event space, the second terminal including an input unit and a display unit;
a server device provided to be capable of communicating with the first terminal and the second terminal, the server device having a storage unit storing display data for displaying the simulated event in the virtual event space, corresponding position information that associates a position in the real event space with a position in the virtual event space, and display data of an avatar of a second user participating in the simulated event in the virtual event space;
In a communication system having a second user position acquisition unit that acquires information on a position of an avatar of the second user in the virtual event space, the second terminal or the server device being set by an operation through the input unit,
The first terminal,
a first terminal position acquisition unit that acquires information about a position of the first terminal in the real event space;
a first display control unit that converts the position of the avatar of the second user acquired by the second user position acquisition unit into a position in the real event space based on the corresponding position information, and displays an image in which the avatar of the second user is arranged at the converted position on a display unit of the first terminal ;
a user intermediation unit that accepts a request from the second terminal to specify one of the first users and to establish a connection with the first user, inquires of a first terminal owned by the first user as to whether or not the first terminal accepts the connection with a second user who is an owner of the second terminal, and sets up a link between the first terminal and the second terminal when an operation of acceptance is performed from the first terminal.
Operate as
The user intermediation unit displays an image of the first user acquired by the first terminal on a display unit of a second terminal that is set up to cooperate with the first terminal.
A program characterized by :

Images