JP7630715B2 - Remote experience system, information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to a remote experience system, an information processing device, an information processing method, and a program that provide a user with information about a remote location.

Systems that allow users to participate in events at remote locations, such as distance education and remote conferences, are increasingly being introduced. For example, there is known technology in which a teacher is present at an educational site and transmits information in one direction to a user (participant) located at a remote location.

There is also known a technology that simulates a face-to-face conversation between distant participants by generating a virtual space and displaying virtual participants along with the virtual space. For example, Patent Literature 1 discloses a technology related to a remote conference that includes a mixture of in-house workers who are in the office and remote workers who work remotely. In the technology described in Patent Literature 1, a virtual room is displayed on the terminals of the in-house workers and remote workers, and the in-house workers and remote workers are displayed as virtual participants in the virtual room, and the movements of the in-house workers and remote workers are reflected in the movements of the virtual workers, enhancing the sense of realism.

Patent No. 6888854

On the other hand, depending on the content of the event, it may be desirable to experience various conditions in a real environment rather than a virtual space. The above-mentioned Patent Document 1 discloses the participation of virtual participants in a virtual space, but does not disclose a technology that allows remote participants in an event using a real environment to experience the real environment.

The present disclosure has been made in consideration of the above, and aims to provide a remote experience system that enables remote participants in an event using a real environment to experience the real environment.

In order to solve the above-mentioned problems and achieve the object, a remote experience system according to the present disclosure provides a remote experience system that allows a plurality of remote participants to experience an event using a real environment in which real participants are participating, the remote experience system including: a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment; a detection device that detects the real avatar and the real participant; a virtual space generation information processing device that generates a virtual space representing the real environment including the real avatar detected by the detection device and the real participant; a virtual avatar-reflected virtual space image generating means that generates a virtual avatar-reflected virtual space image representing a second remote participant among the plurality of remote participants but reflecting a virtual avatar that does not exist in the real environment in the virtual space; and a providing means that provides the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space generating means to the real participant, the first remote participant, and the second remote participant .

The remote experience system disclosed herein has the effect of enabling remote participants in an event that uses a real environment to experience the real environment.

FIG. 1 is a diagram showing a configuration example of a remote experience system according to an embodiment; FIG. 1 is a diagram showing a schematic diagram of remote learning using a remote experience system according to an embodiment of the present invention; A flowchart showing an example of a control procedure of a device in an information processing apparatus according to an embodiment. A flowchart showing an example of a procedure for generating virtual information related to a virtual avatar in an information processing device according to an embodiment. A flowchart showing an example of a procedure for generating expression information in the information processing device according to the embodiment. FIG. 1 shows an example of a real participant, a real avatar user, and a virtual avatar user participating in an event. FIG. 1 is a diagram showing an example in which a real avatar user and a virtual avatar user participate in an event. A diagram showing an example of generating a virtual space corresponding to a virtual avatar. A diagram showing an example of an image synthesized with a virtual space. Diagram showing reduction of sensors in equipment FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes an information processing device according to an embodiment.

Below, the remote experience system, information processing device, information processing method and program relating to the embodiments are described in detail with reference to the drawings.

Embodiment
1 is a diagram showing a configuration example of a remote experience system according to an embodiment. The remote experience system 100 according to the present embodiment is used for an event using a real environment 10. The real environment 10 is, for example, a place where the event is held. The event may be, for example, remote education, remote experience, remote skill transfer, remote maintenance, remote training, experiential travel, healing, astronomy observation, on-site games, on-site golf, local sightseeing, boat trips, car trips, exploration, and the like, but is not limited to these.

The remote experience system 100 of this embodiment can provide three methods of participating in an event: participating as a real participant who actually exists in the real environment 10; participating using the device 1 as a real avatar that is an avatar of the user; and participating using a virtual avatar that is an avatar that has no physical presence in the real environment 10. Both the user who participates using a real avatar and the user who participates using a virtual avatar are virtual participants who do not exist in the real environment 10, and are remote participants.

When a real avatar is used, the device 1 can transmit information obtained from the real environment 10 to the user by the operation of the device 1, such as the device 1 moving in the real environment 10 or touching surrounding objects. On the other hand, when a real avatar is used, the device 1 needs to be prepared according to the number of participating users, and depending on the number of participants in the event, costs may be incurred. For this reason, the remote experience system 100 of this embodiment allows participation as a virtual avatar, so that the number of participants in the event can be increased while keeping costs down. Details of the real avatar and virtual avatar will be described later. Below, an example in which real participants, real avatars, and virtual avatars can coexist will be mainly described, but the remote experience system 100 may be constructed so that only real participants and real avatars, or only real avatars, can participate.

As shown in FIG. 1, the remote experience system 100 comprises equipment 1, sensors 3-1 to 3-3, and presentation devices 4-1, 4-2 installed in a real environment 10, participant motion detection devices 5-1, 5-2 and presentation devices 6-1, 6-2 installed at a participation location 7, which is a location where users participating in the event are present, and an information processing device 2.

Participant movement detection devices 5-1, 5-2 installed at the participation location 7 detect the movements of users participating in the event and transmit the detected information as movement information to the information processing device 2. Participant movement detection devices 5-1, 5-2 are set at the participation location 7 where at least a user (first user) participating using a real avatar is present and detect the movements of that user. Participant movement detection devices 5-1, 5-2 may also be set at the participation location 7 where a user (second user) participating using a virtual avatar is present and detect the movements of that user, as described below.

The participant motion detection devices 5-1 and 5-2 are, for example, positioning sensors, cameras, microphones, etc. The expression devices 6-1 and 6-2 receive expression information indicating the state of the real environment 10 from the information processing device 2, and convey the state of the real environment 10 to the user based on the expression information. The expression devices 6-1 and 6-2 are devices capable of expressing something that the user can recognize with at least one of the five senses, and can present an image to the user, for example. The expression devices 6-1 and 6-2 are, for example, displays, speakers, haptic devices, etc. The displays used as the expression devices 6-1 and 6-2 may be head-mounted displays, VR (Virtual Reality) goggles, terminals such as smartphones and personal computers, or cockpit displays that allow the user to move the display together with the user by boarding the device. The expression devices 6-1 and 6-2 may also be, for example, hologram displays, aerial projection displays, etc. Furthermore, the presentation devices 6-1 and 6-2 may include an olfactory presentation device, a blower, an air conditioner, and the like.

Hereinafter, when the participant motion detection devices 5-1 and 5-2 are shown without distinction, they will be referred to as participant motion detection device 5, and when the expression devices 6-1 and 6-2 are shown without distinction, they will be referred to as expression device 6. In FIG. 1, two participant motion detection devices 5 and two expression devices 6 are shown, but the number of each of the participant motion detection devices 5 and the expression devices 6 is not limited to the example shown in FIG. 1. In addition, in FIG. 1, one participation location 7 is shown, but when users participate from multiple different locations, a similar device is provided at each participation location 7. In addition, when multiple users participate in an event from the same participation location 7, the participant motion detection device 5 and the expression device 6 are basically provided for each user, but some may be shared. For example, when a hologram display, an aerial projection display, or the like is used as the expression device 6, the expression device 6 may transmit information to multiple users. In addition, the participant motion detection device 5 may detect the motions of multiple users.

The sensors 3-1 to 3-3 are real environment sensors that are installed in the real environment 10 and detect information about objects in the real environment 10. The objects include, for example, at least one of animals, plants, and air that exist in the real environment 10. The sensors 3-1 to 3-3 are sensors that detect the environment in the real environment 10, and are sensors that detect, for example, objects, sounds, temperature, wind speed, wind volume, humidity, illuminance, odors, and types of gas in the real environment 10. That is, the sensors 3-1 to 3-3 detect, for example, at least one of temperature, humidity, wind, and odor in the real environment 10. In addition, the detection of objects may be performed, for example, by capturing images as images, or by collecting sounds. That is, the sensors 3-1 to 3-3 may include a capturing device such as a camera or may include a microphone. Detection information indicating the detection results is transmitted to the information processing device 2. Hereinafter, when the sensors 3-1 to 3-3 are referred to individually without distinction, they will be referred to as sensor 3.

Device 1 is an avatar that represents the user participating in the event, and more specifically, is a real avatar that exists as an object. Device 1 may be, for example, a humanoid robot, a mobile robot having a manipulator and a head that mimics a human face, or a manipulator. As described below, device 1 operates based on control information received from information processing device 2. Device 1 includes a control information receiving unit 11, a drive control unit 12, a drive unit 13, sensors 14-1 and 14-2, and a detection information transmitting unit 15.

The control information receiving unit 11 receives control information from the information processing device 2 and outputs the received control information to the drive control unit 12. The drive control unit 12 controls the drive unit 13 based on the control information. The drive unit 13 includes one or more actuators. The actuators are, for example, physical actuators, moving devices, speakers, etc. that operate each joint in the device 1, such as a manipulator.

The sensors 14-1 and 14-2 are mounted on the device and detect information about objects in the real environment 10. The sensors 14-1 and 14-2 include a camera or other imaging device, a speaker, and the like, as with the sensor 3, but may detect at least one of the temperature, humidity, wind, and odor in the real environment 10. The sensors 14-1 and 14-2 may also detect haptics. The sensors 14-1 and 14-2 output detection information indicating the detection result to the detection information transmission unit 15. The detection information transmission unit 15 transmits the detection information to the information processing device 2. The sensors 14-1 and 14-2 are mounted on the device 1, so they can detect information according to the location of the device 1. Hereinafter, the sensors 14-1 and 14-2 will be referred to as sensor 14 when not being individually distinguished. For example, if the device 1 is provided with a part simulating a face and a camera is provided as the sensor 14 in the part corresponding to the eyes, the sensor 14 can acquire images that depend on the movement of the device 1 and the direction of the face. Although two sensors 14 are shown in FIG. 1, the number of sensors 14 is not limited to the example shown in FIG.

For example, device 1 may include a manipulator, and sensor 14 may include a force-tactile sensor that detects the force-tactile sensation of the hand of the manipulator. If the representation device 6 corresponding to the user using the real avatar includes a force-tactile device that transmits force-tactile sensations to the hand of the user using the real avatar, the force-tactile sensations detected by device 1 can be transmitted to the user using the real avatar, as described below.

The presentation devices 4-1 and 4-2 are devices that perform presentations to communicate the virtual environment superimposed on the real environment 10 to users participating in an event in the real environment 10. The presentation devices 4-1 and 4-2 are, for example, displays, speakers, etc. The displays used as the presentation devices 4-1 and 4-2 are similar to the displays used as the presentation device 6 described above, but only need to be able to present the parts that will become the virtual environment, as described below. Hereinafter, when the presentation devices 4-1 and 4-2 are referred to without being individually distinguished, they will be referred to as presentation device 4.

As described below, depending on the configuration of device 1 and the participation method of users participating in the event, a representation device 4 may not be provided in real environment 10. Also, depending on the configuration of device 1 and the participation method of users participating in the event, a sensor 3 may not be provided in real environment 10. Also, although three sensors 3 and two representation devices 4 are illustrated in FIG. 1, the number of sensors 3 and representation devices 4 is not limited to the example shown in FIG. 1. Also, although one device 1 is illustrated in FIG. 1, the number of devices 1 is not limited to the example shown in FIG. 1.

The information processing device 2 has a control information transmitting unit 21, an equipment control unit 22, an operation information receiving unit 23, a detection information receiving unit 24, an expression information generating unit 25 and an expression information transmitting unit 26.

The motion information receiving unit 23 receives motion information indicating the motions acquired by the participant motion detection device 5 from the participant motion detection device 5 at the participation location 7, and outputs the received motion information to the equipment control unit 22. Identification information capable of identifying the corresponding user is added to the motion information. This identification information may be user identification information determined, for example, when the user registers for participation in the event, or may be identification information of the participant motion detection device 5. When the identification information added to the motion information is identification information of the participant motion detection device 5, it is assumed that the identification information of the participant motion detection device 5 and the user identification information are associated with each other, for example, when the user registers for participation in the event.

The device control unit 22 uses the operation information to generate control information for making the device 1 perform an operation corresponding to the operation information. That is, the device control unit 22 uses the operation information to generate control information for controlling the device 1 so that the device 1, which is the real avatar of the user, performs an operation corresponding to the user's operation, and outputs the generated control information to the control information transmission unit 21. The control information is, for example, information indicating how to drive each actuator of the device 1. The device control unit 22 holds correspondence information indicating the correspondence between the device 1 and the user who uses the device 1 as a real avatar, and identifies the device 1 corresponding to the received operation information using the correspondence information and the identification information added to the operation information. The correspondence information may be determined, for example, when the user registers to participate in an event, or may be selected by the user at the start of the event. The function and position of each device 1 may be confirmed, and the user may select the device 1 to be the real avatar, and notify the information processing device 2 of the selected device 1 via the user's terminal (not shown). The control information transmission unit 21 transmits the control information received from the device control unit 22 to the device 1.

The detection information receiving unit 24 receives detection information from at least one of the sensor 14 and the sensor 3, and outputs the received detection information to the expression information generating unit 25. The expression information generating unit 25 uses the detection information to generate expression information indicating the content to be expressed by the expression device 6, and outputs the generated expression information to the expression information transmitting unit 26, which transmits the expression information to the expression device 6. Note that, similar to the correspondence between the device 1 and the user described above, the expression information generating unit 25 holds correspondence information indicating the correspondence between the expression device 6 and the user. In addition, the expression information generating unit 25 generates expression information to be transmitted to a user participating in the real environment 10, and outputs the generated expression information to the expression information transmitting unit 26, which transmits the expression information to the expression device 4.

The communication line between the information processing device 2 and each device may be a wireless line, a wired line, or a mixture of a wireless line and a wired line. Any communication method may be used for communication between the information processing device 2 and each device. For example, by performing communication according to Beyond 5G (5th Generation: 5th generation mobile communication system), which realizes large-capacity and low-latency transmission, information can be transmitted to the user with low latency, and the sense of realism can be enhanced. That is, at least one of the communication line between the information processing device 2 and the participant motion detection device 5, the communication line between the information processing device 2 and the sensor 14, the communication line between the information processing device 2 and the sensor 3, the communication line between the information processing device 2 and the device 1, the communication line between the information processing device 2 and the expression device 6, and the communication line between the information processing device 2 and the expression device 4 may be a Beyond 5G line.

Next, an application example of the remote experience system 100 of this embodiment will be described. FIG. 2 is a diagram showing a model of remote education using the remote experience system 100 of this embodiment. In the example shown in FIG. 2, for example, real participants 8-1 to 8-4 actually exist in a forest, which is the real environment 10. Also, in the forest, which is the real environment 10, there exists a device 1 used as a real avatar. Furthermore, there exists a user who participates in the real environment 10 as a virtual avatar 9, which does not have a physical form. The real participant 8-1 is, for example, a teacher or other organizer, and the real participants 8-2 to 8-4, the user who participates as the virtual avatar 9, and the user who participates using the device 1 as a real avatar are students. Hereinafter, when the real participants 8-1 to 8-4 are referred to without being individually distinguished, they will be referred to as real participants 8.

In the example shown in FIG. 2, the real environment 10 is a forest, and the presence of birds, animals, plants, etc. creates smells, air flows, and changes in sound caused by the movement of birds and animals. In addition, by touching the animals and plants present in the forest, participants can have experiences unique to the real environment 10. By conveying various conditions to the user in addition to images and sounds, it is possible to promote, for example, an understanding of the user's Umwelt. In this embodiment, a more realistic experience of the real environment 10 is provided even to users who participate in distance education remotely.

The motion of a user (hereinafter, also referred to as a real avatar user) participating using the device 1 as a real avatar is detected by the participant motion detection device 5 at the participation location 7 shown in FIG. 1, and motion information indicating the motion is transmitted to the information processing device 2. The information processing device 2 controls the motion of the device 1 using the motion information, so that the device 1 performs a motion according to the motion of the real avatar user. In addition, since the information processing device 2 causes the representation device 6 to represent information indicating the state of the real environment 10, the real avatar user can recognize, for example, plants and animals in the forest, which is the real environment 10, by images, sounds, and the like. Therefore, when the real avatar user performs a motion to touch a plant in the real environment 10, the device 1, which is the real avatar, can touch the plant in the real environment 10. The haptic sensation caused by touching the plant is detected by the sensor 14 in the device 1 and transmitted to the representation device 6, such as a haptic glove, via the information processing device 2. This allows the representation device 6 to transmit the haptic sensation to the user.

In addition, by using a sensor that detects air conditions such as temperature, wind speed, air volume, humidity, odor, and type of gas as the sensor 3, the real avatar user can experience not only images and sounds, but also the air conditions in the real environment 10. In this manner, in this embodiment, the user can participate using the device 1 present in the real environment 10 as a real avatar, so that remote participants in an event using the real environment 10 can more appropriately experience the real environment 10, and interaction with the real environment can be realized. By fusing the real environment 10 with a virtual object, the real avatar user can experience the real environment 10 with a sense of realism.

Note that, by using an odor reproduction device, an air conditioner, a blower, or the like as the presentation device 6 that presents the air state, the air state may be presented as being similar to the detection information, but the air state may be converted into at least one of visual information and auditory information. Similarly, the haptic detection information may be converted into at least one of visual information and auditory information.

In addition, with regard to device 1, which is a real avatar, device 1 may be transmitted as an image directly to the real avatar user and the user participating using a virtual avatar (hereinafter also referred to as a virtual avatar user), and the real participants may directly view device 1, or information processing device 2 may generate an image simulating the real avatar user and replace the device 1 portion with that image.

Although the virtual avatar does not have a physical presence in the real environment 10, the position of the virtual avatar in the real environment 10 is specified, and information simulating the presence of the virtual avatar user at that position is provided to the user. For example, by using a 360-degree free viewpoint camera, a microphone for realizing 360-degree stereophonic sound, or the like as the sensor 3, it is possible to provide the virtual avatar user with images and sounds according to the position and face direction of the virtual avatar. The position and face direction of the virtual avatar may be determined by the information processing device 2 based on the motion information detected by the participant motion detection device 5, as in the case of a real avatar user, or may be specified by the virtual avatar user using a terminal not shown. In addition, when a sensor for detecting the state of air, such as temperature, wind speed, air volume, humidity, odor, type of gas, or the like, is used as the sensor 3, the state of air in the real environment 10 is transmitted to the virtual avatar user, as in the case of a real avatar user, so that the virtual avatar user can also experience the state of air in the real environment 10.

Although the virtual avatar cannot interact with real objects in the real environment 10, such as by touching or moving the objects, the information processing device 2 may generate a virtual space around the virtual avatar based on an image of the real environment 10, and allow the virtual avatar to perform actions such as touching or moving objects in the virtual space. In this case, the type of object may be determined from the image of the real environment 10, and the determination result may be used to generate information indicating the haptic sensation obtained by touching the object, and the information may be transmitted to the representation device 6, thereby conveying the sensation of touching the object to the virtual avatar user. The initial values of the position and orientation of the virtual avatar in the real environment 10 may be specified by the virtual avatar user or may be determined in advance.

Furthermore, the information processing device 2 may generate an image simulating the virtual avatar user or a three-dimensional image of an arbitrary shape based on an image of the virtual avatar user, and transmit the image to the representation device 6 and the representation device 4 as representation information by superimposing the image on an image of the real environment 10. The virtual avatar may also be provided as an image to the real avatar user and the real participant, or the virtual avatar user may be able to select whether or not to display the virtual avatar. Similarly, for the real avatar, an image simulating the real avatar user or a three-dimensional image of an arbitrary shape may be generated and displayed at a location corresponding to the device 1, and the real avatar user may be able to select whether or not to display the image.

In addition, the above-mentioned virtual space generated for the virtual avatar user may be reflected in the representation information transmitted to the real avatar user so that the real avatar user can share information with the virtual avatar user. In addition, the virtual space may be represented by the representation device 4 so that the virtual space is superimposed on the real environment 10 that the real participants directly view so that the real participants can share information with the virtual avatar user.

Next, the operation of the information processing device 2 of this embodiment will be described. FIG. 3 is a flowchart showing an example of a control procedure for the device 1 in the information processing device 2 of this embodiment. The information processing device 2 acquires motion information (step S1). In detail, the motion information receiving unit 23 acquires motion information by receiving motion information from the participant motion detection device 5 that detects the motion of the real avatar user, and outputs the acquired motion information to the device control unit 22.

Next, the information processing device 2 generates control information for reflecting the motion information in the motion of the real avatar (step S2). In detail, the information processing device 2 uses the motion information to generate control information for causing the device 1, which is the real avatar of the real avatar user corresponding to the motion information, to perform the motion corresponding to the motion information, and outputs the generated control information to the control information transmission unit 21.

Next, the information processing device 2 transmits the control information (step S3). In detail, the control information transmitting unit 21 transmits the control information to the device 1. The above-described process is performed, for example, for each control period, so that the movement of the real avatar user is reflected in the device 1, which is the real avatar corresponding to the real avatar user.

Figure 4 is a flowchart showing an example of a procedure for generating virtual information related to a virtual avatar in the information processing device 2 of this embodiment. The information processing device 2 carries out step S1 in the same manner as the example shown in Figure 3. However, the movement information is movement information related to a virtual avatar user, and the movement information receiving unit 23 outputs the movement information to the expression information generating unit 25.

Next, the information processing device 2 generates virtual information corresponding to the virtual avatar reflecting the motion information (step S4). In detail, the expression information generating unit 25 uses the motion information to generate virtual information, which is expression information such as images and sounds corresponding to a virtual space including the virtual avatar of the virtual avatar user corresponding to the motion information. The above-described process is performed, for example, for each control period, to generate virtual information for expressing a virtual space reflecting the motion of the virtual avatar user. Note that the generation of the virtual avatar and the generation of information regarding the virtual space can be generated using technologies such as spatial reconstruction, MR (Mixed Reality), and AR (Augmented Reality), but the generation of the virtual avatar and the generation of information regarding the virtual space are not limited to these examples.

Figure 5 is a flowchart showing an example of a processing procedure for generating expression information in the information processing device 2 of this embodiment. The information processing device 2 acquires detection information by a sensor (step S11). In detail, the detection information receiving unit 24 acquires detection information by receiving detection information from at least one of the sensor 14 and the sensor 3, and outputs the acquired detection information to the expression information generating unit 25.

Next, the information processing device 2 generates the first synthesis information by synthesizing the virtual information with the detection information of the synthesis target among the detection information (step S12). In detail, the expression information generating unit 25 generates the first synthesis information by synthesizing the virtual information generated in step S4 shown in FIG. 4 with the detection information of the synthesis target among the detection information. The detection information of the synthesis target is detection information to be synthesized with the virtual information. For example, if the virtual information is an image, the detection information of the synthesis target is an image, and if the virtual information is an image and sound, the detection information of the synthesis target is an image and sound. Note that here, an example is described in which the image and sound are acquired by the sensor 14 mounted on the device 1, but if the image and sound are acquired by the sensor 3 such as a 360-degree free viewpoint camera, the expression information generating unit 25 uses the detection information to generate information indicating an image and sound corresponding to a virtual avatar or a real avatar, and synthesizes the virtual information with the generated information. In the case of multiple remote participants, the virtual information is synthesized with the corresponding detection information or the information generated using the detection information of the sensor 3 for each remote participant. Note that synthesis of the virtual information does not have to be performed.

Next, the information processing device 2 converts the detection information to be converted from among the detection information into another type of information, and generates the second composite information by synthesizing the converted information with the first composite information (step S13). In detail, the expression information generating unit 25 converts the detection information to be converted from among the detection information into another type of information, synthesizes the converted information with the first composite information to generate the second composite information, and outputs the second composite information to the expression information transmitting unit 26 as the expression information. For example, when converting the detection information of the state of the air, such as temperature and humidity, into visual information, the detection information of the state of the air becomes the detection information to be converted, and the other type of information is characters, images, etc. This conversion can be performed, for example, by learning in advance, by supervised machine learning using clinical trial results, the contents of the other type of information that a person can feel a similar state according to the value of the detection information to be converted, and inputting the detection information to be converted into the learned model, thereby obtaining the conversion result. The conversion method is not limited to this example. Examples of expressing the state of air with visual information, auditory information, etc. include, but are not limited to, generating audio information that exaggerates the sound of birds or insects moving, displaying a mirage-like image shaking to represent humidity, and displaying a clear edge to represent coldness. Similarly, information about taste may be converted into visual information, auditory information, etc. If there is no detection information to be converted, step S13 is not performed, and the first synthesis information is output to the expression information transmission unit 26 as expression information. If there are multiple remote participants, steps S12 and S13 are performed for each remote participant.

Next, the information processing device 2 transmits the second synthesis information and the detection information that is not a target for synthesis and conversion to the representation devices 6 of the remote participants (step S14). In detail, the representation information generation unit 25 transmits the second information and the detection information that is not a target for synthesis and conversion to the representation devices 6 of the remote participants, i.e., the representation devices 6 of the virtual avatar users and the real avatar users.

Next, the information processing device 2 transmits the virtual information to the representation device 4 of the real environment 10 (step S15). In detail, the representation information generating unit 25 transmits the virtual information to the representation device 4 of the real environment 10, that is, the representation device 4 that transmits information to the real participants. The information converted in step S13 and the virtual information may be combined and transmitted to the representation device 4 of the real participants. Since the real participants can directly feel the state of the air in the real environment 10, the information converted in step S13 does not need to be transmitted to the representation device 4, but the converted information is also transmitted to the real participants, so that the real participants can share information with the virtual avatar user and the real avatar user. The above-mentioned process is performed, for example, every control period, whereby information indicating the state of the real environment 10 is transmitted to the remote participants and virtual information is transmitted to the real participants.

In this case, the expression device 6 corresponding to the real avatar user (first user) is defined as the first expression device, and the expression device 6 corresponding to the virtual avatar user (second user) is defined as the second expression device. The first expression device only needs to be capable of expressing something that the first user can recognize with at least one of the five senses, and the second expression device only needs to be capable of presenting an image to at least the second user. The sensor 3 includes an imaging device that detects information about objects in the real environment 10 by imaging at least the real environment 10 in order to generate an image to be provided to the virtual avatar user. The expression information generating unit 25 generates first expression information to be transmitted to the first user (real avatar user) using the first detection information, which is the detection information received from the sensor 14, and generates second expression information including an image corresponding to a virtual viewpoint in the real environment 10 of the second user (virtual avatar user) using the second detection information received from the sensor 3. The expression information transmitting unit 26 transmits the first expression information to the first expression device and transmits the second expression information to the second expression device.

As described above, the expression information generating unit 25 may convert the first detection information indicating at least one of the detection results of temperature, humidity, wind, and odor in the real environment 10 into first expression information that can be recognized by at least one of vision and hearing. The expression information generating unit 25 may convert the second detection information indicating at least one of the detection results of temperature, humidity, wind, and odor in the real environment 10 into second expression information that can be recognized by at least one of vision and hearing. The sensor may include information on taste, and the expression information generating unit 25 may generate information on taste in the real environment 10. In this case, the expression device 4 and the expression device 6 include a taste expression device that reproduces taste. Alternatively, the information on taste may be converted into other types of methods such as visual and auditory.

In addition, the expression information generating unit 25 may generate the first expression information using the second detection information. That is, the expression information to be transmitted to the real avatar user may be generated using the information acquired by the sensor 3.

The expression information generating unit 25 may generate second expression information so that the virtual avatar is displayed at the virtual viewpoint, the participant motion detecting device 5 detects the motion of the virtual avatar user, and the expression information generating unit 25 virtually generates an image in which the object of the real environment 10 and the virtual avatar change in accordance with the motion of the virtual avatar user in the virtual space that is the range including the virtual viewpoint based on the motion information of the virtual avatar user and the second detection information (video data of the real environment 10), synthesizes virtual information indicating the generated image with the second expression information, and the expression information transmitting unit 26 may transmit the second expression information with the virtual information synthesized to the second expression device. The expression information generating unit 25 may synthesize the virtual information with the first expression information, and the expression information transmitting unit 26 may transmit the first expression information with the virtual information synthesized to the first expression device. Note that the virtual avatar may have only a virtual viewpoint, or may be capable of hearing and speaking in addition to the virtual viewpoint.

The expression information transmitting unit 26 may also transmit the virtual information to the expression device 4, which is a third expression device capable of presenting an image to a real participant (third user). The information processing device 2 may also be capable of setting whether information regarding the virtual avatar user and the real avatar user is to be made public or private. Information regarding the virtual avatar user and the real avatar user may include, but is not limited to, at least a portion of information that is frequently used for security purposes, such as face information, eye information, and voiceprint, and personal information, such as real name, age, and date of birth. In addition, if a user does not wish to make their actual face or voice public for privacy reasons, the face and voice may be changed and made public.

Next, an example of the participation form of participants will be described. FIG. 6 is a diagram showing an example in which real participants, real avatar users, and virtual avatar users participate in an event. In the example shown in FIG. 6, real participants 8-1 to 8-3 participate in the event in a real environment 10, and from the participation location 7-1, user 70-1, who is a real avatar user, participates in the event using device 1, which is a real avatar, and from the participation locations 7-2 and 7-3, users 70-2 and 70-3, who are virtual avatar users, participate in the event using virtual avatars 9-1 and 9-2, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of the participation locations 7-1 to 7-3. Note that if the virtual avatars 9-1 and 9-2 are fixed and not moved, the participant motion detection device 5 does not need to be provided at the users 70-2 and 70-3, who are virtual avatar users.

The device 1 includes sensors 14 for detecting information related to each of the senses of sight, hearing, smell, and touch, and includes a physical actuator, a moving device, and a speaker as actuators. The sensors 3-1 and 3-2 installed in the real environment 10 include cameras for acquiring images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2, and microphones for collecting sounds for generating sounds at the positions of the virtual avatars 9-1 and 9-2. Although two sensors 3 are illustrated in FIG. 6 for the sake of simplicity, there are generally multiple cameras for acquiring images for generating images from the viewpoints of the virtual avatars 9-1 and 9-2, and multiple microphones for collecting sounds for generating sounds at the positions of the virtual avatars 9-1 and 9-2. At least one of the sensors 3 and 14 may further include a sensor for detecting odors, temperature, wind speed, wind volume, humidity, and types of gas in the real environment 10.

In the example shown in FIG. 6, all information to be transmitted to user 70-1, who is a real avatar user, is detected by a sensor 14 mounted on device 1. As described above, device 1 operates in response to the movement of the real avatar user, and, for example, the field of view of sensor 14, such as a camera that obtains visual information in device 1, is changed, and an image of the real environment 10 corresponding to the movement of the real avatar user is displayed by the representation device 6. The same applies to sounds, smells, etc., and information corresponding to sounds and smells in the real environment 10 is transmitted to the real avatar user as device 1 moves. Furthermore, when device 1 comes into contact with an actual object in the real environment 10 in response to the movement of the real avatar user, information related to haptics is detected by sensor 14 and transmitted to the real avatar user. This allows the real avatar user to recognize the real environment 10 with a sense of realism.

Furthermore, images from the viewpoints of the virtual avatars 9-1, 9-2, sounds from the virtual avatars 9-1, 9-2, etc. are transmitted to the virtual avatar users 70-2, 70-3, respectively, using the detection information acquired by the sensor 3. Furthermore, if a sensor that detects odors is provided as the sensor 3, the information processing device 2 can transmit information indicating odors to the virtual avatar users 70-2, 70-3 via the representation device 6, using the detection information from the sensor 3.

If the real participants 8-1 to 8-3 can see the device 1 as it is, but do not need to see the virtual avatars 9-1 and 9-2, the representation device 4 does not need to be provided in the real environment 10. As described above, if an image corresponding to a real avatar is projected onto the location where the device 1 is present, and an image corresponding to a location corresponding to the virtual avatars 9-1 and 9-2 is projected, a representation device 4 for projecting these is provided. Similarly, if the real participants 8-1 to 8-3 are able to recognize sounds corresponding to the virtual avatars 9-1 and 9-2, the sounds generated by the users 70-2 and 70-3, who are virtual avatar users, are transmitted to the real participants by the representation device 4 via the representation device 4.

Figure 7 is a diagram showing an example in which real avatar users and virtual avatar users participate in an event. In the example shown in Figure 7, real avatar users 70-1, 70-4, and 70-5 participate in the event from participation locations 7-1, 7-4, and 7-5 using devices 1-1 to 1-3, which are their corresponding real avatars, and virtual avatar users 70-2 and 70-3 participate in the event from participation locations 7-2 and 7-3 using virtual avatars 9-1 and 9-2, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of participation locations 7-1 to 7-5. Devices 1-1 to 1-3 are each the device 1 described above.

The configuration of each of devices 1-1 to 1-3 is similar to device 1 shown in Fig. 6. The method of transmitting information to real avatar users and virtual avatar users in the example shown in Fig. 7 is also similar to the example shown in Fig. 6. In the example shown in Fig. 7, since there are no real participants 8, a representation device 4 does not need to be provided in the real environment 10.

Figure 8 is a diagram showing an example of generating a virtual space corresponding to a virtual avatar. In the example shown in Figure 8, similar to the example shown in Figure 6, real participants 8-1 to 8-3 participate in an event in a real environment 10, and from participation location 7-1, user 70-1, who is a real avatar user, participates in the event using device 1, which is a real avatar, and from participation locations 7-2 and 7-3, users 70-2 and 70-3, who are virtual avatar users, participate in the event using virtual avatars 9-1 and 9-2, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of participation locations 7-1 to 7-3.

8, the information processing device 2 acquires motion information from a participant motion detection device 5 corresponding to user 70-3, who is a virtual avatar user, and generates a virtual space 90 around the virtual avatar 9-2 using the acquired information. The information processing device 2 then synthesizes the virtual information indicating the virtual space 90 with video and sound from the viewpoint of the virtual avatar 9-2, which has been generated using the detection information from the sensor 3, and transmits the synthesized information as expression information to the expression device 6 corresponding to user 70-3, who is a virtual avatar user. The synthesized information is also transmitted to user 70-1, who is a real avatar user, and user 70-2, who is a virtual avatar user.

FIG. 9 is a diagram showing an example of an image in which a virtual space is synthesized. In the example shown in FIG. 9, for example, an image 200 of the real environment 10 from the viewpoint of a user 70-2 who is a virtual avatar user is synthesized with an image 201 of a virtual space corresponding to a virtual avatar 9-2. The image 200 is a real image based on information captured by the sensor 3, while the image 201 is information showing a virtual space generated based on information captured by the sensor 3. The image 201 reflects the movement of the virtual avatar 9-2 in the virtual space. Therefore, when the virtual avatar 9-2 touches an object in the virtual space, the object also changes in the virtual space. Similarly, the image in which the virtual space is synthesized is transmitted to a user 70-3 who is a virtual avatar user corresponding to the virtual avatar 9-2, so that the user 70-3 who is a virtual avatar user can experience the same sensation as when touching an object in the real environment 10.

In addition, by transmitting virtual information to the representation devices 4 corresponding to the actual participants 8-1 to 8-3, the actual participants 8-1 to 8-3 can visually see the projected virtual space superimposed on the actual real environment 10.

Figure 10 is a diagram showing the reduction of sensors in devices. In the example shown in Figure 10, similar to the example shown in Figure 6, real participants 8-1 to 8-3 participate in an event in a real environment 10, and from participation location 7-1, user 70-1, who is a real avatar user, participates in the event using device 1, which is a real avatar, and from participation locations 7-2 and 7-3, users 70-2 and 70-3, who are virtual avatar users, participate in the event using virtual avatars 9-1 and 9-2, respectively. Although not shown in the figure, a participant motion detection device 5 and a representation device 6 are provided at each of participation locations 7-1 to 7-3.

In the example shown in FIG. 10, the sensors 3-1 to 3-4 include a camera for acquiring a 360-degree free viewpoint image and a microphone for reproducing a sound according to a position. The sensors 3-1 to 3-4 are used to generate information indicating an image of the real environment 10 from the viewpoint of the users 70-2 and 70-3 who are virtual avatar users and a sound at the positions of the users 70-2 and 70-3 who are virtual avatar users, and are also used to generate information indicating an image of the real environment 10 from the viewpoint of the real avatar and a sound at the position of the real avatar. As a result, the sensor 14 provided in the device 1 does not need to include a sensor for detecting visual and auditory information. As shown in FIG. 10, by generating information about the real avatar using the sensor 3, the configuration of the device 1 can be simplified compared to the example shown in FIG. 6. In the example shown in FIG. 10, visual and auditory information about the real avatar is generated from the detection information of the sensor 3, but this is not limited thereto. One of the visual and auditory information may be generated from the detection information of the sensor 3, and the other may be generated from the detection information of the sensor 14. Furthermore, by using the sensor 3 that detects odors, the number of sensors that detect information related to the sense of smell among the sensors 14 included in the device 1 may be reduced.

Next, the hardware configuration of the information processing device 2 of this embodiment will be described. In the information processing device 2 of this embodiment, a computer program describing the processing in the information processing device 2 is executed on the computer system, so that the computer system functions as the information processing device 2. Figure 11 is a diagram showing an example of the configuration of a computer system that realizes the information processing device 2 of this embodiment. As shown in Figure 11, this computer system comprises a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

In FIG. 11, the control unit 101 is, for example, a processor such as a CPU (Central Processing Unit), and executes a program in which the processing in the information processing device 2 of this embodiment is described. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 is, for example, composed of a keyboard, a mouse, etc., and is used by the user of the computer system to input various information. The storage unit 103 includes various memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory) and a storage device such as a hard disk, and stores the programs to be executed by the control unit 101, necessary data obtained in the process of processing, etc. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is composed of a display, an LCD (Liquid Crystal Display Panel), etc., and displays various screens to the user of the computer system. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is a printer, a speaker, etc. Note that Fig. 11 is just an example, and the configuration of the computer system is not limited to the example of Fig. 11.

Here, an example of the operation of the computer system until the program of this embodiment is in a state where it can be executed will be described. In a computer system having the above-mentioned configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 103 is stored in the main memory area of the storage unit 103. In this state, the control unit 101 executes processing as the information processing device 2 of this embodiment in accordance with the program stored in the storage unit 103.

In the above explanation, a program describing the processing in the information processing device 2 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this. Depending on the configuration of the computer system and the capacity of the program to be provided, for example, a program provided via a transmission medium such as the Internet via the communication unit 105 may be used.

The program of this embodiment causes a computer system to execute, for example, the steps of receiving operation information indicating the operation of a first user participating in an event at a location away from the real environment 10 in which the event is held, using the operation information to generate control information for causing an equipment installed in the real environment 10 to perform an operation according to the operation information, transmitting the control information to the equipment, receiving first detection information detected by an equipment-mounted sensor from an equipment-mounted sensor that is mounted on the equipment and detects information about an object in the real environment 10, and receiving second detection information detected by a real environment sensor from a real environment sensor that is installed in the real environment 10 and detects information about an object in the real environment 10. The program of this embodiment further causes the computer system to execute the following steps: generating first expression information to be transmitted to a first user using the first detection information; generating second expression information using the second detection information, the second expression information including an image corresponding to a virtual viewpoint in the real environment 10 of a second user participating in the event from a location remote from the real environment 10; and transmitting the first expression information to a first expression device capable of producing an expression that the first user can recognize with at least one of the five senses, and transmitting the second expression information to a second expression device capable of presenting an image to the second user.

The device control unit 22 and the expression information generation unit 25 shown in FIG. 1 are realized by the control unit 101 shown in FIG. 11 executing the computer program stored in the storage unit 103 shown in FIG. 11. The storage unit 103 shown in FIG. 11 is also used to realize the device control unit 22 and the expression information generation unit 25 shown in FIG. 1. The control information transmission unit 21, the operation information reception unit 23, the detection information reception unit 24, and the expression information transmission unit 26 shown in FIG. 1 are realized by the communication unit 105 shown in FIG. 11. The information processing device 2 may also be realized by multiple computer systems. For example, the information processing device 2 may be realized by a cloud computer system. Also, some of the functions of the information processing device 2 may be realized by another device provided separately from the information processing device 2. The other device may be provided in the real environment 10, near the location of the user participating in the event, or in another location.

As described above, in this embodiment, in an event using the real environment 10, the physical device 1 is used as a real avatar to realize remote user participation in the event. This allows remote participants in an event using the real environment 10 to experience the real environment 10 more appropriately. In addition, by using both participation by a real avatar using the device 1 and participation by a virtual avatar, even if the number of participants in the event increases, it is possible to flexibly respond with reduced costs. In addition, by transmitting information indicating the air condition of the real environment 10 to the remote participants, it is possible to allow the remote participants to experience a condition closer to the real environment 10. In addition, by generating information corresponding to the real avatar using detection information from the sensor 3 used to provide information corresponding to the virtual avatar to the virtual avatar user, the configuration of the device 1 can be simplified and costs can be reduced.

The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. Also, parts of the configurations may be omitted or modified without departing from the spirit of the invention.

1, 1-1 to 1-3 Equipment, 2 Information processing device, 3-1 to 3-4, 14-1, 14-2 Sensor, 4-1, 4-2, 6-1, 6-2 Expression device, 5-1, 5-2 Participant motion detection device, 7, 7-1 to 7-5 Participation location, 8-1 to 8-4 Actual participant, 9, 9-1, 9-2 Virtual avatar, 10 Actual environment, 11 Control information receiving unit, 12 Drive control unit, 13 Drive unit, 15 Detection information transmitting unit, 21 Control information transmitting unit, 22 Equipment control unit, 23 Motion information receiving unit, 24 Detection information receiving unit, 25 Expression information generating unit, 26 Expression information transmitting unit, 100 Remote experience system.

Claims (18)

A remote experience system that allows multiple remote participants to experience an event using a real environment in which real participants are also participating,
a real avatar representing a first remote participant of the plurality of remote participants and disposed in the real environment;
a detection device for detecting the real avatar and the real participant;
a virtual space generation information processing device that generates a virtual space representing the real environment including the real avatar detected by the detection device and the real participant; and
a virtual avatar-reflected virtual space image generating means for generating a virtual avatar-reflected virtual space image in which a virtual avatar that represents a second remote participant among the plurality of remote participants but does not exist in the real environment is reflected in the virtual space;
providing means for providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating means to the live participants, the first remote participant, and the second remote participant;
A remote experience system comprising: a device that is the real avatar and is installed in the real environment and has an on-device sensor that detects information about an object in the real environment;
a real environment sensor that is installed in the real environment and detects information about an object in the real environment;
a participant motion detector for detecting a motion of the first remote participant;
a first expression device capable of expressing something that the first remote participant can perceive with at least one of the five senses;
a second presentation device capable of presenting the virtual avatar-reflecting virtual space image to the second remote participant;
Equipped with
the real environment sensor includes an imaging device that detects information about an object in the real environment by imaging the real environment;
The virtual space generation information processing device comprises:
a motion information receiving unit that receives, from the participant motion detection device, motion information indicating the motion acquired by the participant motion detection device;
a device control unit that generates control information for causing the device to perform an operation corresponding to the operation information, using the operation information;
a control information transmitting unit that transmits the control information to the device;
a detection information receiving unit that receives, from the device-mounted sensor, first detection information detected by the device-mounted sensor, and receives, from the real environment sensor, second detection information detected by the real environment sensor;
an expression information generating unit that generates first expression information to be transmitted to the first remote participant using the first detection information, and generates second expression information including a virtual avatar-reflecting virtual space image corresponding to a virtual viewpoint in the real environment of the second remote participant using the second detection information;
an expression information transmitting unit that transmits the first expression information to the first expression device and transmits the second expression information to the second expression device;
2. The remote experience system according to claim 1, further comprising: The remote experience system according to claim 2, characterized in that the object includes at least one of animals, plants, and air present in the real environment. The remote experience system according to claim 3, characterized in that the on-board sensor detects at least one of temperature, humidity, wind, and odor in the real environment. The remote experience system according to claim 4, characterized in that the expression information generation unit converts the first detection information, which indicates a detection result of at least one of temperature, humidity, wind, and smell in the real environment, into the first expression information that can be recognized by at least one of sight and hearing. The remote experience system according to claim 3 or 4, characterized in that the real environment sensor detects at least one of temperature, humidity, wind, and odor in the real environment. The remote experience system according to claim 4, characterized in that the expression information generation unit converts the second detection information, which indicates a detection result of at least one of temperature, humidity, wind, and smell in the real environment, into the second expression information that can be recognized by at least one of sight and hearing. The remote experience system according to any one of claims 2 to 6, characterized in that the expression information generation unit generates the first expression information using the second detection information. The instrument includes a manipulator;
The device-mounted sensor includes a force tactile sensor that detects the force and tactile sensation of a hand of the manipulator,
5. The remote experience system of claim 2, wherein the first presentation device includes a haptic device for transmitting haptic sensations to a hand of the first remote participant. The remote experience system according to any one of claims 2 to 4, characterized in that the expression information generation unit generates the second expression information so that the virtual avatar corresponding to the second remote participant is displayed at the virtual viewpoint. a participant motion detection device for detecting a motion of the second remote participant;
Equipped with
the motion information receiving unit receives, from the participant motion detection device that detects the motion of the second remote participant, motion information indicating the motion of the second remote participant acquired by the participant motion detection device;
the expression information generation unit virtually generates an image in which the object in the real environment and the virtual avatar in a virtual space that is a range including the virtual viewpoint change in accordance with the movement of the second remote participant based on the movement information of the second remote participant and the second detection information, and synthesizes virtual information indicating the generated image with the second expression information;
The remote experience system according to claim 10 , wherein the expression information transmitting unit transmits the second expression information in which the virtual information is synthesized to the second expression device. The expression information generating unit combines the virtual information with the first expression information,
The remote experience system according to claim 11 , wherein the expression information transmitting unit transmits the first expression information in which the virtual information is synthesized to the first expression device. a third presentation device capable of presenting the virtual space image reflecting the virtual avatar to the physical participants;
Equipped with
13. The remote experience system according to claim 11, wherein the expression information transmitting unit transmits the virtual avatar-reflecting virtual space image including the virtual information to the third expression device. A remote experience system according to any one of claims 2 to 13, characterized in that at least one of the communication lines between the virtual space generation information processing device and the participant motion detection device, the communication line between the virtual space generation information processing device and the device-mounted sensor, the communication line between the virtual space generation information processing device and the real environment sensor, the communication line between the virtual space generation information processing device and the device, the communication line between the virtual space generation information processing device and the first representation device, and the communication line between the virtual space generation information processing device and the second representation device is a Beyond 5G line. The remote experience system described in any one of claims 1 to 14, characterized in that the virtual space generation information processing device can set information regarding the first remote participant and the second remote participant to be public or private. An information processing device in a remote experience system that allows a plurality of remote participants to experience an event using a real environment in which real participants are participating, comprising:
a virtual space generation information processing device that generates a virtual space representing the real environment including a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment, and the real participant;
a virtual avatar-reflected virtual space image generating means for generating a virtual avatar-reflected virtual space image in which a virtual avatar that represents a second remote participant among the plurality of remote participants but does not exist in the real environment is reflected in the virtual space;
providing means for providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating means to the live participants, the first remote participant, and the second remote participant;
An information processing device comprising: An information processing method for processing information using a remote experience system that allows a plurality of remote participants to experience an event using a real environment in which real participants are participating, comprising:
a virtual space generating step of generating a virtual space representing the real environment including a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment, and the real participant;
a virtual avatar-reflecting virtual space image generating step of generating a virtual avatar-reflecting virtual space image in which a virtual avatar representing a second remote participant among the plurality of remote participants but not present in the real environment is reflected in the virtual space generated by the virtual space generating step;
a providing step of providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating step to the live participant, the first remote participant, and the second remote participant;
An information processing method comprising: A computer system for processing information using a remote experience system that allows multiple remote participants to experience an event using a real environment in which real participants are participating,
a virtual space generating step of generating a virtual space representing the real environment including a real avatar representing a first remote participant among the plurality of remote participants and disposed in the real environment, and the real participant;
a virtual avatar-reflecting virtual space image generating step of generating a virtual avatar-reflecting virtual space image in which a virtual avatar representing a second remote participant among the plurality of remote participants but not present in the real environment is reflected in the virtual space generated by the virtual space generating step;
a providing step of providing the virtual avatar-reflected virtual space image generated by the virtual avatar-reflected virtual space image generating step to the live participant, the first remote participant, and the second remote participant;
A program characterized by executing the above.

Images