JP7645667B2 - VIRTUAL EXPERIENCE PROVIDING DEVICE, VIRTUAL EXPERIENCE PROVIDING SYSTEM, VIRTUAL EXPERIENCE PROVIDING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program.

Conventionally, there is an experience device that allows a person considering the contents of a trip, including the travel destination and travel plan, to share a virtual travel experience (for example, Patent Document 1).

JP 2018-155887 A

The above-mentioned experience device allows the user to share a virtual experience of traveling with a companion at the destination. However, it has been difficult to give the user a sense of actually traveling with the companion on-site during the virtual experience.

The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a virtual experience providing device, a virtual experience providing system, a virtual experience providing method, and a program that can enhance the sense of reality of a person taking a virtual travel experience traveling with a companion as if they were traveling together with the companion.

The virtual experience providing device, the virtual experience providing system, the virtual experience providing method, and the program according to the present invention employ the following configuration.
(1): A virtual experience providing device according to one embodiment of the present invention is a virtual experience providing device for providing a second user with a virtual travel experience of traveling together with a first user, and includes: an image generation unit that generates virtual reality images based on local images captured by an imaging device present around the first user at the travel destination; and a display control unit that displays the virtual reality images on a display device, wherein the image generation unit generates the virtual reality images in which the first user is included in the second user's field of view.

(2): In the aspect (1), an operation content receiving unit that acquires information indicating the content of an operation performed by the second user on an operation unit;
The present invention further includes an adjustment unit that adjusts the distance between the first user and the second user in the virtual reality image based on information acquired by the operation content receiving unit.

(3): In the aspect of (2) above, the image generating unit generates a virtual reality image in response to the movement of a moving object operated by the second user, and the display control unit turns the virtual reality image into a still image while the moving object is stopped.

(4): In any of the above aspects (1) to (3), the image generating unit generates the virtual reality image including a gaze image showing at least one of the first user's gaze or gaze point.

(5): A virtual experience providing system according to one embodiment of the present invention is a virtual experience providing system including any one of the virtual experience providing devices (1) to (4) above, a moving body that the second user boards during the virtual travel experience, and a display device that displays the virtual reality image.

(6): In the above aspect (5), the present invention further includes at least one of a first movable device that can be contacted by the first user or a second movable device that can be contacted by the second user, and a transmitting/receiving device that transmits/receives at least one of first motion information that corresponds to the motion of the first user and is transmitted to the second movable device or second motion information that corresponds to the motion of the second user and is transmitted to the first movable device.

(7): A virtual experience providing method according to one aspect of the present invention is a virtual experience providing method in which a computer of a virtual experience providing device for providing a second user with a virtual travel experience of traveling together with a first user generates virtual reality images based on local images captured by an imaging device present around the first user at a travel destination, and displays the virtual reality images on a display device, and the computer generates the virtual reality images in which the first user is included in the field of view of the second user.

(8): A program according to one aspect of the present invention is a program that causes a computer of a virtual experience providing device for providing a second user with a virtual travel experience traveling together with a first user to generate virtual reality images based on on-site images captured by an imaging device present around the first user at the travel destination, and to execute a process of displaying the virtual reality images on a display device, and causes the computer to generate the virtual reality images in which the first user is included in the field of view of the second user.

Based on (1) to (8), it is possible to enhance the sense of reality that a person taking part in a virtual travel experience traveling with a companion is experiencing as if they are traveling together with the companion.

1 is a diagram illustrating an example of a configuration of a virtual experience providing system 1 according to a first embodiment. FIG. 2 is a perspective view showing a moving body 210. 1 is a diagram showing an overhead view of a participant P2 using the virtual experience facility 200. FIG. A figure showing an example of a first virtual reality image GA1 displayed on display device 232. 10 is a flowchart showing an example of a process of each device 100 at a travel destination. 13 is a flowchart showing an example of processing of the virtual experience providing device 300. A sequence diagram of the virtual experience providing system 1 regarding the first haptic 130 and the second haptic 40. A figure showing an example of the configuration of a virtual experience providing system 1 of a second embodiment. FIG. 2 is a diagram showing an example of a landscape image GC1. A figure showing an example of a second virtual reality image GA2 generated based on the landscape image GC1. 13 is a flowchart showing a part of an example of processing of the virtual experience providing device 300.

Below, with reference to the drawings, an embodiment of the virtual experience providing device, virtual experience providing system, virtual experience providing method, and program of the present invention will be described. The virtual experience providing system is, for example, a system that provides a virtual travel experience of traveling with a companion to a travel destination using a virtual experience facility where a virtual experience can be had. The virtual experience providing system provides the user with a virtual travel experience traveling with a companion, such as a celebrity who would be difficult to travel with in reality, a guide who is knowledgeable about the geography or history of the travel destination, or a friend who was unable to go on a trip that was supposed to accompany the friend, and so the friend ends up traveling alone. The companion may be a person other than a real person, such as a historical figure or character, for example.

First Embodiment
FIG. 1 is a diagram showing an example of the configuration of a virtual experience providing system 1 of the first embodiment. The virtual experience providing system 1 includes, for example, a travel destination device 100, a virtual experience facility 200, and a virtual experience providing device 300, which are provided at a travel destination. The travel destination device 100 includes a communication device 110, a drone 120, and a first haptic 130. At the travel destination, a companion P1 experiences a real trip, for example, on foot, and at the virtual experience facility 200, a user P2 experiences a virtual trip in which the companion P1 travels to the travel destination together with the companion P1. The companion P1 is an example of a first user. The user P2 is an example of a second user.

The communication device 110 is, for example, a wireless communication module for connecting to a network or for directly communicating with other terminal devices, such as the virtual experience providing device 300 of the virtual experience facility 200. The communication device 110 performs wireless communication based on Wi-Fi, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), or other communication standards. The communication device 110 is capable of communicating with the virtual experience facility 200 via the network NW.

The drone 120 is an unmanned aerial vehicle. The drone 120 flies by automatic operation. The drone 120 may also fly by remote control. The drone 120 flies around the flight control device and the companion P1. The drone 120 is equipped with a camera 122. The camera 122 is present around the companion P1. The camera 122 captures images of the scenery of the travel destination including the companion P1 (hereinafter, local images). The camera 122 outputs the captured images of the local scenery (hereinafter, local images) to the communication device 110. The communication device 110 transmits the output local images to the virtual experience facility 200. The camera 122 is an example of an imaging device.

The first haptic 130 is, for example, a movable device that is held or worn by the accompanying person P1 and can come into contact with the accompanying person P1. The first haptic 130 is a so-called tactile feedback device. The first haptic 130 detects, for example, a motion directed toward a person next to the person, such as tapping on the shoulder. The first haptic 130 outputs first motion information based on the detected motion to the communication device 110. The communication device 110 transmits the output first motion information to the virtual experience facility 200.

The first haptic 130 is equipped with an actuator that reproduces the action of a person next to the person, for example, tapping on the shoulder. When second action information, which will be described later and is transmitted by the virtual experience facility 200, is output by the communication device 110, the first haptic 130 executes an action on the accompanying person P1 according to the transmitted second action information. The first haptic 130 is an example of a first movable device. The communication device 110 is an example of a transmitting/receiving device that transmits and receives the first action information or the second action information.

The virtual experience facility 200 includes, for example, a moving body 210, a running space 220 in which the moving body 210 runs, a head-mounted display, for example, VR (Virtual Reality) goggles 230 worn by the experiencer P2, a second haptic 240, and a virtual experience providing device 300. The experiencer P2 wears the VR goggles 230 and the second haptic 240, boards the moving body 210, and moves within the running space 220. The VR goggles 230 are provided with a display device 232 that displays an image that can be viewed by the experiencer P2.

Figure 2 is a perspective view showing the moving body 210. Hereinafter, the front-to-rear direction of the moving body 210 may be referred to as the X direction, the lateral direction as the Y direction, and the direction perpendicular to the X direction and the Y direction as the Z direction. The moving body 210 is a moving body that can be ridden by the experiencer P2 in the virtual experience facility 200. The moving body 210 travels within the travel space 220 in response to the operation of the experiencer P2.

The moving body 210 includes, for example, a base body 211. A control device 212 is housed inside the base body 211. The control device 212 controls the traveling direction and traveling speed of the moving body 210. In addition to the control device 212, a motor, a battery, and the like are housed inside the base body 211. Wheels 213 are attached to the left and right sides of the base body 211.

The control device 212 transmits boarding information to the virtual experience providing device 300 when the experiencer P2 boards the moving body 210. The control device 212 transmits movement information according to the movement of the moving body 210 to the virtual experience providing device 300. The control device 212 transmits movement start information to the virtual experience providing device 300 when the moving body 210 starts moving.

When the moving body 210 stops, the control device 212 transmits stop information to the virtual experience providing device 300. When the stopped moving body 210 resumes moving, the control device 212 transmits movement resumption information to the virtual experience providing device 300. When the experiencer P2 dismounts from the moving body 210, the control device 212 transmits dismount information to the virtual experience providing device 300.

A seating section 214 is provided on the upper part of the base 211. The seating section 214 is like a cushion on which the experiencer P2 sits. The upper surface of the seating section 214 is a seating surface that supports the buttocks of the experiencer P2 from below. The seating section 214 is supported by the base 211. The seating section 214 is supported by the base 211, a first support section 215 arranged in front of the base 211, and a second support section 216 arranged in the rear of the base 211. The first support section 215 and the second support section 216 support the seating section 214 so that it can be raised and lowered.

An operating unit 217 is provided to the side of the seating section 214. The operating unit 217 includes an operator 217A that allows the participant P2 to control the operation of the moving body 210. For example, the operator 217A is an operator for controlling the direction of travel, running speed, braking operation, etc. of the moving body 210. The operating unit 217 transmits an electrical signal corresponding to the operation of the operator 217A to the control device 212. By operating the operator 217A, the participant P2 can make the moving body 210 run and adjust the distance between the accompanying person P1 and the participant P2 displayed in the video.

A load sensor 218 is provided inside the seating section 214. The load sensor 218 detects the position where the experiencer P2 puts his/her weight on the seating section 214. For example, when the experiencer P2 leans his/her body forward, the load sensor 218 detects the load of the experiencer P2 in front of the seating section 214. The load sensor 218 transmits the detected load to the control device 212.

The control device 212 drives the moving body 210 based on the direction in which the load sensor 218 detects the load of the experiencer P2. In this way, the experiencer P2 can drive (move) the moving body 210 by operating the controller 217A or by applying a load in the desired direction. Furthermore, the control device 212 recognizes that the experiencer P2 has boarded and disembarked from the moving body 210 based on the load transmitted by the load sensor 218.

The moving body 210 of the embodiment is a so-called hands-free moving body 210 that can be operated by the experiencer P2 shifting his/her weight, and can be operated by the experiencer P2 operating the operating unit 217, but the operating unit 217 does not have to be provided. The moving body 210 may also be a saddle-type moving body on which the experiencer P2 straddles. The moving body 210 may also be a moving body on which the experiencer P2 stands and rides on a step. The moving body 210 may also be one that moves by walking using the legs instead of wheels.

The running space 220 is a space large enough for the moving body 210 to move in, for example, a base area of several meters by several meters. The running space 220 is provided in an appropriate location, such as inside a building. For example, if there is a travel business (travel agency) that sold a travel product to the accompanying person P1, the running space 220 is provided in or near the premises of the travel business (travel agency).

The VR goggles 230 display virtual reality images that are based on the on-site images transmitted by the communication device 110 and that evoke a virtual space for the participant P2. The VR goggles 230 include a display device 232 that displays the virtual reality images, and a fitting part that is hooked onto the head of the participant P2 and allows the participant P2 to wear the VR goggles 230 so that the participant P2 can see the display device 232. Instead of the VR goggles 230, MR (Mixed Reality) goggles or AR (Augmented Reality) goggles may be used.

The second haptic 240 is, for example, a movable device that is held or worn by the experiencer P2 and can come into contact with the experiencer P2. The second haptic 240 is a so-called tactile feedback device, like the first haptic 130, and detects, for example, a motion directed toward a person next to the experiencer, such as tapping on the shoulder. The second haptic 240 transmits second motion information based on the detected motion to the virtual experience facility 200.

Because there is no person actually beside the experiencer P2, the second haptic 240 may generate the second motion information based on the operation unit 217 by the experiencer P2, instead of a motion directed at the person beside him. For example, when the experiencer P2 tilts the operation element 217A of the operation unit 217 to the right, the second motion information may be generated similar to that generated when a motion of tapping the shoulder of the person beside him on the right is detected.

The second haptic 240 is equipped with an actuator that reproduces the motion of a person riding on the moving body 210, for example, tapping on the shoulder. When first motion information described below is transmitted by the first haptic 130, the second haptic 240 executes an action corresponding to the transmitted first motion information on the experiencer P2. The second haptic 240 is an example of a second movable device.

The virtual experience providing device 300 displays images corresponding to the movement of the moving body 210 on the VR goggles 230 so that the participant P2 can have a virtual travel experience traveling with the companion P1. The participant P2 wearing the VR goggles 230 experiences a virtual travel trip accompanied by the companion P1 through the images displayed on the VR goggles 230 and the movement of the moving body 210.

The virtual experience providing device 300 includes, for example, a communication unit 310, an image generating unit 320, and a display control unit 330. The image generating unit 320 and the display control unit 330 are realized by, for example, a processor such as a CPU (Central Processing Unit), which is hardware, executing a program (software). A part or all of the image generating unit 320 and the display control unit 330 may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or a flash memory (a storage device having a non-transient storage medium), or may be stored in a removable storage medium such as a DVD or a CD-ROM (a non-transient storage medium), and may be installed in the storage device by mounting the storage medium in a drive device. The storage device can be realized by a hard disk drive (HDD), DVD, random access memory (RAM), flash memory, etc.

The communication unit 310 is, for example, a wireless communication module for connecting to a network or for directly communicating with other terminal devices, such as the communication device 110 of each travel destination device 100. The communication unit 310 performs wireless communication based on Wi-Fi, DSRC, Bluetooth (registered trademark), or other communication standards.

The communication unit 310 receives the on-site image and the first operation information transmitted by the communication device 110. The communication unit 310 outputs the received on-site image to the image generation unit 320. The communication unit 310 receives the movement information, movement start information, stop information, and movement resume information transmitted by the control device 212. The communication unit 310 outputs the received movement information and movement start information to the image generation unit 320, and outputs the stop information and movement resume information to the display control unit 330. The moving body 210 is stopped during the period from when the stop information is received until when the movement resume information is received.

When movement start information is output by the communication unit 310, the image generation unit 320 generates a virtual reality image to be displayed on the display device 232 of the VR goggles 230. The image generation unit 320 generates the virtual reality image based on the local image output by the communication unit 310 and the movement information output by the control device 212. In the first embodiment, the virtual reality image is, for example, an image generated based on the local image, and is an image in which the accompanying person P1 is added to the real image.

The image generating unit 320 generates a virtual reality image, for example, based on the local image output by the communication unit 310 at the timing when the movement start information is transmitted by the control device 212. The image generating unit 320 calculates the position of the experiencer P2 in the virtual reality image based on the movement information transmitted by the control device 212. For example, when a movement signal for the moving body 210 to move forward is transmitted, the image generating unit 320 generates a virtual reality image that the experiencer P2 sees when moving forward in the virtual reality image.

By moving the moving body 210 while watching the virtual reality image, the participant P2 can have a virtual experience of traveling through the scenery shown in the virtual reality image. In addition, by moving the moving body 210 as if following the companion P1 in the virtual reality image, the participant P2 can have a virtual travel experience of traveling together with the companion P1.

For example, when the person experiencing P2 is operating the moving object 210 so as to move together with the accompanying person P1, the image generating unit 320 generates a virtual reality image using the on-site image transmitted by the camera 122. When the person experiencing P2 is operating the moving object 210 in such a direction that the person experiencing P2 moves away from the accompanying person P1, the image generating unit 320 generates a virtual reality image using, for example, the on-site image previously transmitted by the camera 122.

When generating the virtual reality image, the image generating unit 320 calculates the position of the viewpoint (point of gaze) of the companion P1 based on the local image. The viewpoint of the companion P1 is calculated by estimation based on, for example, the facial direction of the companion P1 included in the local image. The companion P1 may indicate the viewpoint of the companion P1 by words or actions. The image generating unit 320 generates a virtual reality image by adding a point of gaze image indicating the position of the viewpoint (point of gaze) of the companion P1 to the local image. The image generating unit 320 notifies the display control unit 330 of information about the generated virtual reality image.

The display control unit 330 causes the display device 232 to display a virtual reality image based on the virtual reality image information notified by the image generation unit 320. The experiencer P2 visually recognizes the virtual reality image displayed on the display device 232. The image captured by the camera 122 is an image that includes the companion P1, and is an image captured from a viewpoint (standing point) different from that of the companion P1. The virtual reality image is an image in which the viewpoint (point of gaze) of the experiencer P2 is different from the viewpoint (point of gaze) of the companion P1, and the companion P1 is included in the field of view of the experiencer P2.

When movement start information is output by the communication unit 310, the display control unit 330 causes the display device 232 to display a virtual reality image based on the information notified by the image generation unit 320. When stop information is output by the communication unit 310, the display control unit 330 causes the display device 232 to display the virtual reality image as a stopped image. The virtual reality stopped image is an image in which the virtual reality image based on the information notified by the image generation unit 320 has been stopped. The display control unit 330 also causes the display device 232 to display the stopped image when boarding information is received. While the moving object 210 is stopped, the display control unit 330 causes the virtual reality image to be displayed as a stopped image.

Next, the procedure for using the virtual experience facility 200 will be described. In the virtual experience facility 200, the experiencer P2 puts on the VR goggles 230 before entering the travel space 220. The experiencer P2, wearing the VR goggles 230, moves to the travel space 220 and boards the moving body 210. The experiencer P2 boards the moving body 210 in the travel space 220 and travels the moving body 210 while watching the image displayed on the VR goggles 230, thereby experiencing a virtual trip traveling together with the companion P1.

Figure 3 is a bird's-eye view of a participant P2 using the virtual experience facility 200. The participant P2, who is a user of the virtual experience facility 200, wears VR goggles 230 and moves the moving body 210 in the travel space 220 by operating the controller 217A of the moving body 210 and leaning his/her body on the seat 214 in the direction he/she wants to move. In the virtual experience facility 200, the participant P2 can have a virtual experience of traveling while riding on the moving body 210.

For example, when participant P2 applies weight forward on seat 214, moving body 210 moves straight ahead. At this time, an image of participant P2 moving forward is displayed as a virtual reality image on display device 232 of VR goggles 230 under the control of display control unit 330 of virtual experience providing device 300. Moving body 210 moves in running space 220 due to the movements of participant P2, but if moving body 210 is about to collide with a wall in running space 220, moving body 210 will guide moving body 210 and correct its trajectory without being noticed by participant P2.

The experiencer P2 aboard the moving body 210 experiences a virtual trip while watching the images displayed on the display device 232. FIG. 4 is a diagram showing an example of the first virtual reality image GA1 displayed on the display device 232. The first virtual reality image GA1 displayed on the display device 232 is an image in which a gaze image GB2 showing the gaze of the companion P1 and a gaze point image GB3 showing the gaze point, which are generated by the image generating unit 320, are added to a scenery image GB1 of the scenery at the travel destination captured by the camera 122. As the scenery image GB1 includes an image of the companion P1, the first virtual reality image GA1 also includes a companion image GB4.

Next, the processing in the virtual experience providing system 1 will be described. First, the processing of each travel destination device 100 will be described. Here, as part of the processing of each travel destination device 100, processing excluding processing related to the first haptic 130 will be described. FIG. 5 is a flowchart showing an example of the processing of each travel destination device 100. The processing of each travel destination device 100 is executed, for example, when companion P1 arrives at the travel destination and moves around the travel destination.

As the accompanying person P1 moves, the drone 120 flies around the accompanying person P1 (step S101). While the drone 120 flies around the accompanying person P1, the camera 122 captures local video including the surrounding scenery including the accompanying person P1 (step S103) and outputs the video to the communication device 110. The communication device 110 transmits the local video output by the camera 122 to the virtual experience facility 200 (step S105). The drone 120 and the camera 122 continue to fly and capture images, respectively, and the communication device 110 repeatedly executes the process of transmitting the local video at regular time intervals.

Next, the processing in the virtual experience providing device 300 will be described. Here, as part of the processing of the virtual experience providing device 300, processing excluding processing related to the second haptic 240 will be described. FIG. 6 is a flowchart showing an example of the processing of the virtual experience providing device 300. The processing in the virtual experience providing device 300 is started, for example, after the experiencer P2 visits the virtual experience facility 200 and boards (rides) the moving body 210 in the traveling space 220.

In the virtual experience facility 200, first, the moving object 210 starts moving based on the operation of the experiencer P2. When the moving object 210 starts moving, the control device 212 of the moving object 210 transmits movement start information to the virtual experience providing device 300. The virtual experience providing device 300 receives the movement start information via the communication unit 310 (step S201).

The communication unit 310, which has received the movement start information, receives the local image transmitted by the communication device 110 of each device 100 at the travel destination (step S203) and outputs it to the image generation unit 320. The image generation unit 320 calculates the viewpoint of the companion P1 based on the local image output by the communication unit 310 (step S205). Next, the image generation unit 320 generates a virtual reality image by adding a gaze image GB2 indicating the gaze of the companion P1 and a gaze point image GB3 indicating the position of the gaze point of the companion P1 to the local image (step S207). The image generation unit 320 notifies the display control unit 330 of the generated virtual reality image.

Then, the display control unit 330 causes the virtual reality image notified by the image generation unit 320 to be displayed on the display device 232 of the VR goggles 230 (step S209). The experiencer P2 wearing the VR goggles 230 can experience a virtual trip while viewing the scenery displayed in the virtual reality image by visually checking the image displayed on the display device 232. Furthermore, the virtual reality image shows the on-site companion P1. This enhances the sense of reality of the experiencer P2 who is traveling together with the on-site companion P1. Furthermore, the virtual reality image shows the viewpoint of the companion P1. This makes it easier for the experiencer P2 to feel that he or she is experiencing something in common with the companion P1.

Next, the display control unit 330 determines whether the moving object 210 is stopped (step S211). If stop information has been transmitted by the moving object 210, the display control unit 330 determines that the moving object 210 is stopped, and if stop information has not been transmitted, the display control unit 330 determines that the moving object 210 is not stopped.

If it is determined that the moving object 210 is not stopped, the display control unit 330 returns the process to step S203. If it is determined that the moving object 210 is stopped, the display control unit 330 causes the display device 232 of the VR goggles 230 to display a stopped image (step S213).

The still image displayed on the display device 232 may be, for example, an image obtained by cutting out a frame of the virtual reality image at the timing when the communication unit 310 receives the stop information. The still image displayed on the display device 232 may be another image, or may be an image obtained by cutting out a frame of the virtual reality image at a timing a certain time after the communication unit 310 receives the stop information.

Then, the display control unit 330 determines whether the moving body 210 has resumed movement (step S215). If it is determined that the moving body 210 has resumed movement, the display control unit 330 returns the process to step S203. If it is determined that the moving body 210 has not resumed movement, the display control unit 330 determines whether the experiencer P2 has dismounted from the moving body 210 (step S217). The display control unit 330 determines whether the experiencer P2 has dismounted from the moving body 210 based on whether the communication unit 310 has received dismounting information transmitted by the control device 212 of the moving body 210.

If it is determined that the experiencer P2 has not dismounted from the moving body 210, the display control unit 330 returns the process to step S213 and continues to display the still image. If the display control unit 330 determines that the experiencer P2 has dismounted from the moving body 210, the virtual experience providing device 300 ends the process shown in FIG. 6.

Next, the processing related to the first haptic 130 and the second haptic 240 in each travel destination device 100 and the virtual experience providing device 300 of the virtual experience providing system 1 will be described. FIG. 7 is a sequence diagram of the virtual experience providing system 1 related to the first haptic 130 and the second haptic 240. The processing related to the first haptic 130 and the second haptic 240 is executed independently of the processing shown in FIG. 5 and FIG. 6 after the experiencer P2 boards the moving body 210 until the experiencer P2 disembarks from the moving body 210.

In each travel destination device 100, the first haptic 130 determines whether or not a movement of the companion P1 has been detected (step S301). If it is determined that a movement of the companion P1 has been detected, the first haptic 130 generates first movement information based on the movement of the companion P1 (step S303).

Next, the first haptic 130 transmits the generated first movement information to the second haptic 240 of the virtual experience facility 300 (step S305). If it is determined in step S301 that the movement of the companion P1 has not been detected, the first haptic 130 proceeds to step S317.

Meanwhile, in the virtual experience facility 300, the second haptic 240 determines whether or not the movement of the experiencer P2 has been detected (step S307). If it is determined that the movement of the experiencer P2 has been detected, the second haptic 240 generates second movement information based on the movement of the experiencer P2 (step S309).

Next, the second haptic 240 transmits the generated second movement information to the first haptic 130 of each device 100 at the travel destination (step S311). If it is determined in step S307 that the movement of the experiencer P2 has not been detected, the second haptic 240 proceeds to step S313.

The second haptic 240 then determines whether or not it has received the first action information transmitted by the first haptic 130 (step S313). If it is determined that it has received the first action information, the second haptic 240 executes an action on the experiencer P2 based on the first action information (step S315). In this way, the second haptic 240 ends the process shown in FIG. 7. If it is determined in step S313 that it has not received the first action information, the second haptic 240 simply ends the process shown in FIG. 7.

The first haptic 130 then determines whether or not it has received the second action information transmitted by the second haptic 240 (step S317). If it is determined that the second action information has been received, the first haptic 130 executes an action on the companion P1 based on the second action information (step S319). In this way, the first haptic 130 ends the process shown in FIG. 7. If it is determined in step S317 that the second action information has not been received, the first haptic 130 simply ends the process shown in FIG. 7.

The virtual experience providing system 1 of the first embodiment generates a virtual reality image in which the viewpoint of the participant P2 and the viewpoint of the companion P1 are different and the companion P1 is included in the field of view of the participant P2, and displays the image on the display device 232 of the VR goggles 230. Therefore, by wearing the VR goggles 230 and experiencing a virtual trip, the participant P2 who is experiencing a virtual trip together with the companion P1 can feel more realistic that he or she is traveling together with the companion P1.

Second Embodiment
Next, a second embodiment will be described. The virtual experience providing system 1 of the second embodiment is different from the first embodiment mainly in that, when a companion is not shown in the on-site image, an image imitating the companion, such as an avatar, is displayed in the virtual reality image. Hereinafter, the virtual experience providing device, virtual experience providing system, virtual experience providing method, and program of the second embodiment will be described, focusing on the differences from the first embodiment.

Figure 8 is a diagram showing an example of the configuration of the virtual experience providing system 1 of the second embodiment. In the virtual experience providing system 1 of the second embodiment, each travel destination device 100 includes a communication device 110, a drone 120, a camera 122, a first haptic 130, as well as a travel destination moving body 140 and a second camera 142. In the following description, the camera 122 mounted on the drone 120 is referred to as the first camera 122, and the image captured by the first camera 122 is referred to as the first local image.

The travel destination mobile body 140 is, for example, a vehicle that the companion P1 boards when traveling to the travel destination. The companion P1 can, for example, rent the travel destination mobile body 140 at the travel destination. The travel destination mobile body 140 is the same as the mobile body 210 provided in the virtual experience facility 200. The travel destination mobile body 140 may be a different vehicle from the mobile body 210.

The second camera 142 is mounted, for example, on the tip of the travel destination mobile object 140. The second camera 142 captures local video within the range assumed to be the field of vision of the traveling companion P1 (hereinafter, the second local video). The second camera 142 outputs the captured second local video to the communication device 110. The communication device 110 transmits the output second local video to the virtual experience facility 200.

The virtual experience providing device 300 provided in the virtual experience facility 200 includes a communication unit 310, an image generating unit 320, a display control unit 330, an operation content receiving unit 340, and an adjustment unit 350. The operation content receiving unit 340 and the adjustment unit 350 are realized, for example, by a processor such as a CPU, which is hardware, executing a program together with the communication unit 310, the image generating unit 320, and the display control unit 330. The communication unit 310, the image generating unit 320, the display control unit 330, the operation content receiving unit 340, and the adjustment unit 350 may be realized by an LSI or the like shown in the first embodiment.

The control device 212 mounted on the moving body 210 generates information on the operation performed by the experiencer P2 on the operation unit 217 (hereinafter, operation information) by the experiencer P2 on the moving body 210 operating the operation element 217A of the operation element 217. The content of the operation performed by the experiencer P2 is, for example, an instruction to adjust the distance of the companion P1 from the experiencer P2 in the virtual reality image GA. The control device 212 mounted on the moving body 210 predetermines the instruction of the experiencer P2 according to the operation of the operation element 217A. For example, when the operation element 217A is tilted to the right, the instruction is determined to be an instruction for the companion P1 to move away from the experiencer P2 in the virtual reality image GA, and when the operation element 217A is tilted to the left, the instruction is determined to be an instruction for the companion P1 to move closer to the experiencer P2 in the virtual reality image GA. The control device 212 transmits the generated operation information to the virtual experience providing device 300.

The communication unit 310 in the virtual experience providing device 300 receives operation information transmitted by the control device 212 in addition to the processing described in the first embodiment. The communication unit 310 outputs the received operation information to the operation content receiving unit 340.

The image generating unit 320 determines whether or not the companion P1 is shown in the local image (first local image or second local image) transmitted by each travel destination device 100. If the companion P1 is shown in the local image, the image generating unit 320 generates a virtual reality image including the viewpoint of the companion P1 in the local image.

When the companion P1 is not shown in the on-site image, the image generator 320 generates an image that resembles the companion, for example an avatar, in the on-site image and includes it, and generates a virtual reality image that also includes the viewpoint of the companion P1. The companion P1 cannot directly see his/her own appearance, such as his/her face. For this reason, the viewpoint of the virtual reality image that includes the avatar is different from the viewpoint of the companion P1, and therefore different from the viewpoint of the experiencer P2 who is viewing the virtual reality image and the viewpoint of the companion P1.

Figure 9 is a diagram showing an example of scenery image GC1. Since the second camera 142 is positioned at the tip of the travel destination moving body 140, it is able to capture scenery image GC1 that is assumed to be within the field of vision of the companion P1. However, the companion P1 himself is not included in the image. Therefore, the image generating unit 320 generates a virtual reality image based on the scenery image GC1, in which the companion P1 is displayed together with the scenery image GC1.

Figure 10 is a diagram showing an example of a second virtual reality image GA2 generated based on a landscape image GC1. The second virtual reality image GA2 is an image in which a gaze image GC2 showing the gaze of the companion P1, a gaze point image GC3 showing the gaze point of the companion P1, and an avatar GC4 imitating the companion P1, generated by the image generating unit 320, are added to the landscape image GC1 of the landscape of the travel destination captured by the second camera 142. By experiencing a virtual trip while viewing a virtual reality image including the companion, the experiencer P2 can increase the sense of reality of being together with the companion P1 at the site. However, since the companion P1 is not shown in the landscape image GC1, it is difficult to increase the sense of reality of the experiencer P2 simply by generating a virtual reality image by adding the gaze image GC2 and the gaze point image GC3 to the landscape image GC1.

The image generator 320 then generates the second virtual reality image GA2 by adding the gaze image GC2, the gaze point image GC3, and an avatar GC4 to the scenery image GC1. By displaying the avatar GC4 in the second virtual reality image GA2, the experiencer P2 can feel more real as he or she is traveling with the on-site companion P1.

The avatar GC4 included in the second virtual reality image GA2 may be any type of avatar. For example, the avatar GC4 may be an illustration or photograph of the companion P1, or may be a cartoon character. Alternatively, the avatar GC4 may be a character imitating a famous person (famous person) such as an entertainer or athlete, or may be an illustration or photograph of such a person. The same avatar GC4 may always be displayed, or the displayed avatar may change depending on the display time, the current time, etc. Furthermore, the avatar GC4 may be selectable by the experiencer P2.

The operation content receiving unit 340 acquires operation information transmitted by the control device 212. Based on the acquired operation information, the operation content receiving unit 340 receives an instruction from the experiencer P2 according to the content of the operation performed by the experiencer P2 (hereinafter, operation content). The operation content receiving unit 340 notifies the adjustment unit 350 of the accepted operation content. The instruction based on the operation information is, for example, an instruction to adjust the distance between the companion P1 and the experiencer P2, and an instruction to move the avatar GC4 away from or closer to the experiencer P2 in the second virtual reality image GA2.

The adjustment unit 350 adjusts the display position of the avatar GC4 in the second virtual reality image GA2 based on the instruction of the adjustment information notified by the operation content receiving unit 340. For example, when the instruction based on the operation information is an instruction to move the avatar GC4 away from the experiencer P2 in the second virtual reality image GA2, the adjustment unit 350 moves the display position of the avatar GC4 from the side where the experiencer P2 is assumed to be located to the opposite side.

Next, the processing of the virtual experience providing device 300 of the second embodiment will be described. FIG. 11 is a flowchart showing a part of an example of the processing of the virtual experience providing device 300. After receiving the local image (step S203), the virtual experience providing device 300 determines whether the companion P1 is shown in the local image (step S251).

The determination of whether or not the accompanying person P1 is shown in the local video is performed based on whether the local video is the first local video or the second local video. When the local video is the first video, the video generation unit 320 determines that the accompanying person is shown in the local video, and when the local video is the second video, the video generation unit 320 determines that the accompanying person is not shown in the local video. The determination of whether or not the accompanying person P1 is shown in the local video may be performed by other means. For example, the accompanying person may be searched for by performing a video analysis of the local video.

If it is determined that the companion P1 is shown in the local image, the image generating unit 320 proceeds to step S205. If it is determined that the companion P1 is shown in the local image, the image generating unit 320 generates an avatar for the companion P1 (step S253) and proceeds to step S205. Next, the image generating unit 320 calculates the viewpoint of the companion P1 (step S205) and generates a virtual reality image (step S207), after which the display control unit 330 displays the virtual reality image on the display device 232.

Then, it is determined whether the operation content receiving unit 340 has received an instruction from the experiencer P2 based on the operation information transmitted by the control device 212 (step S255). If the operation content receiving unit 340 determines that an instruction from the experiencer P2 based on the operation information transmitted by the control device 212 has been received, the display control unit 330 moves the display position of the avatar in the direction according to the instruction from the experiencer P2 (step S257). If the operation content receiving unit 340 determines that an instruction from the experiencer P2 based on the operation information transmitted by the control device 212 has not been received, the display control unit 330 proceeds directly to step S211. Thereafter, the virtual experience providing device 300 executes the same processing procedure as in the first embodiment, and ends the processing shown in FIG. 11.

The virtual experience providing system 1 of the second embodiment has the same effect as the virtual experience providing system 1 of the first embodiment. The virtual experience providing system 1 of the second embodiment can further adjust the distance between the companion P1 and the experiencer P2 in the virtual reality image. This can further enhance the sense that the experiencer who is taking part in the virtual travel experience traveling with the companion is traveling together with the companion.

In the second embodiment, the image generating unit 320 of the virtual experience providing device 300 may generate an image with an avatar added as a virtual reality image based on the image always transmitted by the second camera 142. In the second embodiment, the virtual reality image is generated every time both the first local image (image including the companion P1) and the second local image (image not including the companion P1) are transmitted, but conditions may be set to generate the virtual reality image based on the local image only when either the first local image or the second local image is transmitted. For example, if the experiencer P2 wishes to take a virtual trip with an avatar as his or her companion, the virtual reality image may be generated only from the image (not including the companion P1). The image generating unit 320 may also generate a virtual reality image in which the companion included in the first local image is replaced with an avatar.

In the second embodiment described above, the first camera 122 and the second camera 142 are fixed to the drone 120 or the travel destination mobile body 140 and are cameras that capture images of fixed positions relative to the drone 120 or the travel destination mobile body 140, but the second camera 142 may be a camera whose capture position is movable, for example, a 360° camera. Also, when adjusting the distance between the companion P1 and the experiencer P2, the display position of the avatar is moved, but the distance between the companion P1 and the experiencer P2 may be adjusted, for example, by moving the capture position of the on-site image including the companion P1.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1... Virtual experience providing system 40... Second haptic 100... Travel destination equipment 110... Communication device 120... Drone 122... Camera (first camera)
130...First haptic 140...Travel destination mobile object 142...Second camera 210...Mobile object 217...Operation unit 217A...Operator 218...Load sensor 220...Driving space 230...VR goggles 232...Display device 240...Second haptic 300...Virtual experience providing device 310...Communication unit 320...Image generation unit 330...Display control unit 340...Operation content receiving unit 350...Adjustment unit GA...Virtual reality image GA1...First virtual reality image GA2...Second virtual reality image GB1, GC1...Scenery image GB2, GC2...Gaze image GB3, GC3...Gaze point image GB4...Companion image GC4...Avatar NW...Network P1...Companion P2...Experiencer

Claims (7)

A virtual experience providing device for providing a second user with a virtual travel experience traveling together with a first user, comprising:
an image generating unit that generates a virtual reality image based on a local image captured by an imaging device present around the first user at a travel destination;
A display control unit that causes a display device to display the virtual reality image;
an operation content receiving unit for acquiring information indicating the content of an operation performed by the second user on an operation unit;
an adjustment unit that adjusts a distance between the first user and the second user in the virtual reality image based on information acquired by the operation content reception unit;
The image generation unit generates the virtual reality image in which the first user is included in a field of view of the second user.
A device providing virtual experiences. The image generator generates a virtual reality image in response to a movement of a moving object operated by the second user;
The display control unit sets the virtual reality image as a still image while the moving object is stopped.
The virtual experience providing device according to claim 1 . The image generating unit generates the virtual reality image including a gaze image indicating at least one of a gaze or a gaze point of the first user.
The virtual experience providing device according to claim 1 or 2 . The virtual experience providing device according to any one of claims 1 to 3 ;
An imaging device that captures the on-site image;
A moving object on which the second user boards during the virtual travel experience;
A display device for displaying the virtual reality image.
Virtual experience provision system. A virtual experience providing device;
an imaging device that captures local video captured by imaging devices present around a first user at a travel destination;
A moving object on which a second user boards during a virtual travel experience in which the second user travels together with the first user;
a display device that displays a virtual reality image generated based on the on-site image captured by the imaging device; and
At least one of a first movable device accessible to the first user and a second movable device accessible to the second user;
a transmitting/receiving device for transmitting/receiving at least one of first motion information corresponding to a motion of the first user and transmitted to the second movable device, and second motion information corresponding to a motion of the second user and transmitted to the first movable device ;
The virtual experience providing device includes:
A virtual experience providing device for providing the virtual travel experience to the second user,
an image generating unit that generates the virtual reality image;
A display control unit that causes the display device to display the virtual reality image,
The image generation unit generates the virtual reality image in which the first user is included in a field of view of the second user.
Virtual experience provision system. A computer of a virtual experience providing device for providing a second user with a virtual travel experience traveling together with a first user,
generating a virtual reality image based on a local image captured by an imaging device present around the first user at the travel destination;
Displaying the virtual reality image on a display device;
acquiring information indicating the content of an operation performed by the second user on an operation unit;
adjusting a distance between the first user and the second user in the virtual reality image based on the information;
The computer includes:
generating the virtual reality image including the first user in a field of view of the second user;
How to deliver virtual experiences. A computer of a virtual experience providing device for providing a second user with a virtual travel experience traveling together with a first user,
generating a virtual reality image based on a local image captured by an imaging device present around the first user at a travel destination;
Executing a process to display the virtual reality image on a display device;
causing an operation unit to acquire information indicating the content of an operation performed by the second user;
adjusting a distance between the first user and the second user in the virtual reality image based on the information;
The computer includes:
generating the virtual reality image including the first user in a field of view of the second user;
program.

Images