JP7533944B2 - COMPUTER PROGRAM, INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, AND AUGMENTED REALITY IMAGE DISPLAY METHOD - Google Patents

Description

The present invention relates to a computer program, an information processing device, an information processing system , and an augmented reality image display method for displaying an augmented reality image on a glasses-type wearable device worn by a user.

Augmented reality (AR) glasses, smart glasses, head-mounted displays, and other glasses-type wearable devices are becoming more common. For example, AR glasses have a transparent (translucent) display in the area that corresponds to the lens of glasses, and can display various types of information such as text or images on this display. A user wearing the AR glasses can view the real space through the transparent display, and can obtain various types of information displayed on the display. The AR glasses are also equipped with a camera or sensor that captures the surrounding real space, and can provide virtual information to the user by superimposing it on the real space by displaying an image at an appropriate position relative to the real space viewed by the user. This technology is called augmented reality, and is attracting attention.

Patent Document 1 proposes a head-mounted display that includes a group of sensors that detect changes in position relative to the user's head, and a control unit that executes a predetermined process when the change in position is detected. This head-mounted display is shaped like glasses, and includes a touch sensor and a motion sensor provided on the bridge as a group of sensors. The control unit of the head-mounted display executes a predetermined process when the touch sensor detects contact and the motion sensor detects movement.

JP 2016-149587 A

Augmented reality technology using wearable devices such as AR glasses is in its early stages, and we are still exploring how to use this technology and what services it can provide.

The present invention has been made in consideration of the above circumstances, and its purpose is to provide a computer program, an information processing device, an information processing system, and an augmented reality image display method that are expected to realize new services that utilize augmented reality technology.

The computer program of the present invention causes a computer to perform a process of displaying an augmented reality image on a glasses-type wearable device that has a camera that captures images of real space and a display unit that displays the images, and causes the computer to perform a process of detecting markers installed at a live venue based on images captured by the camera, and displaying an augmented reality image corresponding to the markers detected during the live performance on the display unit of the wearable device.

With this invention, it is expected that new services using augmented reality technology will be realized.

1 is a schematic diagram for explaining an overview of an information processing system according to an embodiment of the present invention; 1 is a schematic diagram for explaining an overview of an information processing system according to an embodiment of the present invention; FIG. 1 is a schematic diagram showing the appearance of AR glasses. 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; 2 is a block diagram showing a configuration of a terminal device according to the present embodiment; FIG. 1 is a block diagram showing a configuration of AR glasses according to an embodiment of the present invention; 2 is a block diagram showing a configuration of a server device according to the present embodiment. FIG. 10 is a flowchart showing a procedure of an augmented reality data acquisition process performed by the terminal device according to the present embodiment. 10 is a flowchart showing a procedure of a transmission process of augmented reality data performed by the server device according to the present embodiment. 10 is a flowchart showing a procedure of a live management process performed by the server device according to the present embodiment. 10 is a flowchart showing a procedure of an augmented reality image display process performed by the terminal device according to the present embodiment. FIG. 11 is a schematic diagram for explaining a process performed by an information processing system according to a second embodiment in the preparation stage of a live performance. 11 is a schematic diagram showing an example of a setting screen for setting the display position of an augmented reality image displayed by the terminal device; FIG. 11 is a schematic diagram showing an example of a setting screen for setting the display position of an augmented reality image displayed by the terminal device; FIG. 13 is a schematic diagram for explaining processing performed by an information processing system according to a second embodiment during a live performance. FIG. FIG. 11 is a block diagram showing a configuration of a terminal device according to a second embodiment. 13 is a flowchart showing a procedure of a process performed by a server device according to the second embodiment. 13 is a flowchart showing a procedure of a process performed by a terminal device in accordance with the second embodiment in a preparation stage for a live performance. 13 is a flowchart showing a procedure of a process performed by a terminal device according to the second embodiment during a live performance. 13 is a flowchart showing a procedure of a process performed by a terminal device according to the second embodiment;

Specific examples of information processing systems according to embodiments of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is intended to include all modifications within the scope of the claims and meaning equivalent to the claims.

<First embodiment>
<System Overview>
1 and 2 are schematic diagrams for explaining an overview of an information processing system according to this embodiment. The information processing system according to this embodiment can be used in events such as live music performances or concerts. FIG. 1 shows a live music venue, where an artist 1 such as a singer or performer is on stage, and a large number of spectators 2 have gathered at the venue. In this embodiment, the spectators 2 participate in the live music performance by wearing AR glasses 10 (illustrated in FIG. 3), which are glasses-type wearable devices.

The AR glasses 10 display various images (augmented reality images) according to the markers 4 installed at the live venue, the markers 5a displayed on the display device 5 installed at the live venue, the movements of the artist 1, and the progress of the live performance. The AR glasses 10 may also output audio along with the augmented reality images. Figure 2 shows the state of a live venue as seen through the AR glasses 10. The audience 2 can see the stage and the artist 1 on it through the AR glasses 10, and can also see the image 3 displayed on the AR glasses. This allows the audience 2 to experience augmented reality in which the image 3 is superimposed on the real space of the stage, artist 1, etc.

FIG. 3 is a schematic diagram showing the appearance of the AR glasses. The AR glasses 10 according to the present embodiment are in the shape of so-called glasses, and are configured such that a display unit 12, a camera 13, and an audio output unit 17 are held in a frame 11 of the glasses. The display unit 12 is provided in a location corresponding to the lens of the glasses, and the AR glasses 10 are provided with two display units 12 for the right eye and the left eye. The display unit 12 is a translucent type (translucent), and a user (audience 2) wearing the AR glasses 10 can see the surrounding scenery through the display unit 12. The camera 13 is appropriately arranged, for example, in a part such as the bridge or temple of the frame 11. The camera 13 photographs the front of the user wearing the AR glasses 10, and photographs, for example, the user's field of view. The audio output unit 17 is arranged, for example, in a part such as the temple or temple of the frame 11, which is located near the user's ear when the AR glasses 10 are worn. The audio output unit 17 is a so-called speaker, and is preferably configured to allow the user wearing the AR glasses 10 to hear both the sound from the audio output unit 17 and the surrounding sounds. The audio output unit 17 may be, for example, a bone conduction speaker. The AR glasses 10 may also be equipped with a microphone (not shown) that picks up sounds and the like around the spectator 2 wearing the glasses. The microphone may be disposed, for example, in an appropriate portion of the frame 11.

The AR glasses 10 also have a communication cable 14 connected to an end of the frame 11. The communication cable 14 is connected to a terminal device 20, such as a smartphone or a tablet terminal device. This allows the AR glasses 10 to send and receive data to and from the terminal device 20 via the communication cable 14. The AR glasses 10 display images sent from the terminal device 20 on the display unit 12, output audio sent from the terminal device 20 from the audio output unit, and transmit data such as images captured by the camera 13 and audio acquired by the microphone to the terminal device 20. The terminal device 20 performs a process of recognizing the real space around the user based on the data sent from the AR glasses 10, and a process of displaying images to realize augmented reality on the display unit 12 of the AR glasses 10.

Figure 4 is a block diagram showing the configuration of an information processing system according to this embodiment. The information processing system according to this embodiment includes a server device 30 that controls augmented reality during a live performance. The server device 30 can transmit and receive data to and from the terminal device 20 via a mobile phone communication network such as a fourth generation mobile communication system (4G) or a fifth generation mobile communication system (5G), or wireless communication such as a wireless LAN (Local Area Network). The server device 30 transmits data such as images and audio for realizing augmented reality to the terminal devices 20 owned by each of the numerous spectators 2.

In this embodiment, the server device 30 transmits data such as images and audio to each terminal device 20 before the start of the live performance. For example, some or all of the data is transmitted from the server device 30 to the terminal device 20 before the start of the live performance, and the terminal device 20 holds this data until the end of the live performance. The server device 30 transmits, for example, instructions to the terminal device 20 regarding the timing of displaying the images transmitted in advance, and the terminal device 20 can output images and audio stored in advance in response to these instructions. Furthermore, information such as the timing of display may also be transmitted in advance to the terminal device 20, and the terminal device 20 may display images according to this timing information. However, the data may be transmitted from the server device 30 to the terminal device 20 in real time during the live performance, rather than before the start of the live performance.

In the information processing system according to the present embodiment, for example, the terminal device 20 performs a process of detecting markers installed at a live venue from an image captured by the camera 13 of the AR glasses 10. A marker is a landmark that triggers the display of an augmented reality image, and various things such as pre-registered characters, symbols, figures, shapes, patterns, images, or photographs can be used. The marker may be either two-dimensional or three-dimensional. In this example, for example, the set of the live venue or the costume of the artist 1 can be used as a marker. The terminal device 20 performs a process of displaying an image determined for the detected marker on the display unit 12 of the AR glasses 10 so that the image is located at a position determined relatively based on the position of the marker. The terminal device 20 may also perform a process of outputting a sound determined for the marker from the sound output unit 17.

In addition, in the information processing system according to this embodiment, the markers detected by the terminal device 20 can be displayed, for example, on a large display device 5 installed at a live venue. For example, the markers displayed on the display device 5 during the live performance can be changed as appropriate, either manually by the live performance organizer or automatically by a device such as the server device 30, thereby allowing the augmented reality image displayed on the AR glasses 10 of the audience members 2 to be changed as appropriate.

In addition, in the information processing system according to the present embodiment, a predetermined movement or pose of the artist 1 on the stage of a live performance can be used as a marker. The terminal device 20 detects the movement of the artist 1 from an image captured by the camera 13 of the AR glasses 10, and displays an augmented reality image determined for the predetermined movement of the artist 1 on the display unit 12 of the AR glasses 10. This allows the artist 1 to make a predetermined movement or pose, thereby allowing a predetermined augmented reality image to be displayed on the AR glasses 10 of the audience 2. For example, a predetermined augmented reality image can be displayed on the AR glasses 10 in response to a predetermined choreography or pose in a dance performed by the artist 1. For example, the terminal device 20 may also perform a face detection process for detecting a person's face from an image captured by the camera 13 of the AR glasses 10, and a face recognition process for determining whether the detected face matches the face of a registered person. In this case, the terminal device 20 can determine that a specific artist 1 has appeared on the stage of a live performance based on the face recognition process, and display an augmented reality image associated with this artist 1.

In addition, in the information processing system according to this embodiment, the terminal device 20 can display an augmented reality image on the AR glasses 10 in response to the detection of a specific music or sound. This is done by treating the specific music or sound as a marker. The terminal device 20 acquires music or sound around the audience 2, for example, by a microphone provided on the AR glasses 10 or a microphone provided on the terminal device 20. When the music acquired by the microphone is a specific song, for example, the terminal device 20 displays an augmented reality image defined for this song on the AR glasses 10. The terminal device 20 also performs voice recognition processing on the sound acquired by the microphone, for example, and when the artist 1 utters a specific word or line, the terminal device 20 displays a corresponding augmented reality image on the AR glasses 10.

In addition, in the information processing system according to the present embodiment, the terminal device 20 can display a predetermined augmented reality image on the AR glasses 10 in response to the passage of a predetermined time, for example, 5 minutes, 10 minutes, 1 hour, etc., after the start of the live performance. The timing of displaying the image may be determined, for example, by time, rather than based on the start of the live performance. The timing of displaying the image may be determined by the terminal device 20, or the server device 30 may determine the display timing and give a display command, etc. to the terminal device 20. The server device 30 may notify the terminal device 20 of the timing of the start of the live performance, and the terminal device 20 may determine the display timing in synchronization with this notification. The display of the image according to the time or the time, etc. may be performed regardless of the presence or absence of a marker, or may be performed when the time conditions and the conditions for detecting the marker are met.

In addition, in the information processing system according to this embodiment, the terminal device 20 can display a predetermined augmented reality image on the AR glasses 10 without detecting a marker. For example, the terminal device 20 can display an image according to the passage of time as described above without detecting a marker. Also, for example, the terminal device 20 can display a predetermined image according to a command given by the server device 30. In this case, the server device 30 can transmit an image display command or data of the image to be displayed to the terminal device 20 according to, for example, the progress of the live performance.

For example, the terminal device 20 can acquire location information from a Global Positioning System (GPS) and display a specified image when the terminal device 20 is located at a specified location or within a specified range of locations. For example, at a music festival where live performances are held at multiple venues, the terminal device 20 can determine which venue the user is in based on GPS location information and display an augmented reality image associated with the venue on the AR glasses 10. For example, if the user is located in a location other than the venue, the terminal device 20 can display information such as the location and directions of the venue on the AR glasses 10.

In addition, in the information processing system according to this embodiment, the stage and audience seats in the live venue may be photographed, for example, by a camera (not shown) installed in the live venue. The images captured by this camera are provided to the server device 30, which may determine the state of the live performance from these images and provide commands to the terminal devices 20 of the users participating in the live performance to display an augmented reality image, etc. For example, the server device 30 may determine from the images captured by the camera that more than 80% of the large number of spectators 2 in the live venue are waving penlights, and in response to this, transmit commands to the terminal devices 20 to display a specific augmented reality image, etc.

In addition, in the information processing system according to this embodiment, an augmented reality image may be displayed on the AR glasses 10 in response to a user operation received by the AR glasses 10 or the terminal device 20 during a live performance. For example, the terminal device 20 receives evaluations and the like regarding the live performance during the live performance, and transmits the received evaluations and the like to the server device 30. The server device 30 can, for example, tally up evaluations from the spectators 2 participating in the live performance, and when high evaluations are obtained from 80% or more of the spectators 2, display a specified augmented reality image on the AR glasses 10.

In the information processing system according to this embodiment, the terminal device 20 has a recording function. The image recorded by the terminal device 20 is an image captured by the camera 13 of the AR glasses 10 with an augmented reality image superimposed thereon. The terminal device 20 may also have a function for recording audio captured by a microphone with augmented reality audio superimposed thereon. The terminal device 20 stores the recorded and audio data in a memory or the like, for example, as a single file. The data recorded and audio by the terminal device 20 may be made publicly available to the spectators 2 using a service such as a social networking service (SNS), or may be made unavailable for the purpose of protecting copyrights, etc.

In this embodiment, the terminal device 20 connected to the AR glasses 10 via the communication cable 14 performs image processing for augmented reality, but this is not limited to the above. For example, a configuration that does not use the terminal device 20, so-called standalone AR glasses, may be adopted. For example, the AR glasses 10 may have the functions of the terminal device 20. In this case, the AR glasses 10 may communicate with the server device 30 to receive images, and may perform image display processing to detect markers and realize augmented reality. In addition, for example, the server device 30 may perform processing of the terminal device 20. In this case, the AR glasses 10 transmit images captured by the camera 13 to the server device 30, and display the images transmitted from the server device 30 on the display unit 12. The server device 30 transmits image data to be displayed on the display unit 12 to multiple AR glasses 10. The server device 30 may not be a single device, but may be a server system realized by the cooperation of multiple devices.

In addition, in this embodiment, an example of using the information processing system for a live music performance will be described, but the application of this information processing system is not limited to live performances. For example, this information processing system can be applied to various events in which spectators 2 participate, such as product launches, plays, circuses, or sports events. The information processing system according to this embodiment is suitable for events in which multiple spectators 2 gather around a stage or stadium. Furthermore, this information processing system can be applied to, for example, art galleries or museums, or events where art works are exhibited. In this case, for example, markers can be placed around the exhibit, or the exhibit itself can be used as a marker to display a virtual reality image on the AR glasses 10.

<Device Configuration>
5 is a block diagram showing the configuration of the terminal device 20 according to the present embodiment. The terminal device 20 according to the present embodiment is configured to include a processing unit (processor) 21, a memory unit (storage) 22, a wireless communication unit (transceiver) 23, a wired communication unit (transceiver) 24, a display unit (display) 25, and an operation unit 26. The terminal device 20 may be configured using a general-purpose information processing device such as a smartphone, a tablet terminal device, or a personal computer.

The processing unit 21 is configured using an arithmetic processing device such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The processing unit 21 reads and executes a program 22a stored in the storage unit 22, thereby performing various processes such as detecting markers installed or displayed in the live venue and displaying an augmented reality image.

The storage unit 22 is configured using a non-volatile memory element such as a flash memory. The storage unit 22 stores various programs executed by the processing unit 21 and various data required for the processing of the processing unit 21. In this embodiment, the storage unit 22 stores the program 22a executed by the processing unit 21 and augmented reality data 22b of the augmented reality to be displayed on the display unit 12 of the AR glasses 10. The storage unit 22 can also store data recorded and sounded by the terminal device 20.

The program 22a stored in the storage unit 22 is an application program, so-called an app, for performing image display that realizes augmented reality. In this embodiment, the program 22a is distributed by a remote server device or the like, and the terminal device 20 acquires it through communication and stores it in the storage unit 22. However, the program 22a may be written in the storage unit 22, for example, during the manufacturing stage of the terminal device 20. For example, the program 22a may be read by the terminal device 20 from a recording medium 98 such as a memory card or an optical disk and stored in the storage unit 22. For example, the program 22a may be read by a writing device from the recording medium 98 and written to the storage unit 22 of the terminal device 20. The program 22a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 98.

The augmented reality data 22b stored in the storage unit 22 is data of an augmented reality image to be displayed on the display unit 12 of the AR glasses 10. The augmented reality data 22b may also include audio data to be output as the audio of the augmented reality. However, in the following, an example of displaying an image as augmented reality will be described, and an explanation of audio output will be omitted. The terminal device 20 can download the augmented reality data 22b distributed by the server device 30 and store it in the storage unit 22, for example, before the start of the live performance. Also, for example, the terminal device 20 may acquire the augmented reality data 22b from the server device 30 in real time or close to real time during the live performance and store it in the storage unit 22. The image data included in the augmented reality data 22b may be either a two-dimensional image or a three-dimensional image, and may be either a still image or a moving image.

The wireless communication unit 23 communicates with various devices by wireless communication, for example, a mobile phone communication network such as a fourth generation mobile communication system (4G) or a fifth generation mobile communication system (5G), or a wireless LAN (Local Area Network). In this embodiment, the wireless communication unit 23 wirelessly transmits and receives data with the server device 30. The wireless communication unit 23 transmits data provided by the processing unit 21 to the server device 30, and provides data received from the server device 30 to the processing unit 21.

The wired communication unit 24 communicates with other devices connected via a communication cable, for example, according to a USB (Universal Serial Bus) communication standard. In this embodiment, the wired communication unit 24 transmits and receives data to and from the AR glasses 10 connected via a communication cable 14. The wired communication unit 24 transmits data provided by the processing unit 21 to the AR glasses 10, and provides data received from the AR glasses 10 to the processing unit 21. Note that in this embodiment, the terminal device 20 and the AR glasses 10 are configured to perform wired communication, but this is not limited thereto, and wireless communication such as Bluetooth (registered trademark) may also be performed.

The display unit 25 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 21.

The operation unit 26 accepts user operations and notifies the processing unit 21 of the accepted operations. For example, the operation unit 26 accepts user operations through an input device such as a mechanical button or a touch panel provided on the surface of the display unit 25. For example, the operation unit 26 may also be an input device such as a mouse and a keyboard, and these input devices may be configured to be removable from the terminal device 20.

Although not shown in the figure, the terminal device 20 may include a GPS receiver that receives GPS signals and a microphone that captures surrounding sounds.

In addition, in the terminal device 20 according to this embodiment, the processing unit 21 reads out and executes the program 22a stored in the storage unit 22, so that the detection processing unit 21a and the augmented reality processing unit 21b are realized in the processing unit 21 as software-like functional units. The detection processing unit 21a performs processing to detect markers installed or displayed at a live venue based on an image captured by the camera 13 of the AR glasses 10. For example, the detection processing unit 21a compares an image previously stored as a marker with an image captured by the camera 13, and detects whether or not an image of a marker is included in the captured image, and if so, its position, etc. The marker detected by the detection processing unit 21a can be, for example, props, props, or parts thereof installed at a live venue. The marker detected by the detection processing unit 21a can be, for example, an image displayed on a display device 5 installed at a live venue or a part of this image. The marker detected by the detection processing unit 21a can be, for example, the costume of the artist 1, the pose of the artist 1, the movement of the artist 1, etc. For example, the detection processing unit 21a may also detect music, audio, etc. that is predefined as a marker based on music, audio, etc. acquired from a microphone provided in the AR glasses 10 or the terminal device 20.

The augmented reality processing unit 21b performs processing to display an augmented reality image on the display unit 12 of the AR glasses 10 in response to, for example, a marker detected by the detection processing unit 21a, the passage of time from the start of the live performance, or an instruction from the server device 30. The augmented reality processing unit 21b can display an augmented reality image corresponding to the marker on the display unit 12 of the AR glasses 10 by, for example, reading out augmented reality data corresponding to a marker detected by the detection processing unit 21a from the storage unit 22 and transmitting an image in which the position and orientation of the image to be displayed have been adjusted in response to the position of the marker, etc., to the AR glasses 10. In this case, the image displayed on the AR glasses 10 changes in response to the positional relationship between the marker and the spectator 2 wearing the AR glasses 10.

In addition, when a predetermined time has passed since the start of the live performance or when a predetermined display command is given from the server device 30, the augmented reality processing unit 21b reads out augmented reality data corresponding to the passage of time or the display command from the storage unit 22 and transmits the image to be displayed to the AR glasses 10, thereby displaying an augmented reality image corresponding to the passage of time or the display command on the display unit 12 of the AR glasses 10. In this case, the image displayed may be displayed regardless of the presence or absence of a marker, or may be displayed in association with a marker. When the terminal device 20 displays an image regardless of the presence or absence of a marker, the image displayed on the AR glasses 10 is displayed regardless of the position of the audience 2, so that all the audience members 2 can view the same image through the AR glasses 10. For example, by the terminal device 20 displaying an augmented reality image in which a character appears to be present 5 m away from the audience member 2 regardless of the marker, all the audience members 2 can view the character present 5 m away from the audience member 2 through the AR glasses 10.

In addition, similar to the image display described above, the augmented reality processing unit 21b may perform processing to output the augmented reality sound contained in the augmented reality data from the audio output unit 17 of the AR glasses 10 in response to, for example, a marker detected by the detection processing unit 21a, the elapsed time from the start of the live performance, or an instruction from the server device 30.

The augmented reality processing unit 21b may also perform processing such as recording live images seen by the audience 2 through the AR glasses 10 and recording audio heard while wearing the AR glasses 10. The augmented reality processing unit 21b performs recording by storing in the memory unit 22 image data in which an image captured by the camera 13 of the AR glasses 10 and an image displayed on the display unit 12 of the AR glasses 10 are superimposed. The augmented reality processing unit 21b also performs recording by storing in the memory unit 22 audio data in which audio captured by the microphone of the AR glasses 10 or the terminal device 20 and audio output from the audio output unit 17 of the AR glasses 10 are superimposed.

Figure 6 is a block diagram showing the configuration of the AR glasses 10 according to this embodiment. The AR glasses 10 according to this embodiment are configured to include a display unit 12, a camera 13, a processing unit 15, a wired communication unit 16, and an audio output unit 17. The display unit 12 displays an image provided by the processing unit 15. Although one display unit 12 is shown in Figure 6, in this embodiment, the AR glasses 10 have two display units 12 for the right eye and the left eye. However, the AR glasses 10 may have a continuous display unit 12 on the left and right. In this embodiment, the display unit 12 is transparent (translucent), and the audience 2 can view the real space through the display unit 12. However, the display unit 12 may be non-transparent, and in this case, the terminal device 20 may display on the display unit 12 an image of the real space captured by the camera 13 superimposed with an augmented reality image.

The camera 13 is configured using, for example, a lens and an image sensor. The camera 13 is mounted on the frame 11 of the AR glasses 10 so as to capture an image in front of the spectator 2 wearing the AR glasses 10. The camera 13 may have, for example, an angle of view of 180° or more, and is preferably capable of capturing at least the visual field range of the spectator 2. The AR glasses 10 may have multiple cameras 13. The camera 13 provides the captured image to the processing unit 15.

The processing unit 15 is configured using an arithmetic processing device such as a CPU or an MPU. The processing unit 15 performs processing such as transmitting images captured by the camera 13 to the terminal device 20 and displaying augmented reality images received from the terminal device 20 on the display unit 12.

The wired communication unit 16 transmits and receives data to and from the terminal device 20 connected via the communication cable 14, for example, according to the USB communication standard. The wired communication unit 16 transmits images captured by the camera 13 provided by the processing unit 15 to the terminal device 20, and provides augmented reality images received from the terminal device 20 to the processing unit 15.

The audio output unit 17 is a so-called speaker, and outputs various sounds according to the processing of the processing unit 15.

Although not shown in the figure, the AR glasses 10 may be equipped with a GPS receiver that receives GPS signals and a microphone that picks up surrounding sounds.

Figure 7 is a block diagram showing the configuration of the server device 30 according to this embodiment. The server device 30 according to this embodiment is configured to include a processing unit 31, a storage unit 32, and a communication unit 33. Note that in this embodiment, the processing is described as being performed by one server device 30, but the processing may be distributed among multiple server devices 30.

The processing unit 31 is configured using a processor such as a CPU, MPU, or GPU (Graphics Processing Unit), a ROM, a RAM, etc. The processing unit 31 reads and executes the server program 32a stored in the storage unit 32, thereby performing various processes such as sending an augmented reality image to the terminal device 20 of the audience 2 and managing the live performance.

The storage unit 32 is configured using a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for the processing of the processing unit 31. In this embodiment, the storage unit 32 stores a server program 32a executed by the processing unit 31 and augmented reality data 32b of the augmented reality to be transmitted to the terminal device 20.

In this embodiment, the server program 32a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk, and the server device 30 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. However, the server program 32a may be written to the storage unit 32, for example, during the manufacturing stage of the server device 30. Also, for example, the server program 32a may be distributed by another remote server device or the like and acquired by the server device 30 through communication. For example, the server program 32a may be read from the recording medium 99 by a writing device and written to the storage unit 32 of the server device 30. The server program 32a may be provided in a form distributed via a network, or may be provided in a form recorded on the recording medium 99.

The augmented reality data 32b stored in the memory unit 32 is sent to the terminal device 20 of each spectator 2 and is data such as an augmented reality image displayed on the display unit 12 of the AR glasses 10 connected to this terminal device 20 and audio output from the audio output unit 17. The augmented reality data 32b is created in advance, for example, by the organizer of the live event and stored in the memory unit 32 of the server device 30. The augmented reality data 32b may be created by the server device 30 or by a separate device.

The communication unit 33 communicates with various devices via networks including a mobile phone communication network and the Internet. In this embodiment, the communication unit 33 communicates with the terminal devices 20 of multiple spectators 2 participating in the live performance. The communication unit 33 transmits data provided by the processing unit 31 to other devices, and provides data received from other devices to the processing unit 31.

The storage unit 32 may be an external storage device connected to the server device 30. The server device 30 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The server device 30 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that accepts operation input, or a display unit that displays images.

In addition, in the processing unit 31 of the server device 30 according to this embodiment, the processing unit 31 reads and executes the server program 32a stored in the memory unit 32, whereby the data transmission processing unit 31a and the live management processing unit 31b are realized in the processing unit 31 as software functional units.

The data transmission processing unit 31a performs a process of transmitting the augmented reality data 32b stored in the storage unit 32 to the terminal device 20 of the audience member 2. In this embodiment, the data transmission processing unit 31a transmits the augmented reality data 32b to the terminal device 20 before the start of the live performance. A download code for the augmented reality data 32b is printed on the ticket for the live performance, and when the terminal device 20 receives an input of this code, it transmits the received code and requests the server device 30 to transmit the augmented reality data 32b. In response to a transmission request from the terminal device 20, the data transmission processing unit 31a of the server device 30 reads out the augmented reality data 32b of the live performance corresponding to the code from the storage unit 32, and transmits the read augmented reality data 32b to the terminal device 20 that made the request. Note that the above-mentioned transmission method and transmission timing of the augmented reality data 32b are merely examples and are not limited thereto. The data transmission processing unit 31a may transmit images required during the live performance to the terminal device 20 in real time, for example.

The live management processing unit 31b performs processing to manage the display timing of the augmented reality image during the live. For example, the live management processing unit 31b notifies the terminal device 20 of the start and end of the live. The live management processing unit 31b also measures the time that has passed since the start of the live, and when a predetermined time has passed, issues a display command for a predetermined augmented reality image to the terminal device 20. The live management processing unit 31b also determines the display timing of the augmented reality image in response to the operation of the organizer of the live, and transmits a display command to the terminal device 20 when it is determined that the timing to display the image has arrived. The live management processing unit 31b may also manage the image to be displayed on the display device 5 installed at the live venue, for example. When audio output of the augmented reality is performed, the live management processing unit 31b may similarly manage the timing of the audio output of the augmented reality.

The live management processing unit 31b may also determine the display timing, etc., based on an image of the stage or audience seats of the venue captured by a camera installed in the venue. For example, the live management processing unit 31b performs a process to detect spectators 2 waving penlights from an image captured of the audience seats of the venue, and can determine that it is the timing to display a specified augmented reality image when the number of detected spectators 2 exceeds a threshold value, or when the number of detected spectators relative to the total number of spectators exceeds a threshold value, etc.

The live management processing unit 31b may also determine the display timing, etc., based on user operations received by the AR glasses 10 or the terminal device 20 during the live performance. For example, the live management processing unit 31b acquires evaluation information regarding the live performance from the terminal devices 20 of the spectators 2 participating in the live performance, and tally up the acquired evaluations. The live management processing unit 31b can compare the tally up results with a predetermined threshold, and determine that it is time to display a specified augmented reality image if an evaluation higher than the threshold is obtained.

<Augmented reality processing>
In the information processing system according to the present embodiment, the server device 30 transmits augmented reality data required for a live performance in advance to the terminal device 20 of the audience member 2 who has purchased a ticket for the live performance. For example, a download code for the augmented reality data is printed on the ticket for the live performance, and the audience member 2 starts an application (program 22a) on the terminal device 20 and inputs the code. The terminal device 20 that has received the input of the code transmits the code to the server device 30 and requests the transmission of the augmented reality data. In response to the request from the terminal device 20, the server device 30 reads out the augmented reality data required for the live performance corresponding to the code from the storage unit 32 and transmits it to the terminal device 20 that made the request. The code may be transmitted, for example, by e-mail, or may be provided, for example, as a barcode or two-dimensional code.

Figure 8 is a flowchart showing the procedure of the process of acquiring augmented reality data performed by the terminal device 20 according to this embodiment. The processing unit 21 of the terminal device 20 according to this embodiment accepts input of a live performance code printed on a ticket for a live performance or the like from a user (step S1). The processing unit 21 transmits the received code to the server device 30 and requests transmission of the augmented reality data required for displaying an augmented reality image (step S2).

Then, the processing unit 21 determines whether or not augmented reality data has been received from the server device 30 (step S3). If augmented reality data has not been received (S3: NO), the processing unit 21 waits until augmented reality data is received. If augmented reality data has been received (S3: YES), the processing unit 21 stores the received augmented reality data in the storage unit 22 (step S4) and ends the processing.

Figure 9 is a flowchart showing the procedure of the transmission process of augmented reality data performed by the server device 30 according to this embodiment. The data transmission processing unit 31a of the processing unit 31 of the server device 30 according to this embodiment determines whether or not a request for transmission of augmented reality data has been given from the terminal device 20 (step S11). If a request has not been given from the terminal device 20 (S11: NO), the data transmission processing unit 31a waits until a request for transmission of augmented reality data is given from the terminal device 20. If a request for transmission of augmented reality data is given (S11: YES), the data transmission processing unit 31a reads out from the storage unit 32 augmented reality data of the augmented reality used in the live corresponding to the code transmitted together with the request from the terminal device 20 (step S12). The data transmission processing unit 31a transmits the read augmented reality data to the requesting terminal device 20 (step S13) and ends the process.

On the day of the live performance. After arriving at the live performance venue, the audience member 2 connects the AR glasses 10 to the terminal device 20, wears them, and enters the above code into the terminal device 20 to wait for the live performance to begin. When the time for the live performance to begin is reached, the server device 30 notifies the terminal devices 20 of all audience members 2 (terminal devices 20 into which the correct code has been entered) that the live performance has begun. Upon receiving this notification, the terminal devices 20 begin the process of displaying an augmented reality image on the display unit 12 of the AR glasses 10.

The terminal device 20 acquires an image captured by the camera 13 of the AR glasses 10, and detects markers included in the captured image. The terminal device 20 may also detect markers included in audio acquired by the microphone of the AR glasses 10 or its own microphone. In this embodiment, examples of markers detected by the terminal device 20 include the following:
(1) Props, stage equipment, or costumes worn by Artist 1 installed at the venue, or any letters, symbols, images, etc. drawn on them.
(2) Characters, symbols, images, etc. displayed on a display device such as the display device 5 installed at the venue.
(3) Artist 1's movements, poses, etc.
(4) Specified music or audio, etc.

Regarding (1) and (2) above, the terminal device 20 can determine that a marker exists when an image previously registered as a marker matches an image captured by the camera 13 (or a part of it). Regarding (3) above, the terminal device 20 extracts a person (artist 1) captured in the image captured by the camera 13 and determines the movement or pose of the extracted person. The terminal device 20 can determine that a marker exists when the determined movement or pose matches a movement or pose registered as a marker. Regarding (4) above, the terminal device 20 performs processing such as voice recognition based on the voice captured by the microphone, and can determine that an audio marker exists when the voice contains a predetermined song, word, etc.

When the terminal device 20 detects a marker from an image captured by the camera 13, it reads out an augmented reality image associated with the detected marker from the augmented reality data 22b in the memory unit 22. The terminal device 20 generates an image for display by adjusting the orientation and size of the augmented reality image to be displayed according to the position and size of the detected marker, and transmits the generated image to the AR glasses 10. The AR glasses 10 receive the image from the terminal device 20 and display the augmented reality image on the display unit 12. The augmented reality image displayed according to the marker may have a different orientation or size for each audience member 2 depending on the position of the audience member 2 in the live venue. The terminal device 20 may also output the augmented reality sound associated with the detected marker from the audio output unit 17 of the AR glasses 10.

In addition, in this embodiment, the terminal device 20 can display a predetermined augmented reality image on the AR glasses 10 without detecting a marker (the same applies to the augmented reality audio output). For example, the terminal device 20 displays a predetermined image on the AR glasses 10 at a predetermined timing after the start of the live performance. Also, for example, the terminal device 20 displays a predetermined image on the AR glasses 10 in response to a command from the server device 30. The augmented reality image displayed without detecting a marker is displayed regardless of the position of each audience member 2 in the live performance venue, and the same augmented reality image is displayed on the AR glasses 10 of all audience members 2. In other words, if the terminal device 20 displays an augmented reality image on the AR glasses 10 in which a character appears to be present 5 m away regardless of the marker, for example, an image of a character present 5 m away from the audience member 2 is displayed on the AR glasses 10 of all audience members 2.

Figure 10 is a flowchart showing the procedure of the live management process performed by the server device 30 according to this embodiment. Note that this flowchart describes the case where an augmented reality image is displayed, and the case where an augmented reality sound is output is not described. The live management processing unit 31b of the processing unit 31 of the server device 30 according to this embodiment notifies the terminal device 20 of the audience 2 participating in the live that the live has started, for example, in response to an operation by the organizer of the live, or when the start time of the live has arrived (step S21). The live management processing unit 31b starts measuring the elapsed time from the start of the live using its own timer function, etc. (step S22).

Based on a predetermined live schedule and the measurement result of the elapsed time, the live management processing unit 31b determines whether it is time to display the augmented reality image (step S23). The live schedule is prepared in advance, for example, by the organizer of the live, and the schedule data is stored in the server device 30. If it is time to display the augmented reality image (S23: YES), the live management processing unit 31b sends a display command to the terminal device 20 of the audience member 2, for example with information identifying the image to be displayed (step S24), and returns the process to step S23.

If it is not time to display the augmented reality image (S23: NO), the live management processing unit 31b determines whether it is time to end the live based on the predetermined live schedule and the measurement result of the elapsed time (step S25). If it is not time to end the live (S25: NO), the live management processing unit 31b returns the process to step S23. If it is time to end the live (S25: YES), the live management processing unit 31b notifies the terminal device 20 of the audience member 2 that the live has ended (step S26) and ends the process.

Figure 11 is a flowchart showing the procedure for the display process of an augmented reality image performed by the terminal device 20 according to this embodiment. Note that this flowchart describes the case where an augmented reality image is displayed, and a description of the case where augmented reality sound is output is omitted. The processing unit 21 of the terminal device 20 according to this embodiment determines whether or not it has been notified of the start of a live performance by the server device 30 (step S41). If it has not been notified of the start of a live performance (S41: NO), the processing unit 21 waits until it is notified of the start of the live performance.

When notified of the start of a live performance (S41: YES), the detection processing unit 21a of the processing unit 21 acquires an image captured by the camera 13 of the AR glasses 10, performs a marker detection process, and determines whether a specific marker is present in the captured image (step S42). If a marker is present (S42: YES), the augmented reality processing unit 21b of the processing unit 21 reads out augmented reality data of the augmented reality corresponding to the detected marker from the storage unit 22 (step S43). The augmented reality processing unit 21b adjusts the position, orientation, size, etc. of the augmented reality image displayed by the AR glasses 10 according to the position of the detected marker, etc. (step S44). The augmented reality processing unit 21b transmits the adjusted augmented reality image to the AR glasses 10 (step S45), displays the augmented reality image on the AR glasses 10, and proceeds to step S49.

If no marker is present in the captured image (S42: NO), the augmented reality processing unit 21b determines whether or not a display command for an augmented reality image has been given from the server device 30 (step S46). If a display command has not been given (S46: NO), the augmented reality processing unit 21b proceeds to step S49. If a display command has been given (S46: YES), the augmented reality processing unit 21b reads out augmented reality data corresponding to the display command from the server device 30 from the storage unit 22 (step S47). Based on the read out augmented reality data, the augmented reality processing unit 21b transmits the augmented reality image to be displayed to the AR glasses 10 (step S48), displays the augmented reality image on the AR glasses 10, and proceeds to step S49.

The processing unit 21 determines whether or not the server device 30 has notified the end of the live performance (step S49). If the server device 30 has not notified the end of the live performance (S49: NO), the processing unit 21 returns to step S42 and repeats the above-mentioned process. If the server device 30 has notified the end of the live performance (S49: YES), the processing unit 21 ends the process.

<Summary>
The terminal device 20 according to the present embodiment configured as described above detects markers installed at a live venue based on images captured by the camera 13 of the AR glasses 10, and performs processing to display an augmented reality image corresponding to the marker detected during the live performance on the display unit 12 of the AR glasses 10. As a result, an audience member 2 wearing the AR glasses 10 at a live performance venue can experience augmented reality in which images of characters or the like are superimposed around an artist 1 performing at a live performance, for example. Therefore, the information processing system according to the present embodiment is expected to improve the entertainment value of live performances and provide new live performance services using augmented reality technology.

The terminal device 20 according to this embodiment detects, as a marker, a specific movement or pose of the artist 1 performing at the live show based on an image captured by the camera 13 of the AR glasses 10, and displays, on the display unit 12 of the AR glasses 10, a specific augmented reality image corresponding to the detected movement or pose. This allows the information processing system to display an augmented reality image in conjunction with the movement or pose of the artist 1 performing at the live show, which is expected to further improve the entertainment value of the live show.

The terminal device 20 according to this embodiment detects a specific image displayed on a display device 5 or the like installed at a live venue as a marker based on an image captured by the camera 13 of the AR glasses 10, and displays an augmented reality image corresponding to the detected marker on the display unit 12 of the AR glasses 10. This makes it possible to change the augmented reality image displayed on the display unit 12 of the AR glasses 10 in response to changes in the display of the display device 5 or the like.

In addition, when a predetermined display timing corresponding to the elapsed time from the start of the live performance, etc. is reached, the terminal device 20 according to this embodiment displays an augmented reality image corresponding to this display timing on the display unit 12 of the AR glasses 10 in response to a display command given by the server device 30. Note that the determination of whether or not the display timing has been reached may be made by the terminal device 20, rather than the server device 30. This allows the terminal device 20 to display an augmented reality image on the AR glasses 10 corresponding to the schedule of the live performance, etc.

The terminal device 20 according to this embodiment also displays a predetermined augmented reality image on the display unit 12 of the AR glasses 10, regardless of whether a marker is detected, so that the images are arranged in the same way on the AR glasses 10 worn by each of the multiple spectators attending the live performance. This allows the information processing system to display the same augmented reality image in the same arrangement on the AR glasses 10 of each spectator 2, regardless of the position of each spectator 2 at the live venue.

<Modification>
In the information processing system according to the modified example, the augmented reality image displayed on the AR glasses 10 is different depending on the price paid by the audience member 2. For example, audience seats at a live venue are divided into S seats, A seats, B seats, etc., each of which has a different price. Alternatively, the price of the augmented reality may be set separately from the price of the audience seat. In the information processing system according to the modified example, a higher level of augmented reality (for example, more creative, more impressive, more flashy, or with a larger amount of augmented reality image data, etc.) is provided to the audience member 2 who has paid a higher price.

In the information processing system according to the modified example, for example, a ticket corresponding to the price of the fee paid is distributed to a spectator 2 who has paid a fee for a live augmented reality event. The ticket contains a code for downloading augmented reality data of the augmented reality from the server device 30, but in the information processing system according to the modified example, a code corresponding to the price is written on the ticket. By inputting the code written on the ticket into the terminal device 20 and downloading the augmented reality data from the server device 30, the augmented reality data of the augmented reality corresponding to the price is downloaded to the terminal device 20. The server device 30 associates multiple augmented reality data corresponding to the prices with the codes and stores them in a database, and reads out the augmented reality data corresponding to the code given from the terminal device 20 together with a request to transmit the augmented reality data, and transmits it to the requesting terminal device 20.

As a result, each terminal device 20 stores augmented reality data according to the price paid by the audience member 2. During the live performance, augmented reality images are displayed based on this augmented reality data, allowing the audience member 2 wearing the AR glasses 10 to experience an augmented reality live performance according to the price.

The method of providing the audience 2 with an augmented reality live experience according to the price is not limited to the method of using the download code described in this modified example. For example, the AR glasses 10 may be lent to the audience 2 at a live venue, and the AR glasses 10 may be lent according to the price paid by the audience 2. In this case, for example, the AR glasses 10 with functions or performance according to the price may be lent to the audience 2. For example, the AR glasses 10 with the same functions or performance may be lent to the audience 2, but an individual ID according to the price may be set for each AR glass 10, and an augmented reality image according to the price may be transmitted from the server device 30 to the terminal device 20 (or the AR glasses 10) according to this ID. For example, in the case where the AR glasses 10 are configured to store augmented reality data, or in the case where the terminal device 20 is lent together with the AR glasses 10, data of a virtual reality image according to the price may be downloaded in advance to the AR glasses 10 or the terminal device 20 and lent.

<Embodiment 2>
In the information processing system according to the second embodiment, the AR glasses 10 or the terminal device 20 recognize a real space such as a live music venue, and an augmented reality image associated with the recognized real space is displayed on the AR glasses 10. This can be said to be a technology that displays an augmented reality image by regarding the real space as a marker, and displays the augmented reality image using a technology that identifies a position from an image called a Virtual Positioning System (VPS).

Fig. 12 is a schematic diagram for explaining the processing performed by the information processing system according to the second embodiment in the preparation stage for a live performance. In the preparation stage for a live performance, for example, the organizer of the live performance uses a camera 40 to capture images of the live performance venue before the live performance, where the artist 1 and the audience 2 are not present. Note that the camera 40 may be a dedicated device for capturing images, such as a video camera or digital camera, or may be a device such as a smartphone or tablet terminal device with a capture function. Images (moving images or still images) of the live performance venue captured by the camera 40 are transmitted to the server device 30 via communication.

The server device 30 according to the second embodiment generates three-dimensional information that reproduces the live venue as a three-dimensional virtual space based on the images captured by the camera 40. At this time, the server device 30 performs image processing to extract edges, corners, etc. from the two-dimensional images captured by the camera 40, for example, and can calculate the coordinates of multiple points in the three-dimensional virtual space that indicate the positions of corners, etc. of objects that exist in the real space, and use this as the three-dimensional information. The server device 30 stores the created three-dimensional information corresponding to the live venue in the storage unit 32.

In addition, before the live performance, the organizers of the live performance create the augmented reality images to be displayed during the live performance, and set the location in the live performance venue where the augmented reality images will be displayed. In the second embodiment, the organizers perform this work using a terminal device 50 such as a personal computer or a tablet terminal device.

In FIG. 12, the camera 40 and the terminal device 50 are separate devices, but this is not limited thereto. If the terminal device 50 is a device with a shooting function, the terminal device 50 may take pictures of the live venue. In FIG. 12, the camera 40 directly transmits the images it has taken to the server device 30, but this is not limited thereto. The terminal device 50 may obtain the images it has taken, and then transmit the images to the server device 30.

The terminal device 50 acquires three-dimensional information of the live venue generated by the server device 30 based on the image captured by the camera 40, and displays a three-dimensional virtual space that recreates the live venue on the display unit based on the acquired three-dimensional information. The organizer or the like also creates virtual reality images to be displayed during the live performance in advance and stores them in the memory unit of the terminal device 50. The terminal device 50 accepts operations such as designation of coordinates in the displayed three-dimensional virtual space from the organizer or the like, and sets the accepted coordinates as the position at which the augmented reality image is to be displayed. If there are multiple augmented reality images, the terminal device 50 may accept the display position for each image. The terminal device 50 creates a display position setting that associates the augmented reality image with its display position, and transmits the augmented reality image and the display position setting to the server device 30.

FIGS. 13 and 14 are schematic diagrams showing an example of a setting screen for the display position of an augmented reality image displayed by the terminal device 50. Note that FIG. 13 shows the state before the augmented reality image is placed, and FIG. 14 shows the state after the augmented reality image is placed. The setting screen displayed by the terminal device 50 includes a virtual space display area 101, an augmented reality image selection area 102, a performance selection area 103, and an effect selection area 104.

The virtual space display area 101 of the setting screen is an area that displays the live venue recreated as a three-dimensional virtual space based on the three-dimensional information of the live venue generated by the server device 30. The terminal device 50 acquires the three-dimensional information of the live venue from the server device 30, and recreates the three-dimensional virtual space of the live venue based on the acquired three-dimensional information.

The augmented reality image selection area 102 of the setting screen is provided below the virtual space display area 101, and is an area in which one or more augmented reality images created in advance are displayed in a list. The user can select an image to be placed from the images displayed in the list in the augmented reality image selection area 102.

The performance selection area 103 of the setting screen is provided in the form of so-called tabs above the virtual space display area 101. In this example, multiple tabs labeled with "Opening performance," "First song performance," "Encore promotion performance," etc. are provided as the performance selection area 103. By selecting a tab in the performance selection area 103, the user can select the timing of the live performance at which the augmented reality image for which the display position is to be set will be displayed.

The effect selection area 104 is provided to the right of the virtual space display area 101, and is an area for selecting an effect (special effect) to be applied when the augmented reality image is displayed. One or more icons corresponding to effects are displayed side by side in the effect selection area 104. The user can select one or more icons displayed in the effect selection area 104, and the effect corresponding to the selected icon is applied when the augmented reality image is displayed.

When the terminal device 50 is displaying the setting screen, the user can select an image to be placed, and then specify any display position in the virtual space display area 101 to set the display position of the selected image. The terminal device 50 accepts the specification of the display position (coordinates) in the virtual space display area 101, and stores the selected augmented reality image and the display position in association with each other. As shown in FIG. 14, the terminal device 50 displays the augmented reality image with the set display position in the virtual space display area 101 in association with the display position. The terminal device 50 may also store the augmented reality image and the display position in association with information such as the performance selected in the performance selection area 103 and the effect selected in the effect selection area 104.

Note that the setting screens shown in Figures 13 and 14 are examples and are not limited to these.

By the above process, the organizer or the like prepares the virtual reality images to be displayed during the live performance. After the preparations are completed, in the information processing system according to the second embodiment, the server device 30 distributes data including the virtual reality images and the display position settings of these images to the terminal devices 20 used by the audience members of the live performance in advance. The terminal devices 20 receive the information distributed from the server device 30, and store the received information in the storage unit 22 at least until the end of the live performance. Note that in the second embodiment, the augmented reality images and information such as the display position settings are distributed in advance to the audience members' terminal devices 20, but this is not limited thereto, and the information may be distributed in real time during the live performance.

Figure 15 is a schematic diagram for explaining the processing performed by the information processing system according to the second embodiment during a live performance. In the information processing system according to the second embodiment, the timing for displaying an augmented reality image on the AR glasses 10 of the audience 2 during a live performance is determined by the operation of the terminal device 50 by the organizer or the like.

The terminal device 50 according to the second embodiment reads out from a storage unit, for example, a plurality of augmented reality images to be displayed during a live performance that have been created and stored in advance, and displays the read out plurality of images in a predetermined order on the display unit in a list. The predetermined order in which the plurality of images are arranged can be, for example, the order in which they are to be displayed live. The terminal device 50 accepts, from the organizer or the like, an operation to select from the images displayed in the list, and an operation to instruct the display of the selected image.

When the terminal device 50 receives an operation to instruct the display of an augmented reality image, it transmits an instruction to display the augmented reality image to the server device 30. This display instruction may include information such as an image ID that is uniquely assigned to the augmented reality image. Having received the instruction to display the augmented reality image from the terminal device 50, the server device 30 transmits an instruction to display the augmented reality image to the terminal devices 20 of all spectators 2 participating in the live performance.

The terminal device 20 used by the audience member 2 captures images of the live venue using the camera 13 of the AR glasses 10, and generates three-dimensional information that reproduces the live venue as a three-dimensional virtual space based on the captured images. The three-dimensional information generated by the terminal device 20 at this time is substantially the same as the three-dimensional information of the live venue generated by the server device 30 based on images captured by the camera 40 during the preparation stage of the live performance shown in FIG. 12.

The terminal device 20 transmits the generated three-dimensional information to the server device 30, and the server device 30 judges whether the three-dimensional information it has created matches the three-dimensional information created by the terminal device 20. Note that this judgment does not require that the two pieces of three-dimensional information match perfectly, but only that they match to the extent that they can be judged to be three-dimensional images based on the same real space.

If it is determined that the three-dimensional information matches, the server device 30 notifies the terminal device 20 of this fact and transmits information necessary to match the coordinates of its own three-dimensional information with the coordinates of the three-dimensional information of the terminal device 20. The information transmitted at this time may include, for example, information such as the origin position of the coordinates of the three-dimensional virtual space, the x-axis direction, the y-axis direction, and the z-axis direction. The terminal device 20 receives this information and performs a process to correct the coordinates of its own three-dimensional information, making it possible to treat the three-dimensional virtual space of the live venue recognized by the server device 30 and the three-dimensional virtual space of the live venue recognized by the terminal device 20 in the same coordinate system.

After the live performance begins, the terminal device 20 of the audience member 2 receives an instruction to display an augmented reality image from the server device 30. Based on information such as the image ID included in the received display instruction, the terminal device 20 reads out the image to be displayed from among the augmented reality images previously stored in the memory unit 22. The terminal device 20 also reads out the display position setting associated with this image. The terminal device 20 provides the read-out image and the coordinate information set in the display position setting to the AR glasses 10. The AR glasses 10 displays the image provided by the terminal device 20 on the display unit 12 based on the coordinate information provided by the terminal device 20. This allows the user wearing the AR glasses 10 to experience augmented reality in which an augmented reality image is superimposed on the live performance venue.

Fig. 16 is a block diagram showing the configuration of a terminal device 50 according to the second embodiment. The terminal device 50 according to the second embodiment is configured to include a processing unit (processor) 51, a memory unit (storage) 52, a communication unit (transceiver) 53, a display unit (display) 54, and an operation unit 55. The terminal device 50 may be configured using a general-purpose information processing device such as a personal computer or a tablet-type terminal device.

The processing unit 51 is configured using an arithmetic processing device such as a CPU or an MPU, a ROM, a RAM, etc. The processing unit 51 reads out and executes a program 52a stored in the storage unit 52, thereby performing various processes such as accepting operations such as creating and editing an augmented reality image to be displayed during a live performance, accepting settings for the display position of the augmented reality image, and accepting instructions to display an augmented reality image during a live performance.

The storage unit 52 is configured using a storage device such as a hard disk or a flash memory. The storage unit 52 stores various programs executed by the processing unit 51 and various data required for the processing of the processing unit 51. In this embodiment, the storage unit 52 stores the programs 52a executed by the processing unit 51 and the augmented reality image data 52b.

The program 52a stored in the storage unit 22 is an application program that allows the organizer of the live event to perform operations such as generating and displaying an augmented reality image. In the embodiment, the program 52a is distributed by a remote server device, etc., and the terminal device 50 acquires it through communication and stores it in the storage unit 52. However, the program 52a may be written to the storage unit 52, for example, during the manufacturing stage of the terminal device 50. For example, the program 52a may be read by the terminal device 50 from a recording medium 97 such as a memory card or an optical disk and stored in the storage unit 52. For example, the program 52a may be read by a writing device from the recording medium 97 and written to the storage unit 52 of the terminal device 50. The program 52a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 97.

The image data 52b stored in the memory unit 52 is data for augmented reality images to be displayed on the AR glasses 10 during a live performance. These image data may be manually created in the terminal device 50, or may be created in another device and acquired by the terminal device 50 via communication or the like and stored in the memory unit 52. The augmented reality images may be either two-dimensional or three-dimensional images, and may be either still images or moving images.

The communication unit 53 communicates with various devices, for example, via a mobile phone communication network such as a fourth generation mobile communication system (4G) or a fifth generation mobile communication system (5G), or wireless communication such as a wireless LAN (Local Area Network). In this embodiment, the communication unit 53 wirelessly transmits and receives data with the server device 30. The communication unit 53 transmits data provided by the processing unit 51 to the server device 30, and provides data received from the server device 30 to the processing unit 51. Note that the communication unit 53 may perform wired communication instead of wireless communication.

The display unit 54 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 51. Note that a display device such as a liquid crystal display may be provided separately from the terminal device 50, in which case the terminal device 50 may not include the display unit 54, and may output image signals, etc. to a separate display device to display images.

The operation unit 55 accepts user operations and notifies the processing unit 51 of the accepted operations. For example, the operation unit 56 may be configured using input devices such as a keyboard or a mouse, and these input devices may be configured to be removable from the terminal device 50. The operation unit 55 may also be an input device such as a touch panel provided on the surface of the display unit 54.

In addition, in the terminal device 50 according to this embodiment, the processing unit 51 reads and executes the program 52a stored in the memory unit 52, whereby the setting processing unit 51a and the display instruction unit 51b are realized in the processing unit 51 as software functional units.

The setting processing unit 51a performs a process of accepting settings for the display position of the augmented reality image. The setting processing unit 51a acquires three-dimensional information of the live venue from the server device 30, and displays an image of a three-dimensional virtual space that reproduces the live venue based on the acquired three-dimensional information on the display unit 54. The setting processing unit 51a also displays a list of one or more images stored as image data 52b in the memory unit 52 near the display area of the three-dimensional virtual space of the live venue. The setting processing unit 51a accepts the selection of an image to be set from the images displayed in the list, and accepts the specification of the display position in the live venue of the displayed three-dimensional virtual space. The setting processing unit 51a creates a display position setting that associates information on the three-dimensional coordinates of the specified display position with the selected image, stores it in the memory unit 52, and transmits it to the server device 30.

The display instruction unit 51b performs a process of receiving instructions from the organizer of the live performance, etc., to display augmented reality images on the audience's AR glasses 10 during the live performance. For example, the display instruction unit 51b reads out the augmented reality images to be displayed during the live performance from the memory unit 52 and displays them as a list on the display unit 54, and also displays buttons and the like for receiving display instructions for each image. The display instruction unit 51b receives an instruction to display an image by receiving an operation on the operation unit 55, and transmits a display instruction including information such as the ID of the image to be displayed to the server device 30.

Figure 17 is a flowchart showing the procedure of processing performed by the server device 30 according to the second embodiment. The processing unit 31 of the server device 30 according to the second embodiment acquires an image of the live venue captured by the camera 40, for example, via communication (step S61). The processing unit 31 generates three-dimensional information that reproduces the live venue as a three-dimensional virtual space based on the acquired image (step S62). The processing unit 31 stores the generated three-dimensional information of the live venue in the storage unit 32 (step S63).

Then, the processing unit 31 determines whether or not a request for transmission of three-dimensional information has been received from the terminal device 50 used by the organizer of the live performance, etc., in order to display an image that recreates the live venue as a three-dimensional virtual space (step S64). If no request has been received (S64: NO), the processing unit 31 waits until a request is received from the terminal device 50. If a request has been received (S64: YES), the processing unit 31 transmits the three-dimensional information of the live venue stored in the memory unit 32 to the requesting terminal device 50 (step S65).

Then, the processing unit 31 determines whether or not setting information associating the augmented reality image with its display position has been received from the terminal device 50 that transmitted the three-dimensional information (step S66). If setting information has not been received (S66: NO), the processing unit 31 waits until it receives setting information. If setting information relating to the display position of the virtual reality image has been received (S66: YES), the processing unit 31 stores the received setting information in the storage unit 32 (step S67) and ends the process.

Figure 18 is a flowchart showing the procedure of processing performed by the terminal device 50 according to the second embodiment in the preparation stage for a live performance. The setting processing unit 51a of the processing unit 51 of the terminal device 50 according to the second embodiment requests the server device 30 to provide three-dimensional information for displaying a live performance venue reproduced as a three-dimensional virtual space (step S81). The setting processing unit 51a determines whether or not it has received three-dimensional information of the live performance venue transmitted from the server device 30 in response to the request in step S81 (step S82). If it has not received three-dimensional information (S82: NO), the setting processing unit 51a waits until it receives the three-dimensional information.

When three-dimensional information is received from the server device 30 (S82: YES), the setting processing unit 51a displays a setting screen on the display unit 54 for accepting settings for the display position of the augmented reality image (step S83). On this setting screen, for example, an image of a live venue reproduced as a three-dimensional virtual space and one or more augmented reality images created in advance are displayed side by side.

The setting processing unit 51a accepts the selection of the augmented reality image to be set on the setting screen displayed on the display unit 54 based on an operation on the operation unit 55 (step S84). The setting processing unit 51a also accepts the display position of the selected image as coordinate information in a three-dimensional virtual space (step S85). The setting processing unit 51a transmits a display position setting that associates an ID or the like for identifying the augmented reality image with coordinate information of the position at which this image is to be displayed to the server device 30 (step S86), and ends the process.

Fig. 19 is a flowchart showing the procedure of processing performed by the terminal device 50 according to embodiment 2 during a live performance. The display instruction unit 51b of the processing unit 51 of the terminal device 50 according to embodiment 2 displays on the display unit 54 a display instruction acceptance screen on which, for example, an augmented reality image to be displayed during the live performance and buttons for accepting operations to display the image are arranged (step S101).

The display instruction unit 51b determines whether an operation to display an augmented reality image has been performed, for example, by determining whether an operation has been performed on a button displayed together with the image on the display instruction reception screen (step S102). If an operation to display an augmented reality image has not been performed (S102: NO), the display instruction unit 51b waits until an operation is performed. If an operation to display an augmented reality image has been performed (S102: YES), the display instruction unit 51b transmits a display instruction including information such as the ID of the image to be displayed to the server device 30 (step S103).

The display instruction unit 51b determines whether to end the reception of display instructions based on whether an operation to end the display of the display instruction acceptance screen has been received, for example, when the live performance has ended (step S104). If the operation is not to be ended (S104: NO), the display instruction unit 51b returns the process to step S102. If the operation is to be ended (S104: YES), the display instruction unit 51b ends the process by, for example, hiding the display instruction acceptance screen.

Figure 20 is a flowchart showing the procedure of processing performed by the terminal device 20 according to embodiment 2. The processing unit 21 of the terminal device 20 according to embodiment 2 acquires an augmented reality image and its display position setting distributed by the server device 30, for example, before the start of a live performance (step S111). The processing unit 21 stores the acquired augmented reality image and its display position setting in the storage unit 22 (step S112).

After that, for example, before the start of the live performance or during the live performance, the processing unit 21 acquires an image captured by the camera 13 of the AR glasses 10 (step S113). Based on the acquired captured image, the processing unit 21 generates three-dimensional information that reproduces the live venue as a three-dimensional virtual space (step S114). For example, the processing unit 21 transmits the generated three-dimensional information to the server device 30 and receives a response from the server device 30, thereby performing coordinate alignment to match the three-dimensional coordinate system of the live venue that the processing unit 21 recognizes as a three-dimensional virtual space with the three-dimensional coordinate system of the live venue that the server device 30 recognizes as a three-dimensional virtual space (step S115).

For example, after the start of a live performance, the processing unit 21 determines whether or not a display instruction to display an augmented reality image has been received from the server device 30 (step S116). If a display instruction has not been received (S116: NO), the processing unit 21 waits until a display instruction is received. If a display instruction has been received (S116: YES), the processing unit 21 reads out the augmented reality image that has been instructed to be displayed from the storage unit 22 (step S117). The processing unit 21 determines the display position of this image based on the display position setting corresponding to the read image (step S118). The processing unit 21 provides the read image to the AR glasses 10 and notifies the AR glasses 10 of the determined display position, thereby displaying the augmented reality image on the display unit 12 of the AR glasses 10 at the set display position (step S119).

The processing unit 21 determines whether to end the display of the augmented reality image based on whether an operation to end the display of the augmented reality image has been received when the live performance has ended (step S120). If the display is not to be ended (S120: NO), the processing unit 21 returns the process to step S116. If the display is to be ended (S120: YES), the processing unit 21 instructs the AR glasses 10, for example, to end the display of the augmented reality image, and ends the process.

In the information processing system according to the second embodiment configured as described above, the server device 30 generates information of a three-dimensional virtual space corresponding to the live venue based on an image of the live venue captured by the camera 40. The server device 30 transmits the generated three-dimensional information to a terminal device 50 used by the organizer of the live event, receives a setting for the display position of the augmented reality image in the three-dimensional virtual space via the terminal device 50, and stores the received display position setting in the storage unit 32. As a result, in the information processing system according to the second embodiment, it is expected that the organizer of the live event will be able to easily set the display position of the augmented reality image during the preparation stage of the live event.

In addition, in the information processing system according to the second embodiment, the terminal device 50 receives an instruction to display a virtual reality image during a live performance, and transmits the display instruction to the server device 30. In response, the server device 30 transmits the display instruction to all of the terminal devices 20 used by the audience of the live performance. This allows the organizers of the live performance to manipulate the display of the augmented reality image in accordance with the progress of the live performance.

In addition, in the information processing system according to the second embodiment, the terminal device 20 generates information on a three-dimensional virtual space corresponding to the live venue based on an image captured by the camera 13 of the audience's AR glasses 10. The terminal device 20 acquires an augmented reality image and setting information on the display position of this image from the server device 30. The terminal device 20 performs a process of displaying the acquired augmented reality image on the display unit 12 of the AR glasses 10 based on the generated information on the three-dimensional virtual space and the acquired setting information on the display position. This allows the terminal device 20 to appropriately display an augmented reality image whose display position is set in the three-dimensional virtual space corresponding to the live venue.

In addition, in the information processing system according to the second embodiment, the terminal device 20 used by the audience receives display instructions from the terminal device 50 used by the organizer of the live performance, via the server device 30. The terminal device 20 displays an augmented reality image on the display unit 12 of the AR glasses 10 in response to the received display instructions. This allows an augmented reality image to be displayed on the audience's AR glasses 10 in response to operations by the organizer in accordance with the progress of the live performance.

In the second embodiment, the host of the live event uses the terminal device 50 to perform various operations, but this is not limited thereto. If the server device 30 is equipped with a display unit and an operation unit, the host may use the server device 30 to perform various operations. That is, the server device 30 and the terminal device 50 may be a single device. The division of roles between the server device 30 and the terminal device 50 is not limited to that shown in the second embodiment, and may be changed as appropriate. The audience's terminal device 20 generates three-dimensional information corresponding to the live venue based on an image captured by the camera 13 of the AR glasses 10, but this is not limited thereto. The terminal device 20 may transmit the captured image to the server device 30, and the server device 30 may generate three-dimensional information based on the received image and transmit it to the terminal device 20.

In addition, in the second embodiment, the organizer or the like controls the timing of displaying the augmented reality images on the audience's AR glasses 10 by operating the terminal device 50 during the live performance, but this is not limited to this. For example, the augmented reality images may be displayed at a predetermined time or according to the passage of time, or the server device 30 may determine the progress of the live performance and display the augmented reality images.

The information processing system according to the second embodiment may also display a virtual reality image in response to the detection of a marker performed by the information processing system according to the first embodiment. In this case, the information processing system may display, at the timing of detecting the marker, a virtual reality image whose display position is set based on information of a three-dimensional virtual space, for example.

In addition, in the information processing system according to the second embodiment, the live venue is photographed by the camera 40 in the preparation stage for the live performance, and information on a three-dimensional virtual space corresponding to the live performance venue is generated based on the photographed images, but this is not limited to this. For example, if information on a three-dimensional virtual space corresponding to the live performance venue has already been generated in a previous live performance, etc., and this information can be reused, it is not necessary to photograph the live performance venue by the camera 40 in the preparation stage for the live performance. The server device 30 may store information on the three-dimensional virtual space corresponding to the live performance venue in a database, and distribute it to the organizers of the live performance who need it.

In addition, other configurations of the information processing system according to embodiment 2 are similar to those of the information processing system according to embodiment 1, so similar parts are given the same reference numerals and detailed descriptions are omitted.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the claims.

1. Artist 2. Audience 3. Image (augmented reality image)
Reference Signs List 4: Marker 5: Display device 5a: Marker 10: AR glasses 11: Frame 12: Display unit 13: Camera 14: Communication cable 15: Processing unit 16: Wired communication unit 20: Terminal device 21: Processing unit 21a: Detection processing unit 21b: Augmented reality processing unit (display processing unit)
22 Storage unit 22a Program 22b Augmented reality data 23 Wireless communication unit 24 Wired communication unit 25 Display unit 26 Operation unit 30 Server device 31 Processing unit 31a Data transmission processing unit 31b Live management processing unit 32a Server program 32b Augmented reality data 33 Communication unit 40 Camera 50 Terminal device 51 Processing unit 51a Setting processing unit 51b Display instruction unit 52 Storage unit 52a Program 52b Image data 53 Communication unit 54 Display unit 55 Operation unit 97, 98, 99 Recording medium

Claims (9)

A computer program that causes a computer to perform a process of displaying an augmented reality image on a glasses-type wearable device having a camera that captures a real space and a display unit that displays an image,
Detecting markers installed at a live venue based on the images captured by the camera;
A computer program that causes a computer to execute a process of displaying, on the display unit of the wearable device, an augmented reality image corresponding to a marker detected during a live performance. Detecting a predetermined movement of a performer appearing in the live show based on the image captured by the camera;
displaying an augmented reality image corresponding to the detected predetermined movement on the display unit of the wearable device;
2. The computer program product of claim 1. Detecting a predetermined image displayed on a display device installed at the live venue based on the image captured by the camera;
displaying an augmented reality image corresponding to the detected predetermined image on the display unit of the wearable device;
3. A computer program according to claim 1 or 2. When a predetermined time point during the live broadcast is reached, an augmented reality image corresponding to the predetermined time point is displayed on the display unit of the wearable device.
A computer program according to any one of claims 1 to 3. displaying a predetermined augmented reality image on the display unit of each of the plurality of spectators participating in the live performance so that the image can be seen from a position a predetermined distance away from the spectator;
A computer program according to any one of claims 1 to 4. displaying an augmented reality image on the display unit of the wearable device according to the amount of the fee paid in relation to the live performance;
A computer program according to any one of claims 1 to 5. An information processing device that performs processing to display an augmented reality image on a glasses-type wearable device having a camera that captures a real space and a display unit that displays an image,
A detection unit that detects markers installed at a live venue based on an image captured by the camera;
and a display processing unit that displays an augmented reality image corresponding to a marker detected during a live performance on the display unit of the wearable device. An information processing system including a marker installed at a live venue and an information processing device that performs processing to display an augmented reality image on a glasses-type wearable device having a camera that captures real space and a display unit that displays an image,
The information processing device includes:
A detection unit that detects markers installed at a live venue based on an image captured by the camera;
and a display processing unit that displays an augmented reality image corresponding to a marker detected during a live performance on the display unit of the wearable device. An augmented reality image display method in which an information processing device displays an augmented reality image on a glasses-type wearable device having a camera that captures a real space and a display unit that displays an image, comprising:
Detecting markers installed at a live venue based on the images captured by the camera;
An augmented reality image display method for displaying an augmented reality image corresponding to a marker detected during a live broadcast on the display unit of the wearable device.

Images