JP6901217B2 - A content providing method, a program that causes a computer to execute the method, and a content providing device. - Google Patents

Description

The present disclosure relates to a technology for providing content, and more specifically to a technology for providing content via a virtual reality space.

Regarding the provision of contents, a method of storing the contents in a recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc) or the like, and a technique of streaming the data of the contents are known. Recently, a technique for providing contents (hereinafter, also referred to as "VR (Virtual Reality) contents") in a virtual reality space (also referred to as a virtual space) is known. If the content provided is digital data, there are problems such as duplication and unauthorized use. In response to such problems, for example, Japanese Patent Application Laid-Open No. 2003-187524 (Patent Document 1) states, "For information such as reproducible music and images recorded on a computer recording medium, prevention of unauthorized use and recording information. It discloses the technology for "providing a rental business system that can protect and manage rental conditions" (see [Summary]).

Japanese Unexamined Patent Publication No. 2003-187524

Since a device called a head mounted device (HMD) is required to view the VR content, the number of viewers of the VR content may depend on the penetration rate of the HMD. Therefore, there is a need for a technology that can provide VR content to many users without depending on the penetration rate of HMD.

The present disclosure has been made in view of the above background, and is to provide a technique for many users to enjoy VR contents.

According to certain embodiments, there is provided a content serving method that is performed on a computer to serve content using a head mount device. This method involves accepting input of access information to access content through a computer interface, sending access information to a management server for managing one or more contents, and from the management server. A step of receiving content data for displaying the content, a step of defining a virtual space for presenting the content using the head mount device, and a step of outputting the content to the head mount device using the content data. including.

In a certain aspect, since the VR content is output based on the access information, more users can enjoy the VR content.

The above and other objectives, features, aspects and advantages of the invention will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

It is a figure which shows the transition of the screen displayed on the monitor 63. It is a figure which shows the outline of the structure of the content providing system 100 according to a certain embodiment. It is a figure which shows the operation performed by the HMD 110 when the user 190 enjoys a VR content. It is a block diagram which shows an example of the hardware composition of the computer 200 which follows one aspect. It is a figure that conceptually represents the uvw field-of-view coordinate system set in the HMD 110 according to a certain embodiment. It is a figure which conceptually represents one aspect which expresses the virtual space 2 according to a certain embodiment. It is a figure which showed the head of the user 190 who wears HMD 110 according to a certain embodiment from the top. It is a figure which shows the YZ cross section which looked at the view area 23 from the X direction in virtual space 2. It is a figure which shows the XZ cross section which looked at the view area 23 from the Y direction in virtual space 2. It is a figure which shows the schematic structure of the controller 160 according to a certain embodiment. An example of the hand object 810 arranged in the virtual space corresponding to the right hand of the user 190 holding the right controller 800 is shown. FIG. 3 is a block diagram showing a computer 200 according to an embodiment as a module configuration. It is a block diagram which shows the hardware structure of a smartphone 180. It is a figure which conceptually represents one aspect of data storage in the storage 154 included in a server 150. It is a flowchart which shows a part of the processing executed by the smartphone 180 attached to the HMD 110. It is a flowchart which exemplifies a part of the process which a smartphone 180 executes in order to display the image taken at the time of reproduction of VR content. It is a figure showing the screen displayed on the monitor 16 installed in a store. It is a figure which shows the transition of the screen on the monitor 63 of the smartphone 180. It is a figure which shows the outline of the structure of the content providing system 1900. It is a figure which shows one aspect of storing data in the storage 154 included in the server 150. It is a flowchart which shows the flow of the procedure which a user 190 executes. It is a flowchart which shows a part of the processing executed by the content providing system 1900. It is a figure which shows one aspect of the screen which the monitor 16 displays in order to notify the situation of waiting in order.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated.

First, an outline of a specific example of the present embodiment will be described. A two-dimensional code (for example, a QR code (registered trademark)) containing information for accessing the VR content is attached to the goods. The goods are related to, for example, a character appearing in the VR content, and for example, a mug cup. , T-shirts, cards, CD cases, clear folders, etc., but not limited to these. Since there may be goods that cannot be attached with a two-dimensional code, tickets with the two-dimensional code printed on them are sold. These goods or tickets may be sold in the same manner as existing operations in the store. The goods are not limited to those related to characters, but may be general products. In this case, promotion of goods. The two-dimensional code may be attached to the goods. For example, it is assumed that the two-dimensional code is attached to the cap or the main body of the PET bottle beverage.

VR headsets will be available at the store. As one aspect of the VR headset, a smartphone having a camera may be attached to the HMD. The VR headset can shoot the outside of the VR headset when in use.

When the user who visits the store wears the VR headset, the camera of the smartphone is activated, and the image taken by the camera, that is, the front of the user is displayed on the monitor of the HMD (for example, the monitor of the smartphone). Therefore, the user can see the real world through the captured image of the camera like a see-through while wearing the HMD.

When the user reads the two-dimensional code with the camera while wearing the HMD, the VR content data is streamed from the server to the computer connected to the HMD. When this data is further transmitted to the smartphone, the monitor begins to display the VR content.

Since the user knows the operation of reading the two-dimensional code with the camera of the smartphone, the VR content can be played without asking the store staff again (without adding an operation for using the VR headset). it can.

A photo opportunity is provided to the user during playback of VR content. When the user shoots a desired scene, the smartphone stores the two-dimensional code read for VR content reproduction and the shot image in association with each other. After viewing the VR content, the user can view the image taken by using the smartphone application or the like after removing the smartphone from the HMD.

One aspect of the present disclosure will be described with reference to FIG. FIG. 1 is a diagram showing a transition of a screen displayed on the monitor 63. The monitor 63 corresponds to, for example, a monitor incorporated in the HMD, a monitor of a smartphone or other information terminal mounted on the HMD.

As shown in the state A, in a certain aspect, the monitor 63 displays the captured image of the two-dimensional code. When the access information included in the two-dimensional code is extracted, communication is performed between the terminal to which the monitor 63 is connected and the management server for VR contents, and the access information is transmitted from the terminal to the management server. .. The access information includes a content ID, a content authentication number, an expiration date, and the like. In order to prevent unauthorized use of access information and improve the accuracy of authentication, the location information of the terminal whose information is extracted from the two-dimensional code may be further transmitted to the management server. When the management server confirms that the information is legitimate information, the management server sends a message to the terminal before playing the VR content associated with the two-dimensional code. As shown in the state B, the monitor 63 displays the message.

After that, the output of the VR content (for example, video reproduction, audio output, etc.) begins. For example, as shown in the state C, the monitor 63 displays an image of the VR content downloaded based on the two-dimensional code. During playback of the VR content, the user of the monitor 63 can capture an image of the VR content. The captured image can be stored in the terminal. Alternatively, the location (for example, frame number) where the shooting was performed in the content data is transmitted by the terminal to the server together with the ID of the user. In this case, the server can store the user's ID, the identification number of the VR content, and the frame number in the database in association with each other.

When the playback of the VR content is completed, the monitor 63 displays a message such as "Please come again" as shown in the state D. After that, as shown in the state E, the monitor 63 displays the date and time taken during the reproduction of the VR content and the image of the VR content taken at that time.

<First Embodiment>
Hereinafter, the first embodiment according to the present disclosure will be described. In the first embodiment, the two-dimensional code attached to the goods includes the access information. The access information is used to access the VR content.

[HMD system configuration]
The content providing system 100 using the HMD will be described with reference to FIG. FIG. 2 is a diagram showing an outline of the configuration of the content providing system 100 according to a certain embodiment. In a certain aspect, the content providing system 100 is arranged in a store, an amusement facility, or the like. In the present embodiment, the HMD may be either a so-called head-mounted display provided with a monitor or a head-mounted device to which a smartphone or other terminal having a monitor can be attached. In the following, the case where the smartphone can be attached to and detached from the head mount device will be mainly described.

The content providing system 100 includes a monitor 16, an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a gaze sensor 140. The smartphone 180 can be attached to the HMD 110. The controller 160 may include a motion sensor 130.

In some aspects, the computer 200 can connect to the Internet or other network 19 and communicate with the server 150 or other computer connected to the network 19.

In another aspect, the HMD 110 may include the sensor 114 instead of the content providing system 100 including the HMD sensor 120.

The server 150 includes a processor 151, a volatile memory 152, a communication interface 153, and a non-volatile storage 154. The server 150 is realized by a computer having a well-known configuration.

The HMD 110 may be worn on the user's head and provide the user with virtual space during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the monitor 63, respectively. When each eye of the user visually recognizes the image, the user can recognize the image as a three-dimensional image based on the parallax of both eyes.

When the content providing system 100 is used in a store, the user 190 wearing the HMD 110 can visually recognize the mug 101 sold in the store as an image displayed on the monitor 63. A two-dimensional code (for example, a QR code (registered trademark)) is attached to the mug 101.

The monitor 63 is realized as, for example, a non-transparent display device. In some aspects, the monitor 63 may be pre-positioned on the body of the HMD 110 so that it is located in front of both eyes of the user. Therefore, the user can immerse himself in the virtual space by visually recognizing the three-dimensional image displayed on the monitor 63. In certain embodiments, the virtual space includes, for example, a background, user-operable objects, and user-selectable menu images. In a certain embodiment, when the HMD 110 is configured to be equipped with a so-called smartphone or other information display terminal, the monitor 63 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor included in the information display terminal.

In some aspects, the monitor 63 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 63 may be configured to display the image for the right eye and the image for the left eye as a unit. In this case, the monitor 63 includes a high speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

The HMD sensor 120 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. The HMD sensor 120 uses this function to detect the position and tilt of the HMD 110 in the real space.

In another aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and tilt of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include the sensor 114 as the position detector instead of the HMD sensor 120. The HMD 110 may use the sensor 114 to detect the position and tilt of the HMD 110 itself. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 uses any of these sensors instead of the HMD sensor 120 to detect its position and inclination. Can be done. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in real space over time. The HMD 110 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle. Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance. Further, the field-of-view image may include a configuration for presenting the real space as a part of the image constituting the virtual space. For example, an image taken by a camera mounted on the HMD 110 may be superimposed on a part of the field of view image and displayed, or by setting a high transmittance of a part of the transmissive display device, the field of view image can be displayed. The real space may be visible from a part.

The gaze sensor 140 detects the direction (line-of-sight direction) in which the eyes of the right eye and the left eye of the user 190 are directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In certain aspects, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the angle of rotation of each eyeball by receiving the reflected light from the cornea and the iris with respect to the irradiation light. .. The gaze sensor 140 can detect the line-of-sight direction of the user 190 based on each detected rotation angle.

The server 150 may send the program to the computer 200. In another aspect, the server 150 may communicate with another computer 200 to provide virtual reality to the HMD used by other users. For example, in an amusement facility, when a plurality of users play a participatory game, each computer 200 communicates a signal based on the operation of each user with another computer 200, and the plurality of users are common in the same virtual space. Allows you to enjoy the game.

The controller 160 receives an instruction input from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be grippable by the user 190. In another aspect, the controller 160 is configured to be wearable on a body or part of clothing of the user 190. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal sent from the computer 200. In another aspect, the controller 160 accepts operations given by the user 190 to control the position and movement of objects placed in the space that provides virtual reality.

The motion sensor 130 is attached to the user's hand in a certain aspect to detect the movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, the number of rotations, and the like of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove-shaped controller 160. In certain embodiments, for safety in real space, it is desirable that the controller 160 be attached to something that does not easily fly by being attached to the user 190's hand, such as a glove type. In another aspect, a sensor not attached to the user 190 may detect the movement of the user 190's hand. For example, the signal of the camera that captures the user 190 may be input to the computer 200 as a signal indicating the operation of the user 190. The motion sensor 130 and the computer 200 are connected to each other by wire or wirelessly. In the case of wireless communication, the communication mode is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication method is used.

With reference to FIG. 3, the behavior of the user 190, the behavior of the HMD 110, and the behavior of the character of the VR content will be described. FIG. 3 is a diagram showing an operation performed by the HMD 110 when the user 190 enjoys the VR content.

When the user 190 visits the store, he / she purchases the mug 101 and other goods (step S310). After that, the user 190 attaches the smartphone 180, which is the property thereof, to the HMD 110 (step S312). When the user 190 operates the smartphone 180 according to a predetermined procedure, an application for receiving the VR content is started. When the execution of the application starts, the HMD 110 displays a message such as "Please attach the HMD" on the monitor 63 of the smartphone 180, or outputs a voice (step S314).

The user 190 wearing the HMD 110 on his head activates the camera application of the smartphone 180 and shoots the two-dimensional code 102 attached to the purchased goods (for example, the mug 101) (step S320).

The HMD 110 accesses the server 150 via the smartphone 180 and the computer 200 (step S322). The smartphone 180 transmits the image data obtained by shooting to the server 150. After that, the VR content is downloaded from the server 150 to the HMD 110.

When the character of the VR content is displayed on the monitor 63, the monitor 63 displays a message such as "Thank you for coming to the library today" (step S324). At this time, the speaker (not shown) built in the HMD 110 may output the message by voice based on the voice signal output from the smartphone 180.

The HMD 110 displays a message such as "Move your head and move the white circle in front of you" or output a voice (step S330). The user 190 moves his / her head while wearing the HMD 110 on his / her head (step S332). Further, the HMD 110 displays a message such as "Move the white circle here" and outputs a voice (step S340). The user 190 moves the head wearing the HMD 110, moves the white circle to a predetermined position, and selects the start screen (step S342).

When the playback of the VR content starts, the HMD 110 displays a message such as "You can take a picture only once during the performance" (step S350). The HMD 110 further displays a message such as "There is a touch panel here on the headset. Touch it" (step S352). The HMD110 displays the message "You can take a picture only once during the live. Don't miss the best shot" (step S354). After that, the HMD 110 displays the message "Then it is ready" (step S356), and starts playing the VR content.

When the VR content is displayed on the monitor 63, the character of the VR content displays a message such as "Enjoy the live performance" and outputs a voice (step S358).

While playing the VR content, the HMD 110 displays a message such as "Live" in, for example, one corner of the screen (step S360). During playback of the VR content, the user 190 can shoot a live scene a predetermined number of times for each VR content (step S370). The playback scene may be arbitrarily selected by the user 190, or may be predetermined. Further, a scene according to the preference of the user 190 may be recommended. When the playback of the VR content is completed, the character of the VR content displays the message "Thank you for coming to the library today" (step S372). Furthermore, the character displays a message "I'm glad you come again" (step S374).

The monitor 63 of the HMD 110 displays a message "Please remove the HMD" or outputs a voice (step S376). At this time, the HMD 110 may display a live demo video of other VR contents as a two-dimensional video.

User 190 removes the HMD 110 from his head (step S378). The user registers the serial number displayed when the two-dimensional code is read and the user's own personal information on the site that provides the VR content (step S380). For example, when the user 190 accesses the link destination displayed on the monitor 63, the serial number and the personal information are transmitted to the server 150.

[Hardware configuration]
The computer 200 according to the present embodiment will be described with reference to FIG. FIG. 4 is a block diagram showing an example of the hardware configuration of the computer 200 according to one aspect. The computer 200 includes a processor 10, a volatile memory 11, a non-volatile storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on the signal given to the computer 200 or when a predetermined condition is satisfied. In a certain aspect, the processor 10 is realized as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array) or other device.

The memory 11 temporarily stores programs and data. The program is loaded from storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In a certain aspect, the memory 11 is realized as a RAM (Random Access Memory) or other volatile memory.

The storage 12 permanently holds programs and data. The storage 12 is realized as, for example, a ROM (Read-Only Memory), a hard disk device, a flash memory, or other non-volatile storage device. The program stored in the storage 12 includes a program for providing a virtual space in the content providing system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data, objects, and the like for defining the virtual space.

In another aspect, the storage 12 may be realized as a detachable storage device such as a memory card. In yet another aspect, a configuration that uses programs and data stored in an external storage device may be used instead of the storage 12 built into the computer 200. According to such a configuration, for example, in a scene where a plurality of content providing systems 100 are used such as an amusement facility, it is possible to collectively update programs and data.

In certain embodiments, the input / output interface 13 communicates a signal with the HMD 110, the HMD sensor 120, or the motion sensor 130. In a certain aspect, the input / output interface 13 is realized by using a USB (Universal Serial Bus) interface, a DVI (Digital Visual Interface), an HDMI (registered trademark) (High-Definition Multimedia Interface), and other terminals. The input / output interface 13 is not limited to the above.

In certain embodiments, the input / output interface 13 may further communicate with the controller 160. For example, the input / output interface 13 receives an input of a signal output from the motion sensor 130. In another aspect, the input / output interface 13 sends an instruction output from the processor 10 to the controller 160. The command instructs the controller 160 to vibrate, output voice, emit light, and the like. Upon receiving the instruction, the controller 160 executes either vibration, audio output, or light emission in response to the instruction.

The communication interface 14 is connected to the network 19 and communicates with another computer (for example, the server 150) connected to the network 19. In a certain aspect, the communication interface 14 is realized as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication) or other wireless communication interface. Will be done. The communication interface 14 is not limited to the above.

In one aspect, the processor 10 accesses the storage 12, loads one or more programs stored in the storage 12 into the memory 11, and executes a series of instructions contained in the program. The one or more programs may include an operating system of the computer 200, an application program for providing the virtual space, game software that can be executed in the virtual space using the controller 160, and the like. The processor 10 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 63 based on the signal.

In the example shown in FIG. 4, the computer 200 is configured to be provided outside the HMD 110, but in another aspect, the computer 200 may be built in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including a monitor 63 may function as a computer 200.

Further, the computer 200 may have a configuration commonly used for a plurality of HMD 110s. According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space. In another aspect, when a plurality of HMD 110s are connected to the computer 200, each HMD 110 may play VR content different from the VR content played on the other HMD 110.

In a certain embodiment, in the content providing system 100, a global coordinate system is preset. The global coordinate system has three reference directions (axes) that are parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-back direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction of real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the real space.

In some aspects, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect a temporal change in the position and inclination of the HMD 110 by using each value detected over time.

The global coordinate system is parallel to the coordinate system in real space. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination of the HMD 110 around three axes in the global coordinate system. The HMD sensor 120 sets the uvw field of view coordinate system to the HMD 110 based on the tilt of the HMD 110 in the global coordinate system. The uvw field-of-view coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 sees an object in the virtual space.
[Uvw field coordinate system]
The uvw field coordinate system will be described with reference to FIG. FIG. 5 is a diagram conceptually representing the uvw field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and tilt of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 10 sets the uvw field coordinate system to the HMD 110 based on the detected values.

As shown in FIG. 5, the HMD 110 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user wearing the HMD 110. More specifically, the HMD 110 defines the global coordinate system in the horizontal, vertical, and front-back directions (x-axis, y-axis, z-axis) by the inclination of the HMD 110 around each axis in the global coordinate system. The three directions newly obtained by tilting each around the axis are set as the pitch direction (u-axis), the yaw direction (v-axis), and the roll direction (w-axis) of the uvw field coordinate system in the HMD 110.

In a certain aspect, when the user 190 wearing the HMD 110 is upright and visually recognizing the front, the processor 10 sets the uvw field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch direction (u-axis) and yaw direction (v-axis) of the uvw field coordinate system in the HMD 110. , And the roll direction (w-axis).

After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (change amount of the inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110, respectively. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

The HMD sensor 120 sets the uvw field coordinate system in the HMD 110 after the HMD 110 moves, based on the detected tilt angle of the HMD 110, in the HMD 110. The relationship between the HMD 110 and the uvw field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

In one aspect, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between the points (eg, the distance between the points). The position of the above in the real space may be specified as a relative position with respect to the HMD sensor 120. Further, the processor 10 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. FIG. 6 is a diagram conceptually representing one aspect of expressing the virtual space 2 according to a certain embodiment. The virtual space 2 has an all-sky spherical structure that covers the entire center 21 in the 360-degree direction. In FIG. 6, the celestial sphere in the upper half of the virtual space 2 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 2. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system defined in the virtual space 2. The computer 200 associates each partial image constituting the content (still image, moving image, etc.) that can be expanded in the virtual space 2 with each corresponding mesh in the virtual space 2, and the virtual space image 22 that can be visually recognized by the user. Provides the user with the virtual space 2 in which is expanded.

In a certain aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the XYZ coordinate system are defined as the X-axis, the Y-axis, and the Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the global coordinate system, and the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the global coordinate system. The Z-axis (front-back direction) is parallel to the z-axis of the global coordinate system.

When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 1 is arranged at the center 21 of the virtual space 2. The virtual camera 1 moves in the virtual space 2 in the same manner in conjunction with the movement of the HMD 110 in the real space. As a result, changes in the position and orientation of the HMD 110 in the real space are similarly reproduced in the virtual space 2.

The virtual camera 1 is defined with an uvw field-of-view coordinate system as in the case of the HMD 110. The uvw field-of-view coordinate system of the virtual camera 1 in the virtual space 2 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 1 also changes accordingly. Further, the virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user wearing the HMD 110 in the real space.

Since the orientation of the virtual camera 1 is determined according to the position and inclination of the virtual camera 1, the reference line of sight (reference line of sight 5) when the user visually recognizes the virtual space image 22 depends on the orientation of the virtual camera 1. It is decided. The processor 10 of the computer 200 defines the field of view 23 in the virtual space 2 based on the reference line of sight 5. The field of view 23 corresponds to the field of view of the user wearing the HMD 110 in the virtual space 2.

The line-of-sight direction of the user 190 detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user 190 visually recognizes an object. The uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 63. Further, the uvw field-of-view coordinate system of the virtual camera 1 is linked to the uvw field-of-view coordinate system of the HMD 110. Therefore, the content providing system 100 according to a certain aspect can consider the line-of-sight direction of the user 190 detected by the gaze sensor 140 as the line-of-sight direction of the user in the uvw field-of-view coordinate system of the virtual camera 1.

[User's line of sight]
The determination of the line-of-sight direction of the user will be described with reference to FIG. 7. FIG. 7 is a top view of the head of the user 190 wearing the HMD 110 according to an embodiment.

In one aspect, the gaze sensor 140 detects each line of sight of the user 190's right and left eyes. In one aspect, when the user 190 is looking closer, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 identifies the gaze point N1 which is the intersection of the lines of sight R1 and L1 based on the detected values. On the other hand, when the computer 200 receives the detected values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point N1. The computer 200 specifies the line-of-sight direction N0 of the user 190 based on the position of the specified gazing point N1. For example, the computer 200 detects the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the extending direction of the straight line passing through the gazing point N1 as the line-of-sight direction N0. The line-of-sight direction N0 is the direction in which the user 190 actually directs the line of sight with both eyes. Further, the line-of-sight direction N0 corresponds to the direction in which the user 190 actually directs the line of sight with respect to the field of view area 23.

In another aspect, the content providing system 100 may include a microphone and a speaker in any of the parts constituting the content providing system 100. The user can give a voice instruction to the virtual space 2 by speaking to the microphone.

In another aspect, the content providing system 100 may include a television broadcast receiving tuner. According to such a configuration, the content providing system 100 can display a television program in the virtual space 2.

In yet another aspect, the content providing system 100 may include a communication circuit for connecting to the Internet or a call function for connecting to a telephone line.

[Visibility area]
The field of view 23 will be described with reference to FIGS. 8 and 9. FIG. 8 is a diagram showing a YZ cross section of the field of view 23 as viewed from the X direction in the virtual space 2. FIG. 9 is a diagram showing an XZ cross section of the field of view 23 as viewed from the Y direction in the virtual space 2.

As shown in FIG. 8, the field of view region 23 in the YZ cross section includes the region 24. The area 24 is defined by the reference line of sight 5 of the virtual camera 1 and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α centered on the reference line of sight 5 in the virtual space as a region 24.

As shown in FIG. 9, the field of view region 23 in the XZ cross section includes the region 25. The region 25 is defined by the reference line of sight 5 and the XZ cross section of the virtual space 2. The processor 10 defines a range including the azimuth angle β centered on the reference line of sight 5 in the virtual space 2 as a region 25.

In a certain aspect, the content providing system 100 provides the user 190 with a virtual space by displaying the view image 26 on the monitor 63 based on the signal from the computer 200. The visual field image 26 corresponds to a portion of the virtual space image 22 that is superimposed on the visual field region 23. When the user 190 moves the HMD 110 attached to the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the field of view 23 in the virtual space 2 changes. As a result, the visual field image 26 displayed on the monitor 63 is updated to an image of the virtual space image 22 that is superimposed on the visual field region 23 in the direction in which the user faces in the virtual space 2. The user can visually recognize the desired direction in the virtual space 2.

While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the content providing system 100 can give the user a high sense of immersion in the virtual space 2.

In a certain aspect, the processor 10 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 10 specifies an image area (that is, a view area 23 in the virtual space 2) projected on the monitor 63 of the HMD 110 based on the position and orientation of the virtual camera 1 in the virtual space 2.

According to certain embodiments, the virtual camera 1 preferably includes two virtual cameras, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Further, it is preferable that an appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In the present embodiment, the virtual camera 1 includes two virtual cameras, and the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD 110. The technical idea according to the present disclosure will be illustrated as being configured as such.

[controller]
An example of the controller 160 will be described with reference to FIG. FIG. 10 is a diagram showing a schematic configuration of a controller 160 according to an embodiment. In some aspects, the controller 160 may include a right controller 800 and a left controller. The right controller 800 is operated by the right hand of the user 190. The left controller is operated by the left hand of the user 190. In one aspect, the right controller 800 and the left controller are symmetrically configured as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller 800 and the left hand holding the left controller respectively. In another aspect, the controller 160 may be an integrated controller that accepts operations of both hands. Hereinafter, the right controller 800 will be described.

The right controller 800 includes a grip 30, a frame 31, and a top surface 32. The grip 30 is configured to be gripped by the right hand of the user 190. For example, the grip 30 may be held by the palm of the user 190's right hand and three fingers (middle finger, ring finger, little finger).

The grip 30 includes buttons 33 and 34 and a motion sensor 130. The button 33 is arranged on the side surface of the grip 30 and accepts an operation by the middle finger of the right hand. The button 34 is arranged on the front surface of the grip 30 and accepts an operation by the index finger of the right hand. In one aspect, the buttons 33, 34 are configured as trigger-type buttons. The motion sensor 130 is built in the housing of the grip 30. If the operation of the user 190 can be detected from around the user 190 by a camera or other device, the grip 30 does not have to include the motion sensor 130.

The frame 31 includes a plurality of infrared LEDs 35 arranged along its circumferential direction. The infrared LED 35 emits infrared rays as the program progresses while the program using the controller 160 is being executed. The infrared rays emitted from the infrared LED 35 can be used to detect each position and orientation (tilt, orientation) of the right controller 800 and the left controller (not shown). In the example shown in FIG. 8, infrared LEDs 35 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. An array of one column or three or more columns may be used.

The top surface 32 includes buttons 36 and 37 and an analog stick 38. The buttons 36 and 37 are configured as push-type buttons. Buttons 36 and 37 accept operations by the thumb of the user 190's right hand. The analog stick 38 accepts an operation 360 degrees in any direction from the initial position (neutral position) in a certain aspect. The operation includes, for example, an operation for moving an object arranged in the virtual space 2.

In one aspect, the right controller 800 and the left controller include a battery for driving the infrared LED 35 and other components. Batteries include, but are not limited to, rechargeable, button type, dry cell type and the like. In another aspect, the right controller 800 and the left controller may be connected to, for example, the USB interface of the computer 200. In this case, the right controller 800 and the left controller do not require batteries.

FIG. 11 shows an example of a hand object 810 arranged in the virtual space corresponding to the right hand of the user 190 holding the right controller 800. For example, the yaw, roll, and pitch directions are defined for the hand object 810 corresponding to the right hand of the user 190. For example, when the input operation is performed on the button 34 of the right controller 800, the index finger of the hand object 810 is grasped, and when the input operation is not performed on the button 34, the hand object 810 is held. It is also possible to keep the index finger extended. For example, in the hand object 810, when the thumb and the index finger are extended, the direction in which the thumb is extended is the yaw direction, and the direction in which the index finger is extended is the roll direction, the yaw direction axis, and the direction perpendicular to the plane defined by the roll direction axis. The direction is defined by the hand object 810 as the pitch direction.

[Control device for HMD110]
The control device of the HMD 110 will be described with reference to FIG. In certain embodiments, the control device is implemented by a computer 200 having a well-known configuration. FIG. 12 is a block diagram showing a computer 200 according to an embodiment as a module configuration.

As shown in FIG. 12, the computer 200 includes a display control module 220, a virtual space control module 230, a memory module 240, and a communication control module 250. The display control module 220 includes a virtual camera control module 221, a view area determination module 222, a view image generation module 223, a reference line-of-sight identification module 224, an authentication module 225, and a content reproduction module 226 as submodules. .. The virtual space control module 230 includes a virtual space definition module 231, a virtual object generation module 232, and a controller management module 233 as submodules.

In certain embodiments, the display control module 220 and the virtual space control module 230 are implemented by the processor 10. In another embodiment, a plurality of processors 10 may operate as the display control module 220 and the virtual space control module 230. The memory module 240 is realized by the memory 11 or the storage 12. The communication control module 250 is realized by the communication interface 14.

In a certain aspect, the display control module 220 controls the image display on the monitor 63 of the HMD 110. The virtual camera control module 221 arranges the virtual camera 1 in the virtual space 2 and controls the behavior, orientation, and the like of the virtual camera 1. The visual field determination module 222 defines the visual field region 23 according to the orientation of the head of the user wearing the HMD 110. The visual field image generation module 223 generates a visual field image 26 to be displayed on the monitor 63 based on the determined visual field region 23.

The reference line-of-sight identification module 224 identifies the line-of-sight of the user 190 based on the signal from the gaze sensor 140.

The authentication module 225 determines whether or not the data transmitted from the HMD 110 is legitimate data registered in advance based on the data transmitted from the HMD 110 and the data transmitted from the server 150. Here, the regular data is, for example, identification data of video content or other VR content prepared in advance.

The content reproduction module 226 reproduces the content data transmitted from the server 150 and transmits it to the HMD 110 as a stream image.

The virtual space control module 230 controls the virtual space 2 provided to the user 190. The virtual space definition module 231 defines the virtual space 2 in the content providing system 100 by generating virtual space data representing the virtual space 2. The virtual object generation module 232 can generate an object to be arranged in the virtual space 2.

The controller management module 233 receives the operation of the user 190 in the virtual space 2 and controls the controller object according to the operation. A controller object according to an embodiment functions as a controller that gives instructions to other objects arranged in the virtual space 2. In a certain aspect, the controller management module 233 generates data for arranging the controller object that receives control in the virtual space 2 in the virtual space 2. When the HMD 110 receives this data, the monitor 63 may display the controller object.

The memory module 240 holds data used by the computer 200 to provide the virtual space 2 to the user 190. In a certain aspect, the memory module 240 holds the spatial information 241 and the user information 242 and the content 243.

Spatial information 241 holds one or more templates defined to provide virtual space 2. The user information 242 includes the identification information of the user 190 of the HMD 110, the authority associated with the user 190, and the like. The authority includes, for example, account information (user ID, password) for accessing a website that provides an application. Content 243 includes, for example, VR content presented by the HMD 110.

The communication control module 250 can communicate with the server 150 and other information communication devices via the network 19. The communication control module 250 may realize at least one of wireless and wired communication.

In certain aspects, the display control module 220 and the virtual space control module 230 can be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

The processing in the computer 200 is realized by the hardware and the software executed by the processor 10. Such software may be pre-stored in a hard disk or other memory module 240. The software may also be stored in a computer-readable non-volatile data recording medium such as a CD-ROM and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other networks. Such software is read from a data recording medium by an optical disk drive or other data reader, or downloaded from a server 150 or other computer via a communication control module 250, and then temporarily stored in a storage module. .. The software is read from the storage module by the processor 10 and stored in RAM in the form of an executable program. Processor 10 executes the program.

The hardware that constitutes the computer 200 is general. Therefore, it can be said that the most essential part according to the present embodiment is the program stored in the computer 200. Since the operation of the hardware of the computer 200 is well known, the detailed description will not be repeated. The data recording medium is not limited to CD-ROM, FD (Flexible Disk), and hard disk, but also magnetic tape, cassette tape, and optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc). ), IC (Integrated Circuit) card (including memory card), optical card, mask ROM, EPROM (Electronically Programmable Read-Only Memory), EEPROM (Electronically Erasable Programmable Read-Only Memory), semiconductor memory such as flash ROM, etc. It may be a non-volatile data recording medium that fixedly carries the program. The program referred to here may include not only a program that can be directly executed by the processor 10, but also a source program format program, a compressed program, an encrypted program, and the like.
[Configuration of smartphone 180]
The configuration of the smartphone 180 will be described with reference to FIG. FIG. 13 is a block diagram showing a hardware configuration of the smartphone 180. The smartphone 180 includes a CPU (Central Processing Unit) 50, an antenna 51, a communication device 52, an input switch 53, a camera 54, a flash memory 55, a RAM (Random Access Memory) 56, and a ROM (Read-Only). Memory) 57, memory card drive 58, microphone 61, speaker 62, audio signal processing circuit 60, monitor 63, LED (Light Emitting Diode) 64, communication interface 65, vibrator 66, GPS. It includes an antenna 68, a GPS module 67, an acceleration sensor 69, and a geomagnetic sensor 70. A memory card 59 may be mounted on the memory card drive device 58.

The antenna 51 receives a signal transmitted by the base station, or transmits a signal for communicating with another communication device via the base station. The signal received by the antenna 51 is front-end processed by the communication device 52, and the processed signal is sent to the CPU 50.

The CPU 50 executes a process for controlling the operation of the smartphone 180 based on an instruction given to the smartphone 180. When the smartphone 180 receives the signal, the CPU 50 executes a predetermined process based on the signal sent from the communication device 52, and sends the processed signal to the audio signal processing circuit 60. The audio signal processing circuit 60 executes a predetermined signal processing on the signal, and sends the processed signal to the speaker 62. The speaker 62 outputs sound based on the signal.

The input switch 53 receives an input of a command to the smartphone 180. The input switch 53 is realized by a touch sensor, a button provided on the housing of the smartphone 180, and the like. The signal corresponding to the input command is input to the CPU 50.

The microphone 61 receives an utterance to the smartphone 180 and sends a signal corresponding to the uttered voice to the voice signal processing circuit 60. The audio signal processing circuit 60 executes a predetermined process for a call based on the signal, and sends the processed signal to the CPU 50. The CPU 50 converts the signal into data for transmission, and sends the converted data to the communication device 52. The communication device 52 uses the data to generate a signal for transmission, and sends the signal toward the antenna 51.

The flash memory 55 stores data sent from the CPU 50. Further, the CPU 50 reads the data stored in the flash memory 55 and executes a predetermined process using the data.

The RAM 56 temporarily holds the data generated by the CPU 50 based on the operation performed on the input switch 53. The ROM 57 stores a program or data for causing the smartphone 180 to execute a predetermined operation. The CPU 50 reads the program or data from the ROM 57 and controls the operation of the smartphone 180.

The memory card driving device 58 reads the data stored in the memory card 59 and sends it to the CPU 50. The memory card drive device 58 writes the data output by the CPU 50 into the storage area of the memory card 59.

The voice signal processing circuit 60 executes signal processing for a call as described above. In the present embodiment, the CPU 50 and the audio signal processing circuit 60 are illustrated as separate configurations, but in other aspects, the CPU 50 and the audio signal processing circuit 60 may be integrally configured.

The monitor 63 is a touch operation type monitor, but the mechanism for receiving the touch operation is not particularly limited. The monitor 63 displays an image defined by the data based on the data acquired from the CPU 50. For example, a still image, a moving image, a map, or the like stored in the flash memory 55 is displayed.

The LED 64 emits light based on the signal output from the CPU 50. The communication interface 65 is realized by WiFi, Bluetooth (registered trademark), NFC (Near Field Communication) and the like. In another aspect, the communication interface 65 accepts the attachment of a cable for data communication. The communication interface 65 transmits a signal output from the CPU 50. Alternatively, the communication interface 65 transmits data included in the signal received from the outside of the smartphone 180 to the CPU 50. In a certain aspect, when the smartphone 180 is attached to the HMD 110, the communication interface 65 becomes communicable with the communication interface of the HMD 110.

The vibrator 66 executes an oscillation operation at a predetermined frequency based on a signal output from the CPU 50.

The GPS antenna 68 receives GPS signals transmitted from each of four or more satellites. Each received GPS signal is input to the GPS module 67. The GPS module 67 executes positioning processing using each GPS signal and a known technique, and acquires the position information of the smartphone 180.

The acceleration sensor 69 detects the acceleration acting on the smartphone 180. In one aspect, the accelerometer 69 is realized as a three-axis accelerometer. The detected acceleration is input to the CPU 50. The CPU 50 detects the movement and posture (tilt) of the smartphone 180 based on this acceleration.

The geomagnetic sensor 70 detects the direction in which the smartphone 180 is facing. The information acquired by this detection is input to the CPU 50.

[data structure]
The data structure of the server 150 will be described with reference to FIG. FIG. 14 is a diagram conceptually showing an aspect of data storage in the storage 154 included in the server 150. Storage 154 holds tables 1410, 1420, 1430, 1440, 1450.

Table 1410 includes the content ID 1411, the content data 1412, the number of playbacks 1413, and the last playback date and time 1414. Content ID 1411 identifies the VR content provided by the HMD 110. The content data 1412 is the data of the VR content. The number of playbacks 1413 represents the number of times the VR content has been played by the HMD 110 of each store. The last playback date and time 1414 represents the date and time when the VR content was last played.

Table 1420 holds the playback history of the VR content. More specifically, the table 1420 includes a playback date and time 1421, a playback location 1422, a content ID 1423, a user ID 1424, a terminal ID 1425, and a two-dimensional code 1426. The playback date and time 1421 represents the date and time when the VR content was played back. The reproduction location 1422 represents the location where the VR content is reproduced (for example, the name, address, coordinate value, etc. of the store). Content ID 1423 identifies VR content. The user ID 1424 identifies the user who has viewed the VR content. The terminal ID 1425 identifies the terminal (HMD110 or smartphone 180) in which the VR content is reproduced.

Table 1430 holds data about the scene selected and shot by the user. For example, when a user logs in to the user screen of the VR content from a personal computer at home and browses a captured image, a record of the browsing is held in the table 1430. More specifically, the table 1430 includes an access date and time 1431, an access location 1432, a content ID 1433, a frame number 1434, a user ID 1435, and a terminal ID 1436. The access date and time 1431 represents the date and time when the VR content was accessed. The access location 1432 represents a location where the VR content is accessed (for example, the IP (Internet Protocol) address, geographical coordinate values, address, and other location information of the user 190's home personal computer). The frame number 1434 is a frame number in the VR content data as a moving image, and identifies the captured image. The user ID 1435 identifies the user who has acquired the captured image by viewing the VR content. The terminal ID 1436 identifies the terminal (for example, a personal computer at home) that has made the access.

Table 1440 corresponds to a database of advertisements. More specifically, table 1440 includes content ID 1441, advertising ID 1442, and advertising data 1443. The content ID 1441 identifies the VR content in the same manner as the content ID 1411. The advertisement ID 1442 identifies the advertisement associated with the VR content. One or more advertisements are associated with one VR content. The advertisement data 1443 represents the data of the advertisement. Each ad can be pre-associated with VR content.

Table 1450 includes an advertisement ID 1451, a delivery date and time 1452, and a user ID 1453. The advertisement ID 1451 identifies the advertisement in the same manner as the advertisement ID 1442. The delivery date and time 1452 represents the date and time when the advertisement specified by the advertisement ID 1451 was delivered. The user ID 1453 represents the user to whom the advertisement is delivered (presented). For example, when the user 190 accesses the server 150 from the personal computer at home and browses the captured image of the VR content, the advertisement associated with the VR content is displayed on the monitor of the personal computer, and the distribution history at this time is a table. It is stored in 1450.

[VR content playback procedure]
The procedure for reproducing the VR content will be described with reference to FIG. FIG. 15 is a flowchart showing a part of the processing executed by the smartphone 180 mounted on the HMD 110. This process is executed when the user 190 attaches the smartphone 180 to the HMD 110 and views the VR content at the store.

In step S1510, the CPU 50 of the smartphone 180 detects that the smartphone 180 is attached to the HMD 110 connected to the computer 200. For example, the CPU 50 detects the attachment by detecting that the charging interface of the smartphone 180 is connected to the terminal of the HMD 110.

In step S1515, the CPU 50 starts the camera application and turns on the camera 54.

In step S1520, the CPU 50 photographs the two-dimensional code 102 printed on the goods (for example, the mug 101) with the camera 54 based on the operation by the user 190, and saves the image data of the two-dimensional code in the flash memory 55. To do.

In step S1525, the CPU 50 extracts access information for accessing the VR content from the two-dimensional code. The access information is created in advance by a VR content provider or the like, and includes, for example, a content ID, an expiration date of the VR content, and the like. In another aspect, the access information may include location information of a store or other sales place where the goods are sold. By subjecting the location information to the authentication, even if the access information is illegally copied, the use of the illegally acquired access information can be prevented by performing the authentication using the location information.

In step S1530, the CPU 50 transmits the access information of the management server (for example, the server 150) of the service that provides the VR content together with the user ID of the smartphone 180 via the computer 200. In another aspect, the CPU 50 may transmit the location information of the smartphone 180 (that is, the information for identifying the place where the access information is acquired). When the access information is authenticated as legitimate information, the server 150 reads the content data and transmits it to the computer 200. Content data is transmitted, for example, by stream distribution.

In step S1535, the CPU 50 receives the content data for reproducing the VR content from the server 150. In step S1540, the CPU 50 displays the VR content on the monitor 63 using the received content data.

In step S1545, the CPU 50 captures a scene of the VR content with the camera 54 based on the operation of the user 190. The user 190 can arbitrarily determine the scene to be photographed. In another aspect, the scene may be predetermined by the provider of the VR content. In yet another aspect, the scene may be recommended to the user 190 based on the shooting history by another user viewing the same VR content or the recommendation level or other comments entered by the other user.

In step S1550, the CPU 50 associates the captured image with the information of the two-dimensional code and saves it in the flash memory 55. In a certain aspect, the stored data is temporarily held in the smartphone 180, and the information for identifying the scene where the shooting was performed and the information of the two-dimensional code are transmitted from the computer 200 to the server 150. The server 150 manages the data received from the computer 200, and at a later date, in response to a request from the user 190, transfers the captured image data to the terminal (smartphone 180, home personal computer, etc.) used by the user 190. Can be sent. At this time, the server 150 can further transmit advertising data, promotion information, and the like associated with the VR content. As a result, information according to the preference of the user 190 can be provided to the user 190. In step S1555, the CPU 50 detects that the smartphone 180 has been removed from the HMD 110.

In step S1560, the CPU 50 activates the photo application and displays the captured image on the monitor 63 based on the operation of the user 190. This display may be either a display using image data non-volatilely held in the smartphone 180 or a display using data temporarily held in the RAM 56. In another aspect, the user 190 may enter a comment while looking at the image. The input comment is transmitted from the terminal displaying the image to the server 150. The server 150 holds the comment in association with the VR content. When another user views the same VR content as the VR content, the comment may be provided to the other user. In this way, data indicating the preference of each user is accumulated in the VR content provider, so that the user who browses the VR content or the image can be provided with the advertisement according to the preference.

[Reproduction processing of captured images]
The control structure of the smartphone 180 will be described with reference to FIG. FIG. 16 is a flowchart illustrating a part of the process executed by the smartphone 180 to display the image taken during the reproduction of the VR content. This process is executed, for example, when the user 190 operates the smartphone 180 in a place other than the store, for example, at home or in a train.

In step S1610, the CPU 50 activates an application for displaying a photo based on the operation of the user 190.

In step S1620, the CPU 50 displays an image taken when the user 190 views the VR content in the store on the monitor 63 based on the touch operation for selection by the user 190. Information associated with the image, such as content ID, image ID and other attribute information, is also loaded into the RAM 56.

In step S1630, the CPU 50 accepts the input of the user ID and password based on the touch operation of the user 190 on the monitor 63.

In step S1635, the CPU 50 transmits the user ID and password to the management server (for example, server 150). In step S1640, the CPU 50 accesses the management server and establishes communication. In step S1645, the CPU 50 transmits the content ID, the image ID (frame number), and the user ID to the management server.

In step S1650, the CPU 50 receives the advertisement data from the management server. In step S1655, the CPU 50 displays an advertisement on the monitor 63 based on the received advertisement data. When a comment by another user who has taken the image is registered in the server 150, the server 150 may send the comment to the smartphone 180. In this case, since the CPU 50 displays comments by other users on the monitor 63, the user 190 can know the impressions of other users on the image.

[Screen display mode]
A screen display mode according to the present embodiment will be described with reference to FIGS. 17 and 18. FIG. 17 is a diagram showing a screen displayed on a monitor 16 installed in a store. FIG. 18 is a diagram showing a transition of the screen on the monitor 63 of the smartphone 180.

As shown in FIG. 17, the monitor 16 displays a screen showing the waiting status to users other than the user 190. This screen is displayed, for example, when the user 190 is wearing the HMD 110 and is viewing VR content using the smartphone 180 fitted in the HMD 110. More specifically, the monitor 16 displays, as the waiting status, the time until the playback of the VR content by the user 190 is completed and the number of users waiting for their turn (number of waiting people). Further, the monitor 16 displays information indicating who the next user is. For example, since the two-dimensional code is attached to various goods (for example, mugs, T-shirts, cards, CDs, bags, etc.), the monitor 16 is based on the goods identification data included in the two-dimensional code. Can show buyers. In another aspect, the monitor 16 may display a user ID registered as a user who receives playback of VR content.

(Summary)
As described above, according to the present embodiment, it is possible to easily provide a virtual reality space to many users. Further, according to the present embodiment, each user can enjoy the VR content using his / her own smartphone or other terminal, and can shoot a desired scene, so that he / she enjoyed watching the VR content. Later, you can enjoy the captured image.

In another aspect, the VR content to be reproduced after reading the two-dimensional code may be randomly determined, for example, by lottery, regardless of the information contained in the two-dimensional code. As an example, when reading a two-dimensional code, for example, one of 20 types of VR contents is selected by lottery as a reproduction target. In this case, the rarity of the VR content in which a plurality of characters appear is higher than that in the VR content in which one character appears. Therefore, the frequency with which the VR content in which a plurality of characters appear is played may be set lower than the frequency with which the VR content in which one character appears is played.

In another aspect, when another user is waiting to play the VR content, the computer 200 sends a predetermined message in the voice of the character presumed to be the other user's preference during the waiting time. It may be sent to the user. In this way, the expectation of the other user can be increased during the waiting time.

In another aspect, the viewed VR content, the viewing user, and the captured scene are associated with each other and stored in the server 150. The VR content and scene are pre-associated with the advertisement. Since the server 150 can provide the advertisement to the user based on the VR content or the shot scene, the possibility of providing the advertisement according to the user's taste can be increased.

In the above embodiment, the VR content is specified by the information included in the two-dimensional code. In another aspect, the VR content may be output randomly. For example, the information contained in the two-dimensional code may be used as a trigger for the server 150 to randomly extract VR contents. If the VR content is randomly output, the user 190 cannot know the content until he / she visually recognizes the VR content to be played, which may increase the interest.

Further, in another aspect, the avatar of the user 190 may be added at the time of shooting a scene of the VR content. For example, the avatar object of the user 190 may be added to a predetermined location when the user 190 takes a picture, and the images may be combined.

<Second embodiment>
Hereinafter, the second embodiment will be described. In this embodiment, the access information given to the card, ticket or other medium is used to access the VR content.

With reference to FIG. 19, the provision of VR content in the present embodiment will be described. FIG. 19 is a diagram showing an outline of the configuration of the content providing system 1900. The content providing system 1900 includes a ticket shelf 1910, a card reader 1920, a computer 200, an HMD 110, and a monitor 16. The content providing system 1900 is arranged in, for example, an animation shop, a character shop, a convenience shop, or other stores. The content providing system 1900 is connected to the server 150 via the network 19.

Tickets 1911 to 1915 for viewing the provided VR content are displayed on the ticket shelf 1910. Each card contains an IC chip 1930. The IC chip 1930 stores access information. When the user 190 purchases a card, the user 190 holds the card over the card reader 1920 to execute the activation process. The data read from the IC chip 1930 includes a ticket ID, a content authentication number, and other access information. The read data is input to the computer 200. The computer 200 transmits the input data to the server 150 and authenticates whether or not the card is a legitimate card. When the cart is authenticated as a legitimate card, the server 150 transmits the data of the VR content specified by the card to the computer 200. The computer 200 transmits the received data to the HMD 110. The smartphone 180 connected to the HMD 110 displays VR content based on the data.

[data structure]
The data structure of the server 150 according to the present embodiment will be described with reference to FIG. FIG. 20 is a diagram showing an aspect of data storage in the storage 154 included in the server 150. The storage 154 includes a table 2100, a table 1410, a table 2120, and a table 1430.

Table 2100 includes a ticket ID 2101, a content authentication number 2102, a sales date and time 2103, and a sales terminal 2104. Ticket ID 2101 identifies the ticket sold at each store. The content authentication number 2102 controls access to VR content that can be played with the ticket. For example, when the content authentication number transmitted from the computer 200 and the content authentication number 2102 match, the CPU 150 determines that the ticket specified by the content authentication number and the ticket ID is valid. The sale date and time 2103 represents the date and time when the ticket was sold. For example, when the ticket is read by the card reader 1920, it is stored in the table 2100 as the sale date and time 2103. The sales terminal 2104 represents a device used at the place where the ticket was sold. For example, the sales terminal 2104 is specified as a POS (Point of Sales) system, a card reader 1920, or a computer 200 arranged in a store.

Table 1410 includes the content ID 1411, the content data 1412, the number of reproductions 1413, and the last reproduction date and time 1414, as in the above-described embodiment.

The table 2120 includes a playback date and time 1421, a playback location 1422, a content ID 1423, a user ID 1424, a terminal ID 1425, and a ticket ID 2126. Ticket ID 2126 identifies a ticket that has been authenticated and determined to be a legitimate ticket for viewing the VR content.

Table 1430 includes an access date and time 1431, an access location 1432, a content ID 1433, a frame number 1434, a user ID 1435, and a terminal ID 1436.

A procedure for viewing VR content in the present embodiment will be described with reference to FIG. 21. FIG. 21 is a flowchart showing the flow of procedures executed by the user 190.

In step S2110, the user 190 visits an anime shop, a character shop, and other shops. In step S2115, the user 190 purchases a ticket 1911 for viewing VR content at the shop. In step S2120, user 190 activates the purchased ticket 1911. For example, ticket 1911 is activated using a terminal by shop staff.

In step S2125, the user 190 holds the activated ticket 1911 over the card reader 1920 to authenticate the identification number of the VR content recorded in the ticket 1911. More specifically, the computer 200 to which the card reader 1920 is connected transmits the information read from the ticket 1911 (for example, the ticket ID, the content authentication number, etc.) to the server 150. The server 150 compares the ticket ID 2101 and the content authentication number 2102 stored in the table 2100 of the storage 154 with the ticket ID and the content authentication number received from the computer 200, and whether or not the ticket is a legitimate ticket. To judge.

In step S2130, the user 190 fits the smartphone 180 into the HMD 190 and wears the HMD 190 on his head. If the purchased ticket is a legitimate ticket, the server 150 transmits the VR content data to the HMD 110, and the smartphone 180 displays the VR content based on the data.

In step S2135, the user 190 experiences the VR content displayed on the monitor 63 of the smartphone 180. In a certain aspect, the user 190 may be able to participate in the VR content as an avatar object in addition to viewing the VR content.

In step S2140, the user 190 shoots a scene of VR content by operating the controller 160 or moving the line of sight and pressing the shooting button.

In step S2145, when the viewing of the VR content is completed, the user 190 removes the smartphone 180 from the HMD 110 and leaves the store. In step S2150, the user 190 browses the website of the service that provides the VR content by using the smartphone 180, the personal computer at home, or the like. In step S2155, the user 190 inputs the identification code and the user ID described in the purchased ticket 1911 into the website and accesses the site.

In step S2160, the user 190 registers personal information (address, name, telephone number, e-mail address, user's preference selected from a predetermined list, etc.) in the user account of the site. When the server 150 detects that the personal information has been registered, the server 150 reads the data captured while the user is viewing the VR content from the storage 154 to the memory 152, and the terminal (smartphone 180) used by the user 190. , Personal computer, etc.) to send image data.

In step S2165, the user 190 confirms the captured image while experiencing the VR content.

In step S2170, the user 190 uploads the captured image to the user account registered in the SNS (Social Network Service). Other users can enjoy the images taken by the user 190 by accessing the public page of the user account. Further, when a comment on the captured image is given by the user 190, the comment may also be displayed.

The control structure of the content providing system 1900 will be described with reference to FIG. 22. FIG. 22 is a flowchart showing a part of the processing executed by the content providing system 1900. The process is executed by the server 150 or the computer 200.

In step S2210, the processor 10 of the computer 200 detects that the purchased ticket has been activated based on the signal from the POS terminal or other terminal. In step S2215, the processor 10 receives the identification code read by the card reader 1920 from the card reader 1920. In step S2220, the processor 10 transmits the received identification code to the server 150.

In step S2225, the processor 151 of the server 150 executes the authentication process and determines whether or not the ticket 1911 is a legitimate ticket. For example, the processor 151 determines whether or not the ticket 1911 is valid based on the comparison between the ticket ID 2101 and the content authentication number 2102 registered in advance in the storage 154 and the ticket ID and the content authentication number received from the computer 200. To judge. When the processor 151 determines that the ticket 1911 is valid (YES in step S2225), it switches control to step S2230. If not (NO in step S2225), processor 151 switches control to step S2270.

In step S2230, the processor 151 reads the content data associated with the ticket ID from the storage 154 and transmits the content data to the computer 200.

In step S2235, the processor 10 of the computer 200 generates a video signal for presenting a video based on the content data in the virtual space. In another aspect, the CPU 50 of the smartphone 180 may generate the video signal. In step S2240, the processor 10 outputs the generated video signal to the HMD 110. The video signal is input to the smartphone 180 mounted on the HMD 110.

In step S2245, the CPU 50 of the smartphone 180 outputs a part of the video signal to the monitor 63. The monitor 63 displays an image of VR content based on the video signal. The user 190 wearing the HMD in which the smartphone 180 is fitted can view the VR content by visually recognizing the image.

In step S2250, the processor 10 of the computer 200 determines whether or not the reproduction of the VR content based on another ticket is waiting based on the presence or absence of the signal from the card reader 1920. When the processor 10 determines that the reproduction of the VR content is waiting (YES in step S2250), the processor 10 switches the control to step S2255. If not (NO in step S2250), the processor 10 ends the process. In step S2255, the processor 10 displays the waiting time of the next viewer on the monitor 63. In step S2260, the processor 10 calls the next viewer by outputting the sound of the character of the VR content from the speaker (not shown).

In step S2270, the processor 151 of the server 150 notifies the computer 200 that the ticket is not valid. Upon receiving this notification, the computer 200 may display on the monitor 16 that the ticket is not valid.

[Screen display mode]
A screen display mode in the content providing system according to the present embodiment will be described with reference to FIG. 23. FIG. 23 is a diagram showing an aspect of a screen displayed by the monitor 16 for notifying the waiting status.

As shown in FIG. 23, the monitor 16 displays a screen showing the waiting status to users other than the user 190. This screen is displayed, for example, when the user 190 is wearing the HMD 110 and is viewing VR content using the smartphone 180 fitted in the HMD 110. More specifically, the monitor 16 waits for the time until the user 190 finishes playing the VR content (“about 2 minutes”) and the number of users waiting for their turn (“3 people”). And are displayed. Further, the monitor 16 displays information indicating who the next user is. For example, the monitor 16 may display the ticket number given to the ticket purchased by each user. In another aspect, the monitor 16 may display the user ID registered as the user who receives the reproduction of the VR content, or the title or character of the VR content to be reproduced.

In another aspect, a mode in which the ticket and the two-dimensional code are used together may be used. For example, when the user 190 offers to the staff of the store to purchase a ticket, a serial number is attached to the ticket in advance, and the two-dimensional code uses one of the serial numbers attached to the ticket as access information. It may be included.

In this way, when the card reader 1920 reads the serial number information at the time of viewing the content, it is possible to present to the waiting user how many users have viewed the content. In addition to the serial number information, by including the ticket identification information (ticket ID) in the two-dimensional code or the IC chip 1930, the user 190 can view the captured image on the smartphone 180 after viewing the VR content. At that time, the information of the user 190 itself (for example, the login information in the SND account) and the information for identifying the ticket are recorded.

When the user 190 purchases the ticket again, views and shoots the VR content, the user 190 performs the operation for viewing the shot image together with the login operation using the application or the like. By this operation, the information identifying the ticket and the information of the user 190 itself are associated with the VR content or the captured image.

By accumulating such information in the server 150, the content provider can know which image is particularly preferred among the captured images in order to understand the behavior of each user. For example, it is presumed that among the captured images, those that are viewed frequently by the user match the user's preference. More specifically, for example, when the number of times the photographed image of a certain character is viewed is larger than the number of times the photographed image of another character is viewed, it is presumed that the certain character is the user's preference.

From the above, the disclosed technical features are summarized as follows. (Structure 1) The content providing method executed on the computer 200 for providing the content using the HMD 110 is the access information for accessing the content (for example, the content ID, the ticket ID, etc.) through the interface of the computer 200. A step of accepting input of (content ID, ticket ID, content authentication number, etc.) (content ID, ticket ID, content authentication number, etc.), a step of transmitting the access information to the server 150 for managing one or more contents, and a step of transmitting the access information from the server 150. It includes a step of receiving content data for displaying the content, a step of defining a virtual space 2 for presenting the content using the HMD 110, and a step of causing the HMD 110 to reproduce the content using the content data.

(Structure 2) As for the content providing method, in addition to the configuration 1, the HMD 110 includes a camera. The steps of accepting the input of access information are a step of photographing a code including access information (for example, a two-dimensional code) using a camera, a step of accepting an input of an image signal obtained by photographing, and an access information from the image signal. Includes steps to extract.

(Structure 3) In addition to the configuration 1, the step of accepting the input of the access information is described in the configuration 1 including the step of acquiring the access information from the medium (for example, tickets 1911 to 1915) in which the access information is recorded. the method of.

(Structure 4) Preferably, the content providing method further includes a step of displaying the content being played on the HMD 110 on the monitor 16 connected to the computer 200.

(Structure 5) Preferably, the content providing method further includes a step of displaying the status of waiting for use of the HMD 110 on the monitor 16 connected to the computer 200.

(Structure 6) Preferably, the content providing method further includes a step of outputting advertising information associated with the content.

(Structure 7) Preferably, the content providing method further includes a step of presenting an avatar object corresponding to the user 190 of the HMD 110 together with the content.

(Structure 8) Preferably, the content providing method further includes a step of shooting the content being reproduced.

(Structure 9) Preferably, the content providing method further includes a step of displaying an image acquired by shooting.

(Structure 10) Preferably, the content providing method further includes a step of displaying a user 190 interface that accepts input of a comment for an image acquired by shooting.

(Structure 11) Preferably, the content providing method is a step of outputting the sound of a character included in the next content to be played after the playback of the content is completed and prompting the viewer of the next content to wear the HMD 110. Including further.

(Structure 12) Preferably, the content providing method further includes a step of transmitting the position information indicating the place where the content is reproduced and the identification data associated with the position information to the server 150.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and it is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 virtual camera, 2 virtual space, 5 reference line of sight, 10,151 processor, 11,152 memory, 12,154 storage, 13 input / output interfaces, 14,65,153 communication interfaces, 15 buses, 16,63 monitors, 19 networks , 21 center, 22 virtual space image, 23 view area, 24, 25 area, 26 view image, 30 grips, 31 frames, 32 top surface, 33, 34, 36, 37 buttons, 38 analog sticks, 51, 68 interfaces.

Claims (7)

A computer including a terminal device, a head mount device, and a management server, which is related to the head mount device, inputs access information for accessing the content managed by the management server. Reception means to accept from
A definition means for acquiring content managed by the management server corresponding to the access information and defining a virtual space for presenting the content .
An output means for outputting the content to the head-mounted device,
A shooting means for shooting at least one scene of the output content is provided.
The terminal device includes a storage means for storing an image of at least one captured scene in association with the access information.
Information processing system with. It said accepting means for accepting an input of the access information is attached accept an input of an image signal obtained by photographing the code that contains the access information,
The computer associated with the head mount device
The information processing system according to claim 1, further comprising an extraction means for extracting the access information from the image signal. The computer associated with the head mount device
A transmission means for transmitting an image of at least one captured scene to the management server in association with the access information is provided.
The management server is a receiving means for receiving the image of the one scene associated with the access information, and
A transmission means for transmitting an image of the one scene associated with the access information to the terminal device in response to a request from the terminal device is further provided.
The storage means of the terminal device receives and stores an image of the one scene associated with the access information transmitted by the transmission means.
The information processing system according to claim 1. Wherein the transmission unit of the management server, in response to said request of the terminal device, the access information and added prior Symbol 1 scene image associated, wherein at least one of the advertisement data or promotional information associated with the content Send to the terminal device,
The information processing system according to claim 3. The terminal device, in response to a request of the terminal device, the access information and added prior Symbol 1 scene image associated, at least receive one from the outside of the advertisement data or promotional information associated with the content,
The information processing system according to claim 3. The terminal device includes a comment attachment means to attach the comment to the front Symbol 1 scene of images the memory,
The management server further includes the comment and a comment transmitting means for transmitting to the management server, and the management server holds the comment and the content in association with each other .
The information processing system according to claims 1 to 4. The information processing system according to claim 6, wherein when the content is viewed on the head mount device, the output means outputs a comment corresponding to the content held in the management server.

Images