JP6927797B2 - Methods, programs and computers for providing virtual space to users via headmount devices - Google Patents

Description

The present disclosure relates to methods, programs and computers for providing a virtual space to a user via a head mount device.

In Patent Document 1, when the virtual camera moves in the virtual space due to a factor other than the movement of the head-mounted device (Head-Mounted Device, HMD) and the movement amount of the virtual camera exceeds a predetermined amount, the field image , A technique for reducing the amount of information is disclosed in a region other than a predetermined range centered on the visual axis.

The technique disclosed in Patent Document 1 is an example of a technique for reducing sickness (so-called "VR sickness") in a virtual reality (VR) space. In the future, it is expected that various technologies for reducing VR sickness will be developed without impairing the virtual experience.

Japanese Unexamined Patent Publication No. 2017-59196

It is an object of the present disclosure to provide a novel method for reducing VR sickness without impairing the virtual experience.

According to one embodiment of the present disclosure, it is a method performed by a computer for providing a virtual space to a user via a head mount device.
The step of operating the virtual camera arranged in the virtual space and
The step of specifying the moving direction of the virtual camera and
A step of generating a field of view image representing the virtual space within the field of view of the virtual camera, and
A step of specifying the degree of decrease in visibility in the moving direction based on the first degree of decrease information that defines the degree of decrease in visibility in one or more directions determined based on the orientation of the virtual camera.
A step of reducing the visibility of the visual field image based on the specified degree of deterioration of the visibility, and a step of reducing the visibility of the visual field image.
A step of outputting the field-of-view image with reduced visibility to the display unit of the head-mounted device, and
Methods are provided, including.

According to another embodiment of the present disclosure, a program is provided for causing a processor to perform the above method.

According to another embodiment of the present disclosure, it is a computer.
By the control provided by the processor of the computer to provide the virtual space to the user via the head mount device.
The step of operating the virtual camera arranged in the virtual space and
The step of specifying the moving direction of the virtual camera and
A step of generating a field of view image representing the virtual space within the field of view of the virtual camera, and
A step of specifying the degree of decrease in visibility in the moving direction based on the first degree of decrease information that defines the degree of decrease in visibility in one or more directions determined based on the orientation of the virtual camera.
A step of reducing the visibility of the visual field image based on the specified degree of deterioration of the visibility, and a step of reducing the visibility of the visual field image.
A step of outputting the field-of-view image with reduced visibility to the display unit of the head-mounted device, and
Is run, a computer is provided.

According to the embodiment of the present disclosure, it is possible to provide a novel method for reducing VR sickness without impairing the virtual experience.

Other features and advantages of the present disclosure will become apparent from the description of embodiments described below, the accompanying drawings and the claims.

It is a figure which shows the structure of the HMD system schematicly. It is a block diagram which shows the example of the basic hardware composition of the computer by one Embodiment of this disclosure. It is a figure that conceptually represents the uvw field-of-view coordinate system set in the HMD according to one embodiment. It is a figure which conceptually represents one aspect which expresses a virtual space according to one Embodiment. It is the figure which showed the head of the user who wears an HMD according to one Embodiment from the top. It is a figure which shows the yz cross section which looked at the visual area from the x direction in a virtual space. It is a figure which shows the xz cross section which looked at the visual area in a virtual space from the y direction. It is a figure which shows the schematic structure of the controller according to one Embodiment. It is a figure which shows the schematic structure of the controller according to one Embodiment. FIG. 3 is a block diagram showing a function of a computer for providing a virtual experience to a user via an HMD system according to an embodiment of the present disclosure. It is a flow diagram of a general process for displaying an image of a virtual space in which a user is immersed on a display unit. It is a flowchart of the method by one Embodiment of this disclosure. It is a flowchart which shows the specific example of the process executed in step 1110 of FIG. It is a figure which shows the state which the avatar corresponding to a user moves in a virtual space in one Embodiment. It is a figure which shows how the moving direction is decomposed into a plurality of components. It is a figure which shows the example of the 1st decrease degree information. It is a figure which shows the state which the avatar corresponding to a user moves in a virtual space in one Embodiment. It is a figure which shows how the moving direction is decomposed into a plurality of components. It is a figure which shows how the moving direction is decomposed into a plurality of components. It is a figure which shows how the moving direction is decomposed into a plurality of components. It is a figure which shows the state which the avatar corresponding to a user moves in a virtual space in one Embodiment. It is a figure which shows the example of the 2nd degree of decrease information. It is a figure which shows the example of the 2nd degree of decrease information which was read from the graph of FIG. 19 and was quantified. It is a figure which shows the example of the field-of-view image with reduced visibility generated according to the embodiment of this disclosure.

[Explanation of Embodiments of the present disclosure]
First, the configurations of exemplary embodiments of the present disclosure will be listed and described. The methods, programs and computers according to the embodiments of the present disclosure may have the following configurations.

(Item 1)
A method performed by a computer to provide a virtual space to a user via a headmount device.
The step of operating the virtual camera arranged in the virtual space and
The step of specifying the moving direction of the virtual camera and
A step of generating a field of view image representing the virtual space within the field of view of the virtual camera, and
A step of specifying the degree of decrease in visibility in the moving direction based on the first degree of decrease information that defines the degree of decrease in visibility in one or more directions determined based on the orientation of the virtual camera.
A step of reducing the visibility of the visual field image based on the specified degree of deterioration of the visibility, and a step of reducing the visibility of the visual field image.
A step of outputting the field-of-view image with reduced visibility to the display unit of the head-mounted device, and
Including methods.

(Item 2)
The first degree of decrease information determines the degree of decrease in visibility in a plurality of axial directions determined based on the orientation of the virtual camera.
The step of specifying the degree of decrease in visibility is
A step of decomposing the moving direction into the plurality of axial components, and
A step of specifying the degree of decrease in visibility for each component in the moving direction by using the first degree of decrease information and the plurality of axial components.
A step of specifying the degree of decrease in visibility in the moving direction based on the degree of decrease in visibility for each of the specified components, and a step of specifying the degree of decrease in visibility in the moving direction.
The method according to item 1.

(Item 3)
The axial component indicates the ratio of movement in the axial direction to the movement direction.
In the disassembling step, the moving direction is decomposed into the plurality of axial components so that the smaller the angle difference between the axial direction and the moving direction, the larger the value of the component in the axial direction.
In the step of specifying the degree of decrease in visibility for each component in the moving direction, the component in the axial direction is added to the degree of decrease in visibility in the axial direction indicated by the first degree of decrease information for each axial direction. By multiplying, the degree of decrease in visibility for each component in the moving direction is specified.
The step according to item 2, wherein in the step of specifying the degree of decrease in visibility in the moving direction, the degree of decrease in visibility for each of the specified components is added up to specify the degree of decrease in visibility in the moving direction. Method.

(Item 4)
The plurality of axial directions include a roll axial direction, a pitch axial direction, and a yaw axial direction.
The method according to item 2 or 3, wherein the first decrease degree information defines the degree of decrease in visibility in the roll axis direction to be smaller than the degree of decrease in visibility in the pitch axis direction and the yaw axis direction. ..

(Item 5)
The first degree of decrease information indicates that the degree of decrease in visibility of the virtual camera in the roll axis direction in the front direction is larger than the degree of decrease in visibility in the direction opposite to the front direction in the roll axis direction. The method according to item 4, which is defined as small.

(Item 6)
Further including a step of specifying the rotation speed of the virtual camera,
In the step of specifying the degree of decrease in visibility, the moving direction and the specification are based on the first degree of decrease information and the second degree of decrease information that determines the degree of decrease in visibility with respect to the rotation speed of the virtual camera. The method according to any one of items 1 to 5, which specifies the degree of decrease in visibility at the said rotation speed.

(Item 7)
The method according to item 6, wherein the second decrease degree information is defined so that the degree of decrease in visibility increases as the rotation speed increases until the rotation speed reaches a predetermined speed.

(Item 8)
A program for causing a processor to execute the method according to any one of items 1 to 7.

(Item 9)
It ’s a computer,
By the control provided by the processor of the computer to provide the virtual space to the user via the head mount device.
The step of operating the virtual camera arranged in the virtual space and
The step of specifying the moving direction of the virtual camera and
A step of generating a field of view image representing the virtual space within the field of view of the virtual camera, and
A step of specifying the degree of decrease in visibility in the moving direction based on the first degree of decrease information that defines the degree of decrease in visibility in one or more directions determined based on the orientation of the virtual camera.
A step of reducing the visibility of the visual field image based on the specified degree of deterioration of the visibility, and a step of reducing the visibility of the visual field image.
A step of outputting the field-of-view image with reduced visibility to the display unit of the head-mounted device, and
The computer on which it runs.

[Details of Embodiments of the present disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, similar elements are designated by the same reference numerals. The same applies to their names and functions. Duplicate description of such elements is omitted.

The configuration of the head-mounted device (HMD) system 100 will be described with reference to FIG. FIG. 1 schematically shows the configuration of the HMD system 100. In one example, the HMD system 100 is provided as a home system or a business system. The HMD may be a so-called head-mounted display provided with a display unit, or may be a head-mounted device to which a terminal such as a smartphone having a display unit can be attached.

The HMD system 100 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a display unit 112 and a gaze sensor 140. The controller 160 may include a motion sensor 130.

In one example, the computer 200 may be able to connect to a network 192 such as the Internet, or may be able to communicate with a computer such as a server 150 connected to the network 192. In another embodiment, the HMD 110 may include a sensor 114 instead of the HMD sensor 120.

The HMD 110 may be worn on the head of the user 190 and provide the user with virtual space during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the display unit 112, respectively. When each eye of the user visually recognizes each image, the user can recognize the image as a three-dimensional image based on the parallax of both eyes.

The display unit 112 is realized as, for example, a non-transparent display device. In one example, the display unit 112 is arranged on the main body of the HMD 110 so as to be located in front of both eyes of the user. Therefore, the user can immerse himself in the virtual space by visually recognizing the three-dimensional image displayed on the display unit 112. In certain embodiments, the virtual space includes, for example, a background, user-operable objects, user-selectable menu images, and the like. In certain embodiments, the display unit 112 can be realized as a liquid crystal display unit or an organic EL (Electroluminescence) display unit included in an information display terminal such as a smartphone.

In one example, the display unit 112 may include a sub-display unit for displaying an image for the right eye and a sub-display unit for displaying an image for the left eye. In another aspect, the display unit 112 may be configured to display the image for the right eye and the image for the left eye as a unit. In this case, the display unit 112 includes a high-speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

In one example, the HMD 110 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. More specifically, the HMD sensor 120 may read a plurality of infrared rays emitted by the HMD 110 to detect the position and inclination of the HMD 110 in the real space.

In some embodiments, the HMD sensor 120 may be implemented by a camera. In this case, the HMD sensor 120 can detect the position and tilt of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include a sensor 114 as a position detector instead of the HMD sensor 120. The HMD 110 can detect the position and tilt of the HMD 110 itself by using the sensor 114. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 detects its position and inclination by using any of these sensors instead of the HMD sensor 120. obtain. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in real space over time. The HMD 110 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle. Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance. Further, the visual field image may include a configuration for presenting the real space as a part of the image constituting the virtual space. For example, an image taken by a camera mounted on the HMD 110 may be superimposed on a part of the field of view image and displayed, or by setting a high transmittance of a part of the transmissive display device, the field of view image can be displayed. The real space may be visible from a part.

The gaze sensor 140 detects the direction (line of sight) to which the eyes of the right eye and the left eye of the user 190 are directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In some embodiments, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the angle of rotation of each eyeball by receiving the reflected light from the cornea and the iris with respect to the irradiation light. .. The gaze sensor 140 can detect the line of sight of the user 190 based on each of the detected rotation angles.

The server 150 may send the program to the computer 200. In another aspect, the server 150 may communicate with another computer 200 to provide virtual reality to the HMD used by other users. For example, in an amusement facility, when a plurality of users play a participatory game, each computer 200 communicates a signal based on the operation of each user with another computer 200, and the plurality of users are common in the same virtual space. Allows you to enjoy the game.

The controller 160 is connected to the computer 200 by wire or wirelessly. The controller 160 receives an instruction input from the user 190 to the computer 200. In some embodiments, the controller 160 is configured to be grippable by the user 190. In another embodiment, the controller 160 is configured to be wearable on a part of the body or clothing of the user 190. In another embodiment, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the computer 200. In another aspect, the controller 160 receives from the user 190 an operation for controlling the position and movement of an object arranged in the virtual space.

In some embodiments, the motion sensor 130 is attached to the user's hand to detect the movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, the number of rotations, and the like of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided on, for example, a glove-shaped controller 160A. In certain embodiments, for safety in real space, it is desirable that the controller 160A be attached to something that does not easily fly by being attached to the user 190's hand, such as a glove type. In another embodiment, a controller 160B having a general structure having a plurality of operation buttons may be used. In another embodiment, a sensor not attached to the user 190 may detect the movement of the user 190's hand. Optical sensors may be used to detect the position, orientation, direction of movement, distance of movement, etc. of various parts of the body of the user 190. For example, the signal of the camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190. As an example, the motion sensor 130 and the computer 200 are wirelessly connected to each other. In the case of wireless communication, the communication mode is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication method is used.

The computer 200 according to the embodiment of the present disclosure will be described with reference to FIG. FIG. 2 is a block diagram showing an example of a basic hardware configuration of the computer 200 according to the embodiment of the present disclosure. The computer 200 includes a processor 202, a memory 204, a storage 206, an input / output interface 208, and a communication interface 210 as main components. Each component is connected to bus 212, respectively.

The processor 202 executes a series of instructions included in the program stored in the memory 204 or the storage 206 based on the signal given to the computer 200 or when a predetermined condition is satisfied. In some embodiments, the processor 202 is realized as a device such as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or an FPGA (Field-Programmable Gate Array).

Memory 204 temporarily stores programs and data. The program is loaded from storage 206, for example. The data includes data input to the computer 200 and data generated by the processor 202. In some embodiments, the memory 204 is realized as a volatile memory such as a RAM (Random Access Memory).

Storage 206 permanently holds programs and data. The storage 206 is realized as, for example, a non-volatile storage device such as a ROM (Read-Only Memory), a hard disk device, or a flash memory. The program stored in the storage 206 includes a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, a program for realizing communication with another computer 200, and the like. The data stored in the storage 206 includes data, objects, and the like for defining the virtual space.

In another aspect, the storage 206 may be realized as a removable storage device such as a memory card. In yet another embodiment, a configuration using programs and data stored in an external storage device may be used instead of the storage 206 built into the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used such as an amusement facility, it is possible to update programs and data at once.

In certain embodiments, the input / output interface 208 communicates signals with the HMD 110, the HMD sensor 120, and the motion sensor 130. In some embodiments, the input / output interface 208 is implemented using terminals such as USB (Universal Serial Bus, USB), DVI (Digital Visual Interface), HDMI® (High-Definition Multimedia Interface), and the like. The input / output interface 208 is not limited to the above.

In certain embodiments, the input / output interface 208 may further communicate with the controller 160. For example, the input / output interface 208 receives input of signals output from the controller 160 and the motion sensor 130. In another embodiment, the input / output interface 208 sends an instruction output from the processor 202 to the controller 160. The command instructs the controller 160 to vibrate, output voice, emit light, and the like. Upon receiving the command, the controller 160 executes vibration, audio output, light emission, and the like in response to the command.

The communication interface 210 is connected to the network 192 and communicates with another computer (for example, the server 150) connected to the network 192. In some embodiments, the communication interface 210 is realized as, for example, a wired communication interface such as LAN (Local Area Network) or a wireless communication interface such as WiFi (Wireless Field), Bluetooth®, NFC (Near Field Communication), etc. Will be done. The communication interface 210 is not limited to the above.

In some embodiments, the processor 202 accesses storage 206, loads one or more programs stored in storage 206 into memory 204, and executes a series of instructions contained in the programs. The one or more programs may include an operating system of a computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space, and the like. The processor 202 sends a signal for providing virtual space to the HMD 110 via the input / output interface 208. The HMD 110 displays an image on the display unit 112 based on the signal.

In the example shown in FIG. 2, the computer 200 is provided outside the HMD 110. However, in another aspect, the computer 200 may be built into the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including a display unit 112 may function as a computer 200.

Further, the computer 200 may have a configuration commonly used for a plurality of HMD 110s. According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space.

In one embodiment, the HMD system 100 has a preset global coordinate system. The global coordinate system has three reference directions (axes) that are parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-back direction orthogonal to both the vertical direction and the horizontal direction. In this embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction of real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the real space.

In some embodiments, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect a temporal change in the position and inclination of the HMD 110 by using each value detected over time.

The global coordinate system is parallel to the coordinate system in real space. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination of the HMD 110 around three axes in the global coordinate system. The HMD sensor 120 sets the uvw field of view coordinate system to the HMD 110 based on the tilt of the HMD 110 in the global coordinate system. The uvw field-of-view coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 sees an object in the virtual space.

The uvw field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw field coordinate system set in the HMD 110 according to a certain embodiment. The HMD sensor 120 detects the position and tilt of the HMD 110 in the global coordinate system when the HMD 110 is activated. Processor 202 sets the uvw field coordinate system to HMD110 based on the detected values.

As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user wearing the HMD 110. More specifically, the HMD 110 defines the global coordinate system in the horizontal direction, the vertical direction, and the front-back direction (x-axis, y-axis, z-axis) by the inclination of each axis of the HMD 110 in the global coordinate system. The three directions newly obtained by tilting each around the axis are set as the pitch direction (u-axis), the yaw direction (v-axis), and the roll direction (w-axis) of the uvw field coordinate system in the HMD 110.

In some embodiments, when the user 190 wearing the HMD 110 is upright and visually recognizing the front, the processor 202 sets the uvw field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch direction (u-axis) and yaw direction (v-axis) of the uvw field coordinate system in the HMD 110. And the roll direction (w-axis).

After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (change amount of the inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110, respectively. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

The HMD sensor 120 sets the uvw field coordinate system in the HMD 110 after the HMD 110 has moved to the HMD 110 based on the detected tilt angle of the HMD 110. The relationship between the HMD 110 and the uvw field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

In some embodiments, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between the points (eg, the distance between the points). The position of the above in the real space may be specified as a relative position with respect to the HMD sensor 120. Further, the processor 202 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually representing one aspect of expressing the virtual space 400 according to a certain embodiment. The virtual space 400 has an all-sky spherical structure that covers the entire center 406 in the 360-degree direction. In FIG. 4, the celestial sphere in the upper half of the virtual space 400 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 400. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system defined in the virtual space 400. The computer 200 associates each partial image constituting the content (still image, moving image, etc.) expandable in the virtual space 400 with each corresponding mesh in the virtual space 400, and creates a virtual space image visible to the user. The virtual space 400 to be expanded is provided to the user.

In some embodiments, the virtual space 400 defines an xyz coordinate system with the center 406 as the origin. The xyz coordinate system is, for example, parallel to the global coordinate system. Since the xyz coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the xyz coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. Therefore, the x-axis (horizontal direction) of the xyz coordinate system is parallel to the x-axis of the global coordinate system, and the y-axis (vertical direction) of the xyz coordinate system is parallel to the y-axis of the global coordinate system. The z-axis (front-back direction) is parallel to the z-axis of the global coordinate system.

When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 404 is arranged at the center 406 of the virtual space 400. In some embodiments, the processor 202 displays an image captured by the virtual camera 404 on the display unit 112 of the HMD 110. The virtual camera 404 moves in the virtual space 400 in the same manner in conjunction with the movement of the HMD 110 in the real space. As a result, changes in the position and orientation of the HMD 110 in the real space can be similarly reproduced in the virtual space 400.

As in the case of the HMD 110, the virtual camera 404 is defined with an uvw field coordinate system. The uvw field-of-view coordinate system of the virtual camera 404 in the virtual space 400 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 404 also changes accordingly. Further, the virtual camera 404 can also move in the virtual space 400 in conjunction with the movement of the user wearing the HMD 110 in the real space.

The processor 202 of the computer 200 defines the viewing area 410 in the virtual space 400 based on the placement position of the virtual camera 404 and the reference line of sight 408. The viewing area 410 corresponds to the area of the virtual space 400 that is visually recognized by the user wearing the HMD 110.

The line of sight of the user 190 detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user 190 visually recognizes an object. The uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the display unit 112. Further, the uvw field-of-view coordinate system of the virtual camera 404 is linked to the uvw field-of-view coordinate system of the HMD 110. Therefore, the HMD system 100 according to a certain aspect can consider the line of sight of the user 190 detected by the gaze sensor 140 as the line of sight of the user in the uvw field of view coordinate system of the virtual camera 404.

The determination of the line of sight of the user will be described with reference to FIG. FIG. 5 is a top view of the head of the user 190 wearing the HMD 110 according to an embodiment.

In some embodiments, the gaze sensor 140 detects each line of sight of the user 190's right and left eyes. In some embodiments, the gaze sensor 140 detects gazes R1 and L1 when the user 190 is looking closer. In another embodiment, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the line of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 identifies the gaze point N1 which is the intersection of the line of sight R1 and L1 based on the detected values. On the other hand, when the computer 200 receives the detected values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 identifies the line of sight N0 of the user 190 based on the position of the identified gazing point N1. The computer 200 detects, for example, the extending direction of the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1 as the line of sight N0. The line of sight N0 is the direction in which the user 190 actually directs the line of sight with both eyes. Further, the line of sight N0 corresponds to the direction in which the user 190 actually directs the line of sight with respect to the viewing area 410.

In another aspect, the HMD system 100 may include a microphone and a speaker in any part constituting the HMD system 100. The user can give a voice instruction to the virtual space 400 by speaking to the microphone.

In another aspect, the HMD system 100 may include a television broadcast reception tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 400.

In still another aspect, the HMD system 100 may include a communication circuit for connecting to the Internet or a telephone function for connecting to a telephone line.

The viewing area 410 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing a yz cross section of the visual recognition area 410 viewed from the x direction in the virtual space 400. FIG. 7 is a diagram showing an xz cross section of the visual recognition area 410 viewed from the y direction in the virtual space 400.

As shown in FIG. 6, the visible region 410 in the yz cross section includes the region 602. The area 602 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the yz cross section of the virtual space 400. The processor 202 defines a range including the polar angle α centered on the reference line of sight 408 in the virtual space as a region 602.

As shown in FIG. 7, the visible region 410 in the xz cross section includes the region 702. The area 702 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the xz cross section of the virtual space 400. The processor 202 defines a range including the azimuth angle β centered on the reference line of sight 408 in the virtual space 400 as the region 702. The polar angles α and β are determined according to the arrangement position of the virtual camera 404 and the orientation of the virtual camera 404.

In some embodiments, the HMD system 100 provides the user 190 with a field of view in virtual space by displaying a field of view image on the display unit 112 based on a signal from the computer 200. The visual field image corresponds to a portion of the virtual space image 402 that is superimposed on the visual field area 410. When the user 190 moves the HMD 110 attached to the head, the virtual camera 404 also moves in conjunction with the movement. As a result, the position of the visible area 410 in the virtual space 400 changes. As a result, the visual field image displayed on the display unit 112 is updated to an image superimposed on the visual field area 410 in the direction facing the user in the virtual space 400 among the virtual space images 402. The user can visually recognize the desired direction in the virtual space 400.

As described above, the direction (tilt) of the virtual camera 404 corresponds to the user's line of sight (reference line of sight 408) in the virtual space 400, and the position where the virtual camera 404 is arranged corresponds to the user's viewpoint in the virtual space 400. Therefore, by moving the virtual camera 404 (including the operation of changing the arrangement position and the operation of changing the orientation), the image displayed on the display unit 112 is updated, and the field of view (including the viewpoint and the line of sight) of the user 190 is moved. Will be done.

While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 402 developed in the virtual space 400 without visually recognizing the real world. Therefore, the HMD system 100 can give the user a high sense of immersion in the virtual space 400.

In some embodiments, the processor 202 may move the virtual camera 404 in the virtual space 400 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 202 identifies an image area (that is, a viewing area 410 in the virtual space 400) projected onto the display unit 112 of the HMD 110 based on the position and orientation of the virtual camera 404 in the virtual space 400.

According to certain embodiments, the virtual camera 404 may include two virtual cameras, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Further, appropriate parallax may be set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 400.

A controller 160A, which is an example of the controller 160, will be described with reference to FIG. 8A. FIG. 8A is a diagram showing a schematic configuration of a controller 160A according to an embodiment.

In some embodiments, the controller 160A may include a right controller and a left controller. For the sake of brevity, controller 160A in FIG. 8A shows the right controller. The right controller is operated by the right hand of the user 190. The left controller is operated by the left hand of the user 190. In some embodiments, the right controller and the left controller are symmetrically configured as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller and the left hand holding the left controller. In another embodiment, the controller 160 may be an integrated controller that accepts operations of both hands. Hereinafter, the right controller 160A will be described.

The right controller 160A includes a grip 802, a frame 804, and a top surface 806. The grip 802 is configured to be gripped by the right hand of the user 190. For example, the grip 802 may be held by the palm of the user 190's right hand and three fingers (middle finger, ring finger, little finger).

The grip 802 includes buttons 808 and 810 and a motion sensor 130. The button 808 is arranged on the side surface of the grip 802 and accepts an operation by the middle finger of the right hand. The button 810 is arranged in front of the grip 802 and accepts an operation by the index finger of the right hand. In some embodiments, the buttons 808, 810 are configured as trigger-type buttons. The motion sensor 130 is built in the housing of the grip 802. If the operation of the user 190 can be detected from around the user 190 by a camera or other device, the grip 802 may not include the motion sensor 130.

The frame 804 includes a plurality of infrared LEDs 812 arranged along its circumferential direction. The infrared LED 812 emits infrared rays as the program progresses during the execution of the program using the controller 160A. The infrared rays emitted from the infrared LED 812 can be used to detect each position and orientation (tilt, orientation) of the right controller 160A and the left controller (not shown). In the example shown in FIG. 8, infrared LEDs 812 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. An array of one column or three or more columns may be used.

The top surface 806 includes buttons 814 and 816 and an analog stick 818. Buttons 814 and 816 are configured as push buttons. Buttons 814 and 816 accept operations by the thumb of the user 190's right hand. In some embodiments, the analog stick 818 accepts an operation 360 degrees in any direction from the initial position (neutral position). The operation includes, for example, an operation for moving an object arranged in the virtual space 400.

In some embodiments, the right controller 160A and the left controller include a battery for driving a member such as an infrared LED 812. The battery may be either a primary battery or a secondary battery, and the shape thereof may be arbitrary such as a button type and a dry battery type. In another embodiment, the right controller 160A and the left controller can be connected, for example, to the USB interface of computer 200. In this case, the right controller 800 and the left controller may be powered via the USB interface.

A controller 160B, which is an example of the controller 160, will be described with reference to FIG. 8B. FIG. 8B is a diagram showing a schematic configuration of a controller 160B according to an embodiment.

The controller 160B includes a plurality of buttons 820 (820a, 820b, 820c, 820d) and 822 (822a, 822b, 822c, 822d), and left and right analog sticks 824L and 824R. Each button 820 and 822 is configured as a push button. Buttons 820 and 822 accept operations by the thumb of the user 190's hand. A trigger-type button (not shown) capable of accepting an operation by the index finger or the middle finger of the user 190's hand may be further provided in the controller 160B. In some embodiments, the analog sticks 824L and 824R accept operations in any direction 360 degrees from the initial position (neutral position), respectively. The operation includes, for example, an operation for moving an object arranged in the virtual space 400. Separate operations for buttons 820 (820a, 820b, 820c, 820d) and 822 (822a, 822b, 822c, 822d) and analog sticks 824L and 824R (and, if included, trigger-type buttons (not shown)). A command is assigned. The operation commands include, for example, a command for giving a command to an object in the virtual space 400, a command for making various settings on a game menu screen, and a computer when the user 190 is experiencing the virtual space 400. Includes any other command that can be entered in 200. The operation command assigned to each button or analog stick may be dynamically changed according to, for example, the progress of the game or the change of the scene.

In some embodiments, the controller 160B may include a plurality of infrared LEDs (not shown) located on its outer surface. The infrared LED emits infrared rays as the program progresses during the execution of the program using the controller 160B. The infrared rays emitted from the infrared LEDs can be used to detect the position and orientation (tilt, orientation) of the controller 160B. Further, the controller 160B includes a battery for driving an internal electronic component. The battery may be either a primary battery or a secondary battery, and the shape thereof may be arbitrary such as a button type and a dry battery type. In another embodiment, the controller 160B may be connected to, for example, the USB interface of the computer 200. In this case, the controller 160B may be powered via the USB interface.

FIG. 9 is a block diagram showing a function of the computer 200 for providing a virtual experience to a user via the HMD system 100 according to an embodiment of the present disclosure. The computer 200 executes various processes mainly based on the inputs from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, and the controller 160.

The computer 200 includes a processor 202, a memory 204, and a communication control unit 205. The processor 202 includes a virtual space identification unit 902, an HMD motion detection unit 904, a line-of-sight detection unit 906, a reference line-of-sight determination unit 908, a field of view area determination unit 910, a virtual viewpoint identification unit 912, and a field of view image generation unit 914. The virtual camera control unit 916, the moving direction analysis unit 918, the rotation speed analysis unit 920, the visibility identification unit 922, and the field of view image output unit 924 may be included. Memory 204 may be configured to store a variety of information. In one example, the memory 204 may include virtual space data 926, object data 928, application data 930, visibility degradation degree information 932, and other data 934. The memory 204 also includes various data necessary for the calculation for providing the output information corresponding to the input from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, the controller 160, etc. to the display unit 112 associated with the HMD 110. It may be. The object data 928 may include data about various objects arranged in the virtual space. The display unit 112 may be built in the HMD 110, or may be a display of another device (for example, a smartphone) that can be attached to the HMD 110.

In FIG. 9, the component included in the processor 202 is only one example of expressing the function executed by the processor 202 as a concrete module. The functionality of multiple components may be achieved by a single component. Processor 202 may be configured to perform the functions of all components.

FIG. 10 is a flow chart of a general process for displaying an image of a virtual space in which a user is immersed on the display unit 112.

With reference to FIGS. 9 and 10, a general process of the HMD system 100 for providing an image of the virtual space will be described. The virtual space 400 can be provided by the interaction of the HMD sensor 120, the gaze sensor 140, the computer 200, and the like.

The process starts in step 1002. As an example, the game application included in the application data 930 may be executed by the computer 200. In step 1004, the processor 202 (or the virtual space identification unit 902) generates a spherical virtual space image 402 constituting the virtual space 400 in which the user is immersed by referring to the virtual space data 926 or the like. The position and tilt of the HMD 110 are detected by the HMD sensor 120. The information detected by the HMD sensor 120 is transmitted to the computer 200. In step 1006, the HMD motion detection unit 904 acquires the position information, tilt information, and the like of the HMD 110. In step 1008, the viewing direction is determined based on the acquired position information and inclination information.

When the gaze sensor 140 detects the movement of the eyeballs of the user's left and right eyes, the information is transmitted to the computer 200. In step 1010, the line-of-sight detection unit 906 specifies the directions in which the lines of sight of the right eye and the left eye are directed, and determines the line-of-sight direction N0. In step 1012, the reference line-of-sight determination unit 908 determines the visual field direction or the user's line-of-sight direction N0 determined by the inclination of the HMD 110 as the reference line-of-sight 408. The reference line of sight 408 may also be determined based on the position and tilt of the virtual camera 404 that follows the position and tilt of the HMD 110.

In step 1014, the field of view area determination unit 910 determines the field of view area 410 of the virtual camera 404 in the virtual space 400. As shown in FIG. 4, the field of view area 410 is a part of the virtual space image 402 that constitutes the user's field of view. The field of view 410 is determined based on the reference line of sight 408. The yz cross-sectional view of the view area 410 viewed from the x direction and the xz cross section of the view area 410 seen from the y direction are shown in FIGS. 6 and 7, respectively, which have already been described.

In step 1016, the visual field image generation unit 914 generates a visual field image based on the visual field region 410. The visual field image includes two two-dimensional images, one for the right eye and one for the left eye. By superimposing these two-dimensional images on the display unit 112 (more specifically, the image for the right eye is output to the display unit for the right eye and the image for the left eye is output to the display unit for the left eye), the three-dimensional image is displayed. The virtual space 400 as an image is provided to the user. In step 1018, the visual field image output unit 924 outputs information regarding the visual field image to the display unit 112. The display unit 112 displays the visual field image based on the information of the received visual field image. The process ends in step 1020.

FIG. 11 is a flowchart of Method 1100 according to an embodiment of the present disclosure. FIG. 12 is a flowchart showing a specific example of the process executed in step 1110 of FIG. In one embodiment of the present disclosure, the computer program may cause the processor 202 (or computer 200) to perform each step shown in FIGS. 11 and 12. Another embodiment of the present disclosure may also be implemented as a computer comprising at least a processor and performing method 1100 under the control of the processor.

Hereinafter, embodiments of the present disclosure will be specifically described. Here, as a specific example to which the embodiment of the present disclosure can be applied, it is assumed that a user can immerse himself / herself in a virtual space in which an avatar, other objects, etc. associated with the user are arranged and enjoy the game. .. However, the embodiments of the present disclosure are not necessarily limited to such aspects. It will be apparent to those skilled in the art that embodiments of the present disclosure may take various aspects within the scope of the claims.

FIG. 13 shows how the avatar corresponding to the user moves in the virtual space in response to factors other than the movement of the HMD (control of the controller by the user, movement of the avatar predetermined by the application, etc.) during game play. Is a diagram schematically showing. User 1304 wears the HMD 110 on his head. In this example, user 1304 operates a controller 160B having the configuration shown in FIG. 8B. It will be understood that this is just one example of a controller. Various controllers can be applied to the embodiments of the present disclosure, including controllers of other embodiments that can be worn on the user's body, such as the controller 160A shown in FIG. 8A. In another example, the user 1304 may not be equipped with the controller, and a sensor not attached to the user 1304 may detect the movement of the user 1304. For example, the signal of the camera that captures the user 1304 may be input to the computer 200 as a signal representing the movement of the user 1304.

In the embodiment of the present disclosure, it is premised that VR sickness is unlikely to occur when the virtual camera does not move. Further, when the virtual camera moves due to the movement of the HMD, it is premised that VR sickness is unlikely to occur even if the virtual camera rotates.

An avatar 1306 operated by the user 1304 is arranged in the virtual space 1300. In this example, it is assumed that the avatar 1306 is on a disk-shaped object 1308 that can move freely in the virtual space 1300. While playing the game, the user 1304 can cause the avatar 1306 associated with the user 1304 to perform various operations by operating the controller 160B or the like.

As shown in FIG. 13, the virtual camera 1302 may be arranged at the position of the avatar 1306. The user 1304 can see the image of the virtual space 1300 obtained by the virtual camera 1302 (from the viewpoint of the avatar 1306).

Returning to FIG. 11, the process starts in step 1102. The processor 202 reads and executes the game program included in the application data 930 stored in the memory 204. The virtual space specifying unit 902 specifies the virtual space data for the executed game based on the virtual space data 926, the object data 928, and the like. The data related to the avatar 1306 may be included in the object data 928, the application data 930, and the like. The virtual space data defines a virtual space that includes an avatar 1306 associated with user 1304.

The process proceeds to step 1104, and the virtual camera control unit 916 operates the virtual camera 1302 in the virtual space 1300 based on the operation by the user 1304. In the present embodiment, the user 1304 does not move the head 190, and the virtual camera 1302 is operated by factors other than the movement of the HMD 110, such as the operation of the user 1304 on the controller 160B and the movement of the avatar 1306 predetermined by the application. I assume that.

The process proceeds to step 1106. In the example of FIG. 13, the avatar 1306 on the object 1308 moves on the horizontal plane from the position 13A to the position 13B. As mentioned above, in this example user 1304 does not move the head 190, so the HMD 110 does not move either. Therefore, the virtual camera 1302 moves in the moving direction indicated by the solid arrow 1312, which is different from the orientation 1310 of the virtual camera 1302, without changing the orientation of the virtual camera 1302 indicated by the dotted arrow 1310. The moving direction analysis unit 918 identifies the moving direction 1312 of the virtual camera 1302. Information about the identified travel direction 1312 may be stored in memory 204.

The process proceeds to step 1108. The field of view image generation unit 914 generates a field of view image based on virtual space data, the position and orientation of the virtual camera 1302, and the like. The field of view image is generated, for example, by the process already described in connection with FIG.

The process proceeds to step 1110, and the visibility specifying unit 922 specifies the degree of decrease in visibility to be applied to the visual field image generated in step 1108.

FIG. 12 is a flowchart showing a specific example of the process executed in step 1110.

In step 1202, the moving direction analysis unit 918 decomposes the moving direction 1312 of the virtual camera 1302 identified in step 1106 into a plurality of axial components.

FIG. 14 is a diagram schematically explaining how the moving direction 1312 is decomposed into a plurality of components. The moving direction analysis unit 918 sets the roll axis direction, the pitch axis direction, and the yaw axis direction with reference to the direction 1310 of the virtual camera 1302. In the example of FIG. 13, the virtual camera 1302 moves horizontally from the position 13A to the position 13B. Therefore, the moving direction 1312 has a roll axis direction component and a pitch axis direction component, and does not have a yaw axis direction component. Each axial component indicates the ratio of movement in the axial direction to the movement direction 1312.

The moving direction analysis unit 918 decomposes the moving direction 1312 into a plurality of axial components so that the smaller the angle difference between the direction of each axis and the moving direction 1312, the larger the value of the component in the axial direction. You may. In this example, the moving direction 1312 is tilted clockwise by 30 ° with respect to the positive roll axis direction (hereinafter, also referred to as “roll (+) axis direction”), and the positive pitch axis direction (hereinafter, “pitch (hereinafter,“ pitch ”). It is tilted by 60 ° counterclockwise with respect to (also called "+) axial direction"). Therefore, the moving direction 1312 may be decomposed so that the component in the roll (+) axis direction is larger than the component in the pitch (+) axis direction. For example, the roll (+) axial component is calculated as (90 ° -30 °) / 90 ° = 60/90, and the pitch (+) axial component is (90 ° -60 °) / 90 ° = 30. It may be calculated as / 90. The components in each axial direction may be calculated by other methods. For example, in the above example, the component in the roll (+) axis direction is calculated as cos (30 °) = √3 / 2, and the component in the pitch (+) axis direction is calculated as cos (60 °) = 1/2. May be done.

The process proceeds to step 1204, and the visibility specifying unit 922 specifies the degree of decrease in visibility for each component in the moving direction 1312. The visibility specifying unit 922 includes first reduction degree information that determines the degree of reduction in visibility in one or more directions determined based on the orientation of the virtual camera 1302, and a plurality of axial components obtained in step 1202. May be used to specify the degree of decrease in visibility for each component.

FIG. 15 shows an example of the first decrease degree information. The first decrease degree information may be defined so that the degree of decrease in visibility in the roll axis direction is smaller than the degree of decrease in visibility in the pitch axis direction and the yaw axis direction. The first degree of decrease information also indicates that the degree of decrease in visibility of the virtual camera 1302 in the roll axis direction in the front direction (roll (+) axis direction) is opposite to the front direction in the roll axis direction (roll). It may be specified so as to be smaller than the degree of decrease in visibility with respect to (-) axial direction). When the user 1304 moves the virtual camera 1302 without moving the head 190 as in the present embodiment, the movement in the roll (+) axis direction is the least likely to cause VR sickness, and is in the roll (-) axis direction. This is because it is considered that VR sickness is less likely to occur next to the movement of, and VR sickness is most likely to occur when moving in the pitch axis direction or the yaw axis direction. This is because the movement in the roll (+) axis direction is the easiest to predict the screen change, and the movement in the roll (-) axis direction is the next easiest to predict the screen change, and the movement in the pitch axis direction and the yaw axis direction is the easiest. This is because movement is considered to be the most difficult to predict screen changes. In the exemplary first reduction degree information 1500 shown in FIG. 15, the roll (+) axial direction, the roll (−) axial direction, the pitch (+) axial direction, the pitch (−) axial direction, the yaw (+) axial direction, and The degree of decrease in visibility corresponding to the yaw (−) axial direction is 10, 30, 50, 50, 50 and 50, respectively. It should be understood that this is only an example and that the first degree of decline information can be variously defined according to the embodiment.

The visibility specifying unit 922 moves in each axial direction by multiplying the degree of decrease in visibility corresponding to the axial direction indicated by the first degree of decrease information by the component in the axial direction obtained in step 1202. The degree of decrease in visibility for each component in 1312 may be specified. For example, as described above, in the case of the example of FIG. 14, the moving direction 1312 has a roll (+) axial component having a size of 60/90 and a pitch (+) having a size of 30/90. It is decomposed into axial components. As shown in FIG. 15, the degree of decrease in visibility corresponding to the roll (+) axial direction is 10, and the degree of decrease in visibility corresponding to the pitch (+) axial direction is 50. Therefore, the degree of decrease in visibility of the component in the roll (+) axial direction in the moving direction 1312 is 10 (degree of decrease in visibility corresponding to the roll (+) axial direction) × (60/90) (roll (+)). Axial component) = 20/3 may be calculated. The degree of decrease in visibility of the components in the pitch (+) axial direction in the moving direction 1312 is 50 (degree of decrease in visibility corresponding to the pitch (+) axial direction) × (30/90) (pitch (+) axial direction). (Component of) = 50/3.

Since steps 1206 and 1208 are processes executed when the virtual camera 1302 rotates, they are not executed in the example of FIG. The processing of these steps will be described later in relation to another example.

The process proceeds to step 1210, and the visibility specifying unit 922 specifies the degree of decrease in visibility in the moving direction 1312 based on the degree of decrease in visibility for each component specified in step 1204. The visibility specifying unit 922 may obtain the degree of decrease in visibility in the moving direction 1312 by adding up the degree of decrease in visibility for each component. For example, the degree of decrease in visibility in the moving direction 1312 is 20/3 (degree of decrease in visibility of components in the roll (+) axial direction) + 50/3 (degree of decrease in visibility of components in the pitch (+) axial direction). ) ≈ 23 may be obtained.

Returning to FIG. 11, in step 1112, the field of view image generation unit 914 reduces the visibility of the field of view image generated in step 1108 by using the degree of decrease in visibility specified in step 1110. For example, when the maximum value of the visibility of the visual field image is 100 (%), the visibility of the visual field image in the example of FIG. 13 is reduced to 100 (%) -23 (%) = 77 (%). The process of reducing visibility can be realized by various methods. For example, a blur treatment, a light amount reduction treatment, a contrast reduction treatment, and the like may be performed. The visibility of any region of the field of view image may be reduced. For example, the visibility of the peripheral region in the field of view image may be reduced.

The process proceeds to step 1114, and the visual field image output unit 924 outputs the visual field image with reduced visibility obtained in step 1112 to the display unit 112 associated with the HMD 110. As a result, when the avatar 1306 moves from the position 13A to the position 13B due to a factor other than the movement of the HMD 110, the visibility of the visual field image provided to the user 1304 via the display unit 112 is reduced.

As described above, according to the present embodiment, it is possible to provide a new method for reducing VR sickness without impairing the virtual experience of the user.

FIG. 16 outlines how the avatar corresponding to the user moves in the virtual space in response to factors other than the movement of the HMD 110 (operation of the controller 160B by the user 1304, movement of the avatar 1306 predetermined by the application, etc.). It is a figure which represents. Unlike the example of FIG. 13, the avatar 1306 (and the virtual camera 1302) moves diagonally upward in the virtual space from position 16A to position 16B, as indicated by the solid arrow 1612. Similar to the example of FIG. 13, the moving direction 1612 is different from the direction of the virtual camera 1302 indicated by the dotted arrow 1610.

The process of reducing the visibility of the visual field image in the example of FIG. 16 is basically the same as that of the example of FIG. However, in FIG. 13, the virtual camera 1302 moves two-dimensionally in the horizontal direction, whereas in FIG. 16, the virtual camera 1302 moves three-dimensionally, so that the processing is partially different. In the following, detailed description of the processing common to the examples of FIGS. 13 and 16 will be omitted as appropriate.

17A, 17B, and 17C are diagrams schematically illustrating how the moving direction 1612 is decomposed into a plurality of components by the process of step 1202. The moving direction analysis unit 918 sets the roll axis direction, the pitch axis direction, and the yaw axis direction with reference to the direction 1610 of the virtual camera 1302. In the example of FIG. 16, the virtual camera 1302 moves three-dimensionally from the position 16A to the position 16B. Therefore, the moving direction 1612 has a component in the roll axis direction, a component in the pitch axis direction, and a component in the yaw axis direction. Each axial component indicates the ratio of movement in the axial direction to the movement direction 1612.

In this example, as shown in FIG. 17A, when viewed from the yaw axis direction, the moving direction 1612 is tilted clockwise by 45 ° with respect to the roll (+) axis direction and with respect to the pitch (+) axis direction. It is tilted counterclockwise by 45 °. Further, as shown in FIG. 17B, when viewed from the pitch axis direction, the moving direction 1612 is tilted clockwise by 60 ° with respect to the yaw (+) axis direction and counterclockwise with respect to the roll (+) axis direction. It is tilted around by 30 °. Further, as shown in FIG. 17C, when viewed from the roll axis direction, the moving direction 1612 is tilted clockwise by 60 ° with respect to the yaw (+) axis direction and counterclockwise with respect to the pitch (+) axis direction. It is tilted around by 30 °.

From the relationship shown in FIG. 17A, the component in the roll (+) axial direction of the moving direction 1612 seen from the yaw axis direction may be obtained by the calculation of (90 ° -45 °) / 90 ° = 45/90. The component in the pitch (+) axial direction may be obtained by the calculation of (90 ° −45 °) / 90 ° = 45/90. From the relationship shown in FIG. 17B, the yaw (+) axial component of the moving direction 1612 viewed from the pitch axis direction may be obtained by the calculation of (90 ° -60 °) / 90 ° = 30/90. The component in the roll (+) axial direction may be obtained by the calculation of (90 ° -30 °) / 90 ° = 60/90. From the relationship shown in FIG. 17C, the component in the yaw (+) axial direction of the moving direction 1612 seen from the roll axis direction may be obtained by the calculation of (90 ° -60 °) / 90 ° = 30/90. The component in the pitch (+) axial direction may be obtained by the calculation of (90 ° -30 °) / 90 ° = 60/90. The components in each axial direction may be calculated by other methods.

Similar to the example of FIG. 13, in step 1204, the visibility specifying unit 922 specifies the degree of decrease in visibility for each component in the moving direction 1612. When the first reduction degree information 1500 shown in FIG. 15 is used, the degree of decrease in visibility with respect to the roll (+) axial direction, the pitch (+) axial direction, and the yaw (+) axial direction is 10, 50, and 50, respectively. Is. As an example, the degree of decrease in visibility of the component in the roll (+) axis direction in the moving direction 1612 is (10 (degree of decrease in visibility of the component in the roll (+) axis direction) × (45/90) (yaw axis). Roll (+) axial component viewed from the direction + 10 x (60/90) (roll (+) axial component viewed from the pitch axis direction) / 3 (exemplary weighting value) = 35/9 The degree of decrease in visibility of the component in the pitch (+) axis direction in the moving direction 1612 is (50 (degree of decrease in visibility of the component in the pitch (+) axis direction) ×. (45/90) (Pitch (+) axial component seen from the yaw axis direction + 50 × (60/90) (Pitch (+) axial component seen from the roll axis direction)) / 3 = 175/9 The degree of decrease in visibility of the component in the yaw (+) axis direction in the moving direction 1612 is (50 (degree of decrease in visibility of the component in the yaw (+) axis direction) ×. (30/90) (Yaw (+) axial component seen from the pitch axis direction + 50 × (30/90) (Yaw (+) axial component seen from the roll axis direction)) / 3 = 100/9 It may be obtained by the calculation.

Similar to the example of FIG. 13, in step 1210, the visibility specifying unit 922 may add up the degree of decrease in visibility of each component. For example, the degree of decrease in visibility in the moving direction 1612 is 35/9 (degree of decrease in visibility of components in the roll (+) axial direction) + 175/9 (degree of decrease in visibility of components in the pitch (+) axial direction). ) + 100/9 (degree of decrease in visibility of components in the yaw (+) axial direction) ≈34. Therefore, in the case of this example, in step 1112, the visual field image generation unit 914 reduces the visibility of the visual field image from the maximum value of 100 (%) to 100 (%)-34 (%) = 66 (%). You may. In step 1114, the visual field image output unit 924 outputs the visual field image with reduced visibility to the display unit 112 associated with the HMD 110.

FIG. 18 is a diagram schematically showing how an avatar corresponding to a user moves in a virtual space according to a factor other than the movement of the HMD. In this example, the avatar 1306 (and virtual camera 1302) moves horizontally in virtual space from position 18A to position 18B, as indicated by solid arrow 1812, and is further indicated by arrows 1814 and 1815. Rotate like. The moving direction 1812 is the same as the moving direction 1312 in FIG. 13, and is different from the direction of the virtual camera 1302 indicated by the dotted arrow 1810. It is assumed that positions 18A and 18B are the same as positions 13A and 13B, respectively. Therefore, the degree of decrease in visibility of the component in the roll (+) axis direction and the degree of decrease in visibility of the component in the pitch (+) axis direction in the moving direction 1812, which are obtained by the processes of steps 1202 and 1204, are 20/3, respectively. And 50/3.

The virtual camera 1302 not only moves from position 18A to position 18B, but also rotates as indicated by arrows 1814 and 1816. In the present embodiment, the rotation is caused by factors other than the movement of the HMD 110 (operation of the user 1304 on the controller 160B, movement of the avatar 1306 predetermined by the application, and the like).

After step 1204, the process proceeds to step 1206, where the rotation speed analysis unit 920 identifies the rotation speed of the virtual camera 1302. The specified rotation speed may be stored in memory 204.

The process proceeds to step 1208, and the rotation speed analysis unit 920 identifies the degree of decrease in visibility corresponding to the rotation speed obtained in step 1206. The rotation speed analysis unit 920 may specify the degree of decrease in visibility by using the second degree of decrease information that determines the degree of decrease in visibility with respect to the rotation speed.

FIG. 19 shows an example of the second decrease degree information. In FIG. 19, the second decrease degree information 1900 is represented as a graph showing the relationship between the rotation speed (rotation / second, rps) and the visibility decrease degree. The second decrease degree information 1900 may be defined so that the degree of decrease in visibility increases as the rotation speed increases until the rotation speed reaches a predetermined speed. Further, the rate of increase in the degree of decrease in visibility may be larger as the rotation speed is lower, and may be slower as the rotation speed is higher. The relationship shown in FIG. 19 may be stored in the memory 204 in advance.

FIG. 20 shows an example of the second decrease degree information read from the graph of FIG. 19 and quantified. In this example, the rotation speeds are 0.05 rps, 0.1 rps, 0.2 rps, 0.3 rps, 0.4 rps, 0.5 rps, 0.6 rps, 0.7 rps, 0.8 rps, 0.9 rps and 1.0 rps. Corresponding visibility reductions are 15, 25, 37.5, 44, 47, 48.5, 49.5, 49.8, 50, 50 and 50, respectively. It should be understood that this is only an example and that the second reduction degree information 2000 can be variously defined according to the embodiment. Assuming that the rotation speed in the example of FIG. 18 is 0.1 rps, in step 1208, the degree of decrease in visibility corresponding to the rotation speed is specified to be 25.

The process proceeds to step 1210. The visibility specifying unit 922 finally determines the degree of decrease in visibility for each component in the moving direction 1812 specified in step 1204 and the degree of decrease in visibility corresponding to the rotation speed specified in step 1208. Identify the degree of loss of visibility. In this example, the degree of decrease in visibility with respect to the moving direction 1812 obtained based on the first degree of decrease information 1500 is 20/3 + 50/3≈23. The degree of decrease in visibility with respect to the rotation speed obtained based on the second degree of decrease information 2000 is 25. The final degree of decrease in visibility may be obtained by adding up the above two degrees of decrease in visibility and calculating 23 + 25 = 48. In this case, in step 1112, the visual field image generation unit 914 reduces the visibility of the visual field image from the maximum value of 100 (%) to 100 (%)-48 (%) = 52 (%). In step 1114, the visual field image output unit 924 outputs the visual field image with reduced visibility to the display unit 112 associated with the HMD 110.

As described above, according to the present embodiment, when the avatar corresponding to the user moves and rotates in the virtual space according to a factor other than the movement of the HMD, the degree of deterioration of visibility due to the movement of the virtual camera and the virtual camera The degree of decrease in visibility of the visual field image is specified based on both the degree of decrease in visibility due to the rotation of the camera.

FIG. 21 shows an example of a field image with reduced visibility generated according to an embodiment of the present disclosure. 21 (A), (B) and (C) correspond to the examples of FIGS. 13, 16 and 18, respectively. As described above, the degree of decrease in visibility in FIGS. 13, 16 and 18, is 23, 34 and 48, respectively. Therefore, the visibility of the visual field image of FIG. 21 (B) is lower than the visibility of the visual field image of FIG. 21 (A), and the visibility of the visual field image of FIG. 21 (C) is the visibility of the visual field image of FIG. 21 (B). Even lower than sex. Any method may be adopted as the method for reducing the visibility, and examples thereof include blur processing, light amount reduction processing, resolution reduction processing, and contrast reduction processing, but are limited to these methods. is not it. In the examples shown in FIGS. 21A, 21B, and 21C, the case where the visibility of the entire visual field image is reduced has been described as an example, but the region where the visibility is reduced is limited to this. It's not a thing. For example, the visibility of a predetermined area including an element on the field of view image gazed at by the reference line of sight 408 of the virtual camera 404 is not lowered, and the visibility is lowered with respect to a region outside the predetermined area. May be good. The predetermined region includes, but is not limited to, a rectangular region or a circular region centered on an element on the field of view image that is gazed at by the reference line of sight 408 of the virtual camera 404.

As described above, according to the embodiment of the present disclosure, even when the virtual camera is moved by a factor other than the movement of the HMD, such as the operation of the controller by the user and the movement of the avatar predetermined by the application, the virtual experience It is possible to provide a novel method for reducing VR sickness without impairing the condition.

It will be apparent to those skilled in the art that embodiments of the present disclosure can be implemented as a processor 202 (or computer 200), method 1100, computer program causing processor 202 (or computer 200) to execute method 1100, and the like. Let's do it.

In the above-described embodiment, the virtual space (VR space) in which the user is immersed by the HMD device has been described as an example, but a transmissive HMD device may be adopted as the HMD device. In this case, augmented reality (AR) space or mixed reality is output by outputting a view image obtained by synthesizing a part of the image constituting the virtual space in the real space visually recognized by the user via the transmissive HMD device. (MR: Mixed Reality) A virtual experience in space may be provided to the user. In this case, instead of the hand object, an action on the target object in the virtual space may be generated based on the movement of the user's hand. Specifically, the processor may specify the coordinate information of the position of the user's hand in the real space, and may define the position of the target object in the virtual space in relation to the coordinate information in the real space. As a result, the processor can grasp the positional relationship between the user's hand in the real space and the target object in the virtual space, and can execute a process corresponding to collision control or the like between the user's hand and the target object. As a result, it becomes possible to give an action to the target object based on the movement of the user's hand.

Although embodiments of the present disclosure have been described, it should be understood that these are merely examples and do not limit the scope of the present disclosure. It should be understood that the embodiments can be changed, added, improved, etc. as appropriate without departing from the spirit and scope of the present disclosure. The scope of the present disclosure should not be limited by any of the embodiments described above, but should be defined by the claims and their equivalents.

100 ... HMD system, 112 ... Display, 114 ... Sensor, 120 ... HMD sensor, 130 ... Motion sensor, 140 ... Gaze sensor, 150 ... Server, 160A, 160B ... Controller, 190 ... Head, 192 ... Network, 200 ... Computer, 202 ... Processor, 204 ... Memory, 205 ... Communication control unit, 206 ... Storage, 208 ... Input / output interface, 210 ... Communication interface, 212 ... Bus, 400 ... Virtual space, 402 ... Virtual space image, 404 ... Virtual camera , 406 ... center, 408 ... reference line of sight, 410 ... view area, 800 ... right controller, 802 ... grip, 804 ... frame, 806 ... top surface, 808, 810, 814, 816 ... button, 812 ... infrared LED, 818 ... Analog stick, 820, 822 ... button, 824R, 824L ... analog stick, 902 ... virtual space identification unit, 904 ... HMD motion detection unit, 906 ... line-of-sight detection unit, 908 ... reference line-of-sight determination unit, 910 ... view area determination unit, 912 ... Virtual viewpoint identification unit, 914 ... Visibility image generation unit, 916 ... Virtual camera control unit, 918 ... Movement direction analysis unit, 920 ... Rotation speed analysis unit, 922 ... Visibility identification unit, 924 ... Visibility image output unit, 926 ... Virtual space data, 928 ... Object data, 930 ... Application data, 932 ... Visibility reduction information, 934 ... Other data, 1300 ... Virtual space, 1302 ... Virtual camera, 1304 ... User, 1306 ... Avatar, 1308 ... Object 1500 ... 1st degree of decrease information, 1900, 2000 ... 2nd degree of decrease information

Claims (8)

A method performed by a computer to provide a virtual space to a user via a headmount device.
The step of operating the virtual camera arranged in the virtual space and
The step of specifying the moving direction of the virtual camera and
A step of generating a field of view image representing the virtual space within the field of view of the virtual camera, and
A step of specifying the degree of decrease in visibility in the moving direction based on the first degree of decrease information that defines the degree of decrease in visibility in one or more directions determined based on the orientation of the virtual camera.
A step of reducing the visibility of the visual field image based on the specified degree of deterioration of the visibility, and a step of reducing the visibility of the visual field image.
A step of outputting the field-of-view image with reduced visibility to the display unit of the head-mounted device, and
Only including,
The first degree of decrease information determines the degree of decrease in visibility in a plurality of axial directions determined based on the orientation of the virtual camera.
The step of specifying the degree of decrease in visibility is
A step of decomposing the moving direction into the plurality of axial components, and
A step of specifying the degree of decrease in visibility for each component in the moving direction by using the first degree of decrease information and the plurality of axial components.
A step of specifying the degree of decrease in visibility in the moving direction based on the degree of decrease in visibility for each of the specified components, and a step of specifying the degree of decrease in visibility in the moving direction.
Including, way. The axial component indicates the ratio of movement in the axial direction to the movement direction.
In the disassembling step, the moving direction is decomposed into the plurality of axial components so that the smaller the angle difference between the axial direction and the moving direction, the larger the value of the component in the axial direction.
In the step of specifying the degree of decrease in visibility for each component in the moving direction, the component in the axial direction is added to the degree of decrease in visibility in the axial direction indicated by the first degree of decrease information for each axial direction. By multiplying, the degree of decrease in visibility for each component in the moving direction is specified.
The first aspect of claim 1, wherein in the step of specifying the degree of decrease in visibility in the moving direction, the degree of decrease in visibility for each of the specified components is added up to specify the degree of decrease in visibility in the moving direction. the method of. The plurality of axial directions include a roll axial direction, a pitch axial direction, and a yaw axial direction.
The first reduction degree information according to claim 1 or 2 , wherein the degree of decrease in visibility in the roll axis direction is set to be smaller than the degree of decrease in visibility in the pitch axis direction and the yaw axis direction. Method. The first degree of decrease information indicates that the degree of decrease in visibility of the virtual camera in the roll axis direction in the front direction is higher than the degree of decrease in visibility in the direction opposite to the front direction in the roll axis direction. The method according to claim 3 , which is defined as small. Further including a step of specifying the rotation speed of the virtual camera,
In the step of specifying the degree of decrease in visibility, the moving direction and the specification are based on the first degree of decrease information and the second degree of decrease information that determines the degree of decrease in visibility with respect to the rotation speed of the virtual camera. The method according to any one of claims 1 to 4 , wherein the degree of decrease in visibility at the rotational speed is specified. The method according to claim 5 , wherein the second decrease degree information is defined so that the degree of decrease in visibility increases as the rotation speed increases until the rotation speed reaches a predetermined speed. A program for causing a processor to execute the method according to any one of claims 1 to 6. It ’s a computer,
By the control provided by the processor of the computer to provide the virtual space to the user via the head mount device.
The step of operating the virtual camera arranged in the virtual space and
The step of specifying the moving direction of the virtual camera and
A step of generating a field of view image representing the virtual space within the field of view of the virtual camera, and
A step of specifying the degree of decrease in visibility in the moving direction based on the first degree of decrease information that defines the degree of decrease in visibility in one or more directions determined based on the orientation of the virtual camera.
A step of reducing the visibility of the visual field image based on the specified degree of deterioration of the visibility, and a step of reducing the visibility of the visual field image.
A step of outputting the field-of-view image with reduced visibility to the display unit of the head-mounted device, and
Is executed ,
The first degree of decrease information determines the degree of decrease in visibility in a plurality of axial directions determined based on the orientation of the virtual camera.
The step of specifying the degree of decrease in visibility is
A step of decomposing the moving direction into the plurality of axial components, and
A step of specifying the degree of decrease in visibility for each component in the moving direction by using the first degree of decrease information and the plurality of axial components.
A step of specifying the degree of decrease in visibility in the moving direction based on the degree of decrease in visibility for each of the specified components, and a step of specifying the degree of decrease in visibility in the moving direction.
Including computers.

Images