JP6907138B2 - A method performed by a computer to communicate over virtual space, a program to cause the computer to perform the method, and a computer device. - Google Patents

Description

The present disclosure relates to a technique for providing a virtual space, and more specifically to the provision of communication via the virtual space.

A technique for providing a virtual space using a head mount device (HMD) has become widespread. For example, Japanese Patent Application Laid-Open No. 2009-223656 (Patent Document 1) discloses a technique for communicating between a plurality of users by the operation of an avatar object presented in a virtual space. This technology presents the avatar object corresponding to each user in the virtual space by communicating the computers of each of the multiple users on the network, and the avatar object in the virtual space is made to correspond to the user's movement in the real space. It is something that moves. Using such a technique, a user can play a game common to another user's avatar object (hereinafter, also referred to as another avatar object) or talk with another avatar object while moving an avatar object existing in the virtual space. (So-called chat) can be done.

JP-A-2009-223656

In the virtual space as described above, when the user speaks, another user operates in the real space in response to the utterance, and another avatar object operates in the virtual space in response to the operation. Such a series of flows is realized by communication between computers owned by each user. Since the time from the user's utterance to the operation of another avatar object depends at least on the communication time between computers, the longer the communication time between computers, the more interaction between users in the virtual space. Is hard to plan. It should be noted that such a problem occurs not only when another user responds to the user's utterance but also when another user responds to the user's movement of a part of the body. obtain.

The present disclosure has been devised in view of such circumstances, and an object in a certain aspect is to facilitate interaction between a plurality of users in a virtual space.

According to certain embodiments, a method performed by a computer for communicating over virtual space is provided. This method involves defining a virtual space, presenting one or more avatar objects to the virtual space, each corresponding to one or more other users with whom the computer user can communicate, and speaking by the computer user. The step of accepting the input of the voice data corresponding to, the step of outputting the voice data to the computer of another user, and the avatar object according to the response data output from the computer of the other user in response to the input of the voice data. Includes a step of moving the avatar object and a step of changing the line of sight of the avatar object between the speech and the movement of the avatar object.

The above and other objectives, features, aspects and advantages of the invention will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

It is a figure which shows the outline of the structure of the HMD system according to a certain embodiment. It is a block diagram which shows an example of the hardware composition of the computer which follows one aspect. It is a figure which conceptually represents the uvw field-of-view coordinate system set in the HMD according to a certain embodiment. It is a figure that conceptually represents one aspect of expressing a virtual space according to a certain embodiment. It is a figure which showed the head of the user who wears an HMD according to a certain embodiment from the top. It is a figure which shows the YZ cross section which looked at the visual field area from the X direction in the virtual space. It is a figure which shows the XZ cross section which looked at the field of view area from the Y direction in a virtual space. It is a figure which shows the schematic structure of the controller according to a certain embodiment. It is a block diagram which shows the computer according to a certain embodiment as a module structure. It is a figure that conceptually represents one aspect of expressing a virtual space presented by each of a plurality of computers. FIG. 5 is a sequence chart representing a portion of the processing performed in an HMD system according to an embodiment. FIG. 5 is a sequence chart representing some of the detailed processing performed in an HMD system according to an embodiment. It is a sequence chart which shows a part of communication between a plurality of computers. It is a flowchart which shows the detailed processing which a processor of a computer performs in one aspect of a certain embodiment. It is a figure which shows an example of the presentation of the object in the virtual space according to a certain embodiment. It is a figure which shows an example of the change of the object presented in the virtual space according to a certain embodiment. It is a figure which shows an example of the change of the object presented in the virtual space according to a certain embodiment. It is a figure which shows an example of the presentation of the object in the virtual space which follows the other aspect of an embodiment. It is a flowchart which shows the detailed processing which a processor of a computer performs in the other aspect of one Embodiment. It is a figure which shows an example of the change of the object presented in the virtual space according to the other aspect of one embodiment. It is a flowchart which shows the detailed processing which a processor of a computer performs in the other aspect of one Embodiment. It is a flowchart which shows the detailed processing which a processor of a computer performs in the other aspect of one Embodiment.

Hereinafter, embodiments will be described with reference to the drawings. In the following description, the same parts and components are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of these will not be repeated.

[HMD system configuration]
The configuration of the HMD system 100 will be described with reference to FIG. FIG. 1 is a diagram showing an outline of the configuration of the HMD system 100 according to an embodiment. In some aspects, the HMD system 100 is provided as a home system or a business system.

The HMD system 100 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a monitor 112, a gaze sensor 140, a speaker 115, and a microphone 119. The controller 160 may include a motion sensor 130.

In some aspects, the computer 200 can connect to the Internet or other network 19 and communicate with the server 150 or other computer connected to the network 19. In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120.

The HMD 110 may be worn on the head of the user 190 and provide the virtual space 2 to the user 190 during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the monitor 112, respectively. When each eye of the user 190 visually recognizes the respective image, the user 190 can recognize the image as a three-dimensional image based on the parallax of both eyes.

The monitor 112 is realized as, for example, a non-transparent display device. In one aspect, the monitor 112 is located in the body of the HMD 110 so that it is located in front of both eyes of the user 190. Therefore, the user 190 can immerse himself / herself in the virtual space 2 when he / she visually recognizes the three-dimensional image displayed on the monitor 112. In certain embodiments, the virtual space 2 includes, for example, a background, an object that the user 190 can manipulate, and an image of a menu that the user 190 can select. If a plurality of computers 200 pass signals based on the actions of each user so that a plurality of users can experience a virtual experience in one virtual space 2, an avatar object corresponding to each user is presented in the virtual space 2. Will be done.

The object is a virtual object existing in the virtual space 2. In some aspects, objects include user-corresponding avatar objects, virtual accessories and virtual clothing worn by avatar objects, virtual buildings, virtual roads, virtual vehicles, and the like. Further, the avatar object is a character symbolizing the user 190 in the virtual space 2, and includes, for example, a human type, an animal type, a robot type, and the like. Objects come in many shapes. The user 190 may present a favorite object from the predetermined objects to the virtual space 2, or may present the object created by the user 190 to the virtual space 2.

In certain embodiments, the monitor 112 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor included in a so-called smartphone or other information display terminal.

In some aspects, the monitor 112 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 112 may be configured to display the image for the right eye and the image for the left eye as a unit. In this case, the monitor 112 includes a high speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

The gaze sensor 140 detects the direction (line-of-sight direction) in which the eyes of the right eye and the left eye of the user 190 are directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In certain aspects, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the angle of rotation of each eyeball by receiving the reflected light from the cornea and the iris with respect to the irradiation light. .. The gaze sensor 140 can detect the line-of-sight direction of the user 190 based on each detected rotation angle.

The speaker 115 outputs the voice (utterance) corresponding to the voice data received from the computer 200 to the outside. The microphone 119 outputs voice data corresponding to the utterance of the user 190 to the computer 200. The user 190 can use the microphone 119 to speak to another user, while the speaker 115 can hear the voice (utterance) of the other user.

More specifically, when the user 190 speaks into the microphone 119, the voice data corresponding to the utterance of the user 190 is input to the computer 200. The computer 200 outputs the voice data to the server 150 via the network 19. The server 150 outputs the voice data received from the computer 200 to another computer 200 via the network 19. The other computer 200 outputs the voice data received from the server 150 to the speaker 115 of the HMD 110 worn by another user. As a result, other users can hear the voice of the user 190 through the speaker 115 of the HMD 110. Similarly, utterances from other users are output from the speaker 115 of the HMD 110 worn by the user 190.

The computer 200 displays on the monitor 112 an image that moves another avatar object corresponding to the other user according to the voice data received from the computer 200 of the other user. For example, in a certain situation, the computer 200 displays an image that moves the mouth of another avatar object on the monitor 112, so that the virtual space 2 is displayed as if the avatar objects are talking to each other in the virtual space 2. Express. By transmitting and receiving voice data between the plurality of computers 200 in this way, a conversation (chat) between a plurality of users is realized in one virtual space 2.

The HMD sensor 120 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. The HMD sensor 120 uses this function to detect the position and tilt of the HMD 110 in the real space.

In another aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and tilt of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include the sensor 114 as the position detector instead of the HMD sensor 120. The HMD 110 may use the sensor 114 to detect the position and tilt of the HMD 110 itself. For example, if the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an accelerometer, or a gyro sensor, the HMD 110 uses any of these sensors instead of the HMD sensor 120 to determine its position and tilt. Can be detected. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in real space over time. The HMD 110 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle.

Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance. Further, the field-of-view image may include a configuration for presenting the real space as a part of the image constituting the virtual space 2. For example, an image taken by a camera mounted on the HMD 110 may be superimposed on a part of the field of view image and displayed, or by setting a high transmittance of a part of the transmissive display device, the field of view image can be displayed. The real space may be visible from a part.

The server 150 may send the program to the computer 200. In another aspect, the server 150 may communicate with another computer 200 to provide virtual reality to the HMD 110 used by other users. For example, in an amusement facility, when a plurality of users play a participatory game, each computer 200 communicates a signal based on the operation of each user with another computer 200, and the plurality of users are common in the same virtual space 2. Allows you to enjoy the game. Further, as described above, the plurality of computers 200 transmit and receive signals based on the actions of each user, so that a plurality of users can enjoy the conversation in one virtual space 2.

The controller 160 receives an instruction input from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be grippable by the user 190. In another aspect, the controller 160 is configured to be wearable on a body or part of clothing of the user 190. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal sent from the computer 200. In another aspect, the controller 160 accepts operations given by the user 190 to control the position and movement of objects placed in the space that provides virtual reality.

The motion sensor 130 is attached to the hand of the user 190 in a certain aspect to detect the movement of the hand of the user 190. For example, the motion sensor 130 detects the rotation speed, the number of rotations, and the like of the hand. Data representing the detection result of the hand movement of the user 190 obtained by the motion sensor 130 (hereinafter, also referred to as detection data) is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove-shaped controller 160. In certain embodiments, for safety in real space, it is desirable that the controller 160 be attached to something that does not easily fly by being attached to the user 190's hand, such as a glove type. In another aspect, a sensor not attached to the user 190 may detect the movement of the user 190's hand. For example, the signal of the camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190. The motion sensor 130 and the computer 200 are connected to each other by wire or wirelessly. In the case of wireless communication, the communication mode is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication method is used.

In another aspect, the HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 2.

In still other aspects, the HMD system 100 may include a communication circuit for connecting to the Internet or a telephone function for connecting to a telephone line.

[Hardware configuration]
The computer 200 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of the computer 200 according to one aspect. The computer 200 includes a processor 10, a memory 11, a storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on the signal given to the computer 200 or when a predetermined condition is satisfied. In a certain aspect, the processor 10 is realized as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array) or other device.

The memory 11 temporarily stores programs and data. The program is loaded from storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In a certain aspect, the memory 11 is realized as a RAM (Random Access Memory) or other volatile memory.

The storage 12 permanently holds programs and data. The storage 12 is realized as, for example, a ROM (Read-Only Memory), a hard disk device, a flash memory, or other non-volatile storage device. The program stored in the storage 12 includes a program for providing the virtual space 2 in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data and objects for defining the virtual space 2.

In another aspect, the storage 12 may be realized as a detachable storage device such as a memory card. In still other aspects, configurations may be used that use programs and data stored in an external storage device instead of the storage 12 built into the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used such as an amusement facility, it is possible to update programs and data at once.

In certain embodiments, the input / output interface 13 communicates signals with the HMD 110, HMD sensor 120, or motion sensor 130. In a certain aspect, the input / output interface 13 is realized by using a USB (Universal Serial Bus) interface, a DVI (Digital Visual Interface), an HDMI (registered trademark) (High-Definition Multimedia Interface), and other terminals. The input / output interface 13 is not limited to the above.

In certain embodiments, the input / output interface 13 may further communicate with the controller 160. For example, the input / output interface 13 receives an input of a signal output from the motion sensor 130. In another aspect, the input / output interface 13 sends an instruction output from the processor 10 to the controller 160. The command instructs the controller 160 to vibrate, output voice, emit light, and the like. Upon receiving the instruction, the controller 160 executes either vibration, audio output, or light emission in response to the instruction.

The communication interface 14 is connected to the network 19 and communicates with another computer (for example, a server 150, another user's computer 200, etc.) connected to the network 19. In a certain aspect, the communication interface 14 is realized as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication) or other wireless communication interface. Will be done. The communication interface 14 is not limited to the above.

In one aspect, the processor 10 accesses the storage 12, loads one or more programs stored in the storage 12 into the memory 11, and executes a series of instructions contained in the program. The one or more programs may include an operating system for the computer 200, an application program for providing the virtual space 2, game software that can be executed in the virtual space 2 using the controller 160, and the like. The processor 10 sends a signal for providing the virtual space 2 to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 112 based on the signal.

In the example shown in FIG. 2, the computer 200 is configured to be provided outside the HMD 110, but in other aspects, the computer 200 may be built in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including a monitor 112 may function as a computer 200.

Further, the computer 200 may have a configuration commonly used for a plurality of HMD 110s. According to such a configuration, for example, the same virtual space 2 can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space 2.

In one embodiment, the HMD system 100 has a preset global coordinate system. The global coordinate system has three reference directions (axes) that are parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-back direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction of real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the real space.

In some aspects, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect a temporal change in the position and inclination of the HMD 110 by using each value detected over time.

The global coordinate system is parallel to the coordinate system in real space. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination of the HMD 110 around three axes in the global coordinate system. The HMD sensor 120 sets the uvw field of view coordinate system to the HMD 110 based on the tilt of the HMD 110 in the global coordinate system. The uvw field-of-view coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 sees an object in the virtual space 2.

[Uvw field coordinate system]
The uvw field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and tilt of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 10 sets the uvw field coordinate system to the HMD 110 based on the detected values.

As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 190 wearing the HMD 110. More specifically, the HMD 110 defines the global coordinate system in the horizontal, vertical, and front-back directions (x-axis, y-axis, z-axis) by the inclination of the HMD 110 around each axis in the global coordinate system. The three directions newly obtained by tilting each around the axis are set as the pitch direction (u-axis), the yaw direction (v-axis), and the roll direction (w-axis) of the uvw field coordinate system in the HMD 110.

In a certain aspect, when the user 190 wearing the HMD 110 is upright and visually recognizing the front, the processor 10 sets the uvw field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch direction (u-axis) and yaw direction (v-axis) of the uvw field coordinate system in the HMD 110. , And the roll direction (w-axis).

After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (change amount of the inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110, respectively. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

The HMD sensor 120 sets the uvw field coordinate system in the HMD 110 after the HMD 110 moves, based on the detected tilt angle of the HMD 110, in the HMD 110. The relationship between the HMD 110 and the uvw field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

In one aspect, the HMD sensor 120 is based on the infrared light intensity obtained based on the output from the infrared sensor and the relative positional relationship between the points (eg, the distance between the points). The position of the above in the real space may be specified as a relative position with respect to the HMD sensor 120. Further, the processor 10 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

[Virtual space]
The virtual space 2 will be further described with reference to FIG. FIG. 4 is a diagram conceptually representing one aspect of expressing the virtual space 2 according to a certain embodiment. The virtual space 2 has an all-sky spherical structure that covers the entire center 21 in the 360-degree direction. In FIG. 4, the celestial sphere in the upper half of the virtual space 2 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 2. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system defined in the virtual space 2. The computer 200 associates each partial image constituting the contents (still image, moving image, etc.) expandable in the virtual space 2 with each corresponding mesh in the virtual space 2, and the virtual space image visible to the user 190. The virtual space 2 in which 22 is deployed is provided to the user 190.

In a certain aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the XYZ coordinate system are defined as the X-axis, the Y-axis, and the Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the global coordinate system, and the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the global coordinate system. The Z-axis (front-back direction) is parallel to the z-axis of the global coordinate system.

When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 1 is arranged at the center 21 of the virtual space 2. The virtual camera 1 moves in the virtual space 2 in the same manner in conjunction with the movement of the HMD 110 in the real space. As a result, changes in the position and orientation of the HMD 110 in the real space are similarly reproduced in the virtual space 2.

As in the case of the HMD 110, the virtual camera 1 is defined with an uvw field-of-view coordinate system. The uvw field-of-view coordinate system of the virtual camera in the virtual space 2 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 1 also changes accordingly. Further, the virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space.

Since the orientation of the virtual camera 1 is determined according to the position and inclination of the virtual camera 1, the reference line of sight (reference line of sight 5) when the user 190 visually recognizes the virtual space image 22 depends on the orientation of the virtual camera 1. Is decided. The processor 10 of the computer 200 defines the field of view 23 in the virtual space 2 based on the reference line of sight 5. The field of view 23 corresponds to the field of view of the user 190 wearing the HMD 110 in the virtual space 2.

The line-of-sight direction of the user 190 detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user 190 visually recognizes an object. The uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 112. Further, the uvw field-of-view coordinate system of the virtual camera 1 is linked to the uvw field-of-view coordinate system of the HMD 110. Therefore, the HMD system 100 according to a certain aspect can consider the line-of-sight direction of the user 190 detected by the gaze sensor 140 as the line-of-sight direction of the user 190 in the uvw field-of-view coordinate system of the virtual camera 1.

[User's line of sight]
The determination of the line-of-sight direction of the user 190 will be described with reference to FIG. FIG. 5 is a top view of the head of the user 190 wearing the HMD 110 according to an embodiment.

In one aspect, the gaze sensor 140 detects each line of sight of the user 190's right and left eyes. In one aspect, when the user 190 is looking closer, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 identifies the gaze point N1 which is the intersection of the lines of sight R1 and L1 based on the detected values. On the other hand, when the computer 200 receives the detected values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 specifies the line-of-sight direction N0 of the user 190 based on the position of the specified gazing point N1. For example, the computer 200 detects the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the extending direction of the straight line passing through the gazing point N1 as the line-of-sight direction N0. The line-of-sight direction N0 is the direction in which the user 190 actually directs the line of sight with both eyes. Further, the line-of-sight direction N0 corresponds to the direction in which the user 190 actually directs the line of sight with respect to the field of view area 23.

[Visibility area]
The field of view 23 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing a YZ cross section of the field of view 23 as viewed from the X direction in the virtual space 2. FIG. 7 is a diagram showing an XZ cross section of the field of view 23 as viewed from the Y direction in the virtual space 2.

As shown in FIG. 6, the field of view region 23 in the YZ cross section includes the region 24. The area 24 is defined by the reference line of sight 5 of the virtual camera 1 and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α centered on the reference line of sight 5 in the virtual space 2 as a region 24.

As shown in FIG. 7, the field of view region 23 in the XZ cross section includes the region 25. The region 25 is defined by the reference line of sight 5 and the XZ cross section of the virtual space 2. The processor 10 defines a range including the azimuth angle β centered on the reference line of sight 5 in the virtual space 2 as a region 25.

In one aspect, the HMD system 100 provides the user 190 with the virtual space 2 by displaying a field of view image on the monitor 112 based on a signal from the computer 200. The field-of-view image corresponds to a portion of the virtual space image 22 that is superimposed on the field-of-view area 23. When the user 190 moves the HMD 110 attached to the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the field of view 23 in the virtual space 2 changes. As a result, the visual field image displayed on the monitor 112 is updated to an image superimposed on the visual field region 23 in the direction in which the user 190 faces in the virtual space 2 among the virtual space images 22. The user 190 can visually recognize a desired direction in the virtual space 2.

While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the HMD system 100 can give the user 190 a high sense of immersion in the virtual space 2.

In a certain aspect, the processor 10 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 10 identifies an image area (that is, a view area 23 in the virtual space 2) projected on the monitor 112 of the HMD 110 based on the position and orientation of the virtual camera 1 in the virtual space 2.

According to certain embodiments, the virtual camera 1 preferably includes two virtual cameras, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Further, it is preferable that an appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In the present embodiment, the virtual camera 1 includes two virtual cameras, and the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD 110. The technical idea according to the present disclosure will be illustrated as being configured as such.

[controller]
An example of the controller 160 will be described with reference to FIG. FIG. 8 is a diagram showing a schematic configuration of a controller 160 according to an embodiment.

As shown in FIG. 8A, in some aspects the controller 160 may include a right controller 800 and a left controller (not shown). The right controller 800 is operated by the right hand of the user 190. The left controller is operated by the left hand of the user 190. In one aspect, the right controller 800 and the left controller are symmetrically configured as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller 800 and the left hand holding the left controller respectively. In another aspect, the controller 160 may be an integrated controller that accepts operations of both hands. Hereinafter, the right controller 800 will be described.

The right controller 800 includes a grip 30, a frame 31, and a top surface 32. The grip 30 is configured to be gripped by the right hand of the user 190. For example, the grip 30 may be held by the palm of the user 190's right hand and three fingers (middle finger, ring finger, little finger).

The grip 30 includes buttons 33 and 34 and a motion sensor 130. The button 33 is arranged on the side surface of the grip 30 and accepts an operation by the middle finger of the right hand. The button 34 is arranged on the front surface of the grip 30 and accepts an operation by the index finger of the right hand. In one aspect, the buttons 33, 34 are configured as trigger-type buttons. The motion sensor 130 is built in the housing of the grip 30. If the operation of the user 190 can be detected from around the user 190 by a camera or other device, the grip 30 does not have to include the motion sensor 130.

The frame 31 includes a plurality of infrared LEDs 35 arranged along its circumferential direction. The infrared LED 35 emits infrared rays as the program progresses while the program using the controller 160 is being executed. The infrared rays emitted from the infrared LED 35 can be used to detect each position and orientation (tilt, orientation) of the right controller 800 and the left controller. In the example shown in FIG. 8, infrared LEDs 35 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. An array of one column or three or more columns may be used.

The top surface 32 includes buttons 36 and 37 and an analog stick 38. The buttons 36 and 37 are configured as push-type buttons. Buttons 36 and 37 accept operations by the thumb of the user 190's right hand. The analog stick 38 accepts an operation 360 degrees in any direction from the initial position (neutral position) in a certain aspect. The operation includes, for example, an operation for moving an object arranged in the virtual space 2.

In one aspect, the right controller 800 and the left controller include a battery for driving the infrared LED 35 and other components. Batteries include, but are not limited to, rechargeable, button type, dry cell type and the like. In another aspect, the right controller 800 and the left controller may be connected to, for example, the USB interface of the computer 200. In this case, the right controller 800 and the left controller do not require batteries.

FIG. 8B shows an example of a hand object 810 arranged in the virtual space 2 corresponding to the right hand of the user 190 holding the right controller 800. For example, the yaw, roll, and pitch directions are defined for the hand object 810 corresponding to the right hand of the user 190. For example, when the input operation is performed on the button 34 of the right controller 800, the index finger of the hand object 810 is grasped, and when the input operation is not performed on the button 34, the segmentation diagram ( As shown in B), the index finger of the hand object 810 can be extended. For example, in the hand object 810, when the thumb and the index finger are extended, the direction in which the thumb is extended is the yaw direction, and the direction in which the index finger is extended is the roll direction, the yaw direction axis, and the direction perpendicular to the plane defined by the roll direction axis. The direction is defined in the hand object 810 as the pitch direction.

[HMD control device]
The control device of the HMD 110 will be described with reference to FIG. In certain embodiments, the control device is implemented by a computer 200 having a well-known configuration. FIG. 9 is a block diagram showing a computer 200 according to an embodiment as a module configuration.

As shown in FIG. 9, the computer 200 includes a display control module 220, a virtual space control module 230, a voice control module 225, a memory module 240, and a communication control module 250.

The display control module 220 includes a virtual camera control module 221, a field of view area determination module 222, a field of view image generation module 223, and a reference line of sight identification module 224 as submodules.

The virtual space control module 230 includes a virtual space definition module 231, a virtual object generation module 232, and a hand object control module 233 as submodules.

In certain embodiments, the display control module 220, the virtual space control module 230, and the voice control module 225 are implemented by the processor 10. In other embodiments, the plurality of processors 10 may operate as the display control module 220, the virtual space control module 230, and the voice control module 225. The memory module 240 is realized by the memory 11 or the storage 12. The communication control module 250 is realized by the communication interface 14.

In one aspect, the display control module 220 controls the image display on the monitor 112 of the HMD 110. The virtual camera control module 221 arranges the virtual camera 1 in the virtual space 2 and controls the behavior, orientation, and the like of the virtual camera 1. The visual field determination module 222 defines the visual field region 23 according to the orientation of the head of the user 190 wearing the HMD 110. The visual field image generation module 223 generates visual field image data (also referred to as visual field image data) displayed on the monitor 112 based on the determined visual field region 23. Further, the visual field image generation module 223 generates visual field image data based on the data received from the virtual space control module 230. The visual field image data generated by the visual field image generation module 223 is output to the HMD 110 by the communication control module 250. The reference line-of-sight identification module 224 identifies the line-of-sight of the user 190 based on the signal from the gaze sensor 140.

The virtual space control module 230 controls the virtual space 2 provided to the user 190. The virtual space definition module 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2.

The virtual object generation module 232 generates data of an object arranged in the virtual space 2. The object may include, for example, other avatar objects and virtual vehicles. The data generated by the virtual object generation module 232 is output to the field of view image generation module 223.

The hand object control module 233 arranges the hand object in the virtual space 2. The hand object corresponds, for example, to the right or left hand of the user 190 holding the controller 160. In a certain aspect, the hand object control module 233 generates data for arranging the hand object corresponding to the right hand or the left hand in the virtual space 2. Further, the hand object control module 233 generates data for moving the hand object in response to the operation of the controller 160 by the user 190. The data generated by the hand object control module 233 is output to the field of view image generation module 223.

In another aspect, if the movement of a part of the body of the user 190 (eg, the movement of the left hand, right hand, left foot, right foot, head, etc.) is associated with the controller 160, the virtual space control module 230 will be the user 190. Generate data for arranging a partial object corresponding to a part of the body in the virtual space 2. When the user 190 operates the controller 160 using a part of the body, the virtual space control module 230 generates data for moving the partial object. These data are output to the field of view image generation module 223.

When the voice control module 225 detects an utterance using the microphone 119 of the user 190 from the HMD 110, the voice control module 225 identifies the computer 200 to which the voice data to be transmitted corresponding to the utterance is transmitted. The voice data is transmitted to the computer 200 identified by the voice control module 225. When the voice control module 225 receives voice data from another user's computer 200 via the network 19, the voice control module 225 outputs the voice (utterance) corresponding to the voice data from the speaker 115.

The memory module 240 holds data used by the computer 200 to provide the virtual space 2 to the user 190. In a certain aspect, the memory module 240 holds the spatial information 241 and the object information 242 and the user information 243.

Spatial information 241 holds one or more templates defined to provide virtual space 2.

The object information 242 holds the content to be reproduced in the virtual space 2 and the information for arranging the object used in the content. The content may include, for example, a game, content representing a landscape similar to the real world, and the like. Further, the object information 242 includes data for arranging a hand object corresponding to the hand of the user 190 who operates the controller 160 in the virtual space 2, and data for arranging the avatar object of each user in the virtual space 2. Includes data for arranging other objects such as virtual vehicles in virtual space 2.

The user information 243 holds a program for operating the computer 200 as a control device of the HMD system 100, an application program for using each content held in the object information 242, and the like. The data and programs stored in the memory module 240 are input by the user 190 of the HMD 110. Alternatively, the processor 10 downloads a program or data from a computer (for example, a server 150) operated by a business operator that provides the content, and stores the downloaded program or data in the memory module 240.

The communication control module 250 can communicate with the server 150 and other information communication devices via the network 19.

In certain aspects, the display control module 220 and the virtual space control module 230 can be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

The processing in the computer 200 is realized by the hardware and the software executed by the processor 10. Such software may be pre-stored in a hard disk or other memory module 240. The software may also be stored in a computer-readable non-volatile data recording medium such as a CD-ROM and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other networks. Such software is read from the data recording medium by an optical disk drive or other data reader, or downloaded from the server 150 or other computer via the communication control module 250, and then temporarily stored in the storage module. .. The software is read from the storage module by the processor 10 and stored in RAM in the form of an executable program. Processor 10 executes the program.

The hardware that constitutes the computer 200 is general. Therefore, it can be said that the most essential part according to the present embodiment is the program stored in the computer 200. Since the operation of the hardware of the computer 200 is well known, the detailed description will not be repeated.

The data recording medium is not limited to CD-ROM, FD (Flexible Disk), and hard disk, but also magnetic tape, cassette tape, and optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc). ), IC (Integrated Circuit) card (including memory card), optical card, mask ROM, EPROM (Electronically Programmable Read-Only Memory), EEPROM (Electronically Erasable Programmable Read-Only Memory), semiconductor memory such as flash ROM, etc. It may be a non-volatile data recording medium that fixedly carries the program.

The program referred to here may include not only a program that can be directly executed by the processor 10, but also a source program format program, a compressed program, an encrypted program, and the like.

[Interaction between users on the network]
The interaction between users on the network 19 will be described with reference to FIG. FIG. 10 is a diagram conceptually representing one aspect of expressing the virtual space 2 presented by each of the plurality of computers 200.

As shown in FIG. 10, each of the computers 200A to 200F can communicate with the server 150 via the network 19. Each computer 200A-200F provides virtual space images 22A-22F in the corresponding HMD110A-110F. In each of the virtual space images 22A to 22F, avatar objects 901 to 904 corresponding to the users of the computers 200A to 200F are presented.

For example, the avatar object 901 corresponds to the user 190 of the HMD 110A. Therefore, in the virtual space image 22A visually recognized by the user 190 of the HMD 110A, the avatar objects 902 to 904 are presented as other avatar objects corresponding to the other users. On the other hand, the avatar object 904 corresponds to the user 190 of the HMD 110B. Therefore, in the virtual space image 22B visually recognized by the user 190 of the HMD 110B, the avatar objects 901 to 903 are presented as other avatar objects corresponding to the other users.

Each HMD 110A to 110F transmits motion detection data corresponding to the position and inclination of each user to the server 150. The server 150 transmits the motion detection data received from each HMD 110A to 110F to another HMD 110 in the network 19. The other HMD 110 changes the position or tilt of the other avatar object according to the motion detection data.

Further, each of the HMDs 110A to 110F transmits data indicating the detection result of the line-of-sight direction of the user 190 by the gaze sensor 140 to the server 150. The server 150 transmits the detection data in the line-of-sight direction received from each HMD 110A to 110F to another HMD 110 in the network 19. The other HMD 110 changes the line of sight of the other avatar object according to the detection data.

Each HMD 110A to 110F transmits voice data corresponding to each user's utterance to the server 150. The server 150 transmits the voice data received from each HMD 110A to 110F to another HMD 110 in the network 19. The other HMD 110 changes the opening degree of the mouth of the other avatar object according to the voice data. Further, the other HMD 110 outputs a voice based on the voice data from the speaker 115.

In this way, when a user 190 moves or talks in the real space, the position or facial expression of the avatar object corresponding to the user 190 changes in the virtual space 2. In response to this, when another user moves or talks in the real space, the position or facial expression of the avatar object corresponding to the other user changes in the virtual space 2. In this way, by using the communication on the network 19, the interaction between the users who wear different HMD 110s is realized in the virtual space 2.

[Control structure of HMD system]
The control structure of the HMD system 100 will be described with reference to FIG. FIG. 11 is a sequence chart showing a part of the processing executed in the HMD system 100 according to an embodiment.

As shown in FIG. 11, in step S1110, the processor 10 of the computer 200 specifies the virtual space image data as the virtual space definition module 231 and defines the virtual space.

In step S1120, the processor 10 initializes the virtual camera 1. For example, the processor 10 arranges the virtual camera 1 at a predetermined center point in the virtual space 2 in the work area of the memory, and directs the line of sight of the virtual camera 1 in the direction in which the user 190 is facing.

In step S1130, the processor 10 generates the field of view image data for displaying the initial field of view image as the field of view image generation module 223. The generated visual field image data is output to the HMD 110 by the communication control module 250.

In step S1132, the monitor 112 of the HMD 110 displays the visual field image based on the visual field image data received from the computer 200. The user 190 wearing the HMD 110 can recognize the virtual space 2 when he / she visually recognizes the field of view image.

In step S1134, the HMD sensor 120 detects the position and tilt of the HMD 110 based on the plurality of infrared rays emitted from the HMD 110. The detection result is output to the computer 200 as motion detection data.

In step S1140, the processor 10 identifies the viewing direction of the user 190 wearing the HMD 110 based on the position and inclination included in the motion detection data of the HMD 110.

In step S1150, the processor 10 executes the application program and presents an object to the virtual space 2 based on the instruction included in the application program. The object presented at this time includes other avatar objects.

In step S1160, the controller 160 detects the operation of the user 190 based on the signal output from the motion sensor 130, and outputs the detection data representing the detected operation to the computer 200. In another aspect, the operation of the controller 160 by the user 190 may be detected based on the image from the camera arranged around the user 190.

In step S1165, the processor 10 detects the operation of the controller 160 by the user 190 based on the detection data acquired from the controller 160.

In step S1170, the processor 10 generates visual field image data for presenting the hand object in the virtual space 2.

In step S1180, the processor 10 generates visual field image data based on the operation of the controller 160 by the user 190. The generated visual field image data is output to the HMD 110 by the communication control module 250.

In step S1192, the HMD 110 updates the visual field image based on the received visual field image data, and displays the updated visual field image on the monitor 112.

A control structure of the HMD system 100 according to an embodiment will be described with reference to FIGS. 12 and 13. FIG. 12 is a sequence chart showing a part of detailed processing performed in the HMD system 100 according to an embodiment. FIG. 13 is a sequence chart showing a part of communication between a plurality of computers 200. 12 and 13 show an example of communication performed between the computer 200a connected to the HMD110a and the computer 200b connected to the HMD110b. The process executed by the computer 200a or the computer 200b is executed by the processor 10 included in each computer 200.

As shown in FIG. 12, when the user 190 of the HMD 110a speaks using the microphone 119 in step S1210, the voice data corresponding to the utterance is output to the computer 200a in step S1215. In step S1220, the computer 200a accepts the input of voice data.

In step S1225, the computer 200a identifies another avatar object to receive the voice data. In step S1230, the computer 200a outputs the object data corresponding to the specified other avatar object and the voice data to the server 150.

In step S1235, the server 150 identifies the computer 200b to receive the voice data based on the received object data. In step S1240, the server 150 outputs voice data to the specified computer 200b.

As shown in FIG. 13, in step S1310, the computer 200b outputs the voice data received from the server 150 to the HMD 110b. At this time, the speaker 115 of the HMD 110b outputs a voice based on the voice data. When the user of the HMD 110b who has heard the voice performs an operation for responding to the voice, the response data corresponding to the operation is output to the computer 200b in step S1320. The user response operation of the HMD110b includes speech, movement of the line of sight, movement of the face, and the like. When the response action is an utterance, the response data includes voice data corresponding to the utterance. When the response action is to move the line of sight, the response data includes eye tracking data indicating the result of tracking the line of sight. When the response action is to move the face, the response data includes face tracking data indicating the result of tracking the face movement. In step S1330, the computer 200b outputs the received response data to the server 150.

As shown in FIG. 12, in step S1245, the server 150 outputs the received response data to the computer 200a.

In a certain aspect, when the response data is voice data, in step S1250, the computer 200a generates voice data according to the response data and outputs the voice data to the HMD 110a.

In step S1260, the HMD 110a outputs a voice based on the received voice data from the speaker 115.

In step S1255, the computer 200a generates a visual field image according to the response data, and outputs the visual field image data corresponding to the visual field image to the HMD 110a. For example, in a certain aspect, when the response data is voice data, the computer 200a generates a visual field image that moves the mouth of another avatar object corresponding to the user of the computer 200b. In another aspect, when the response data is eye tracking data, the computer 200a generates a field of view image that moves the line of sight of the other avatar object corresponding to the user of the computer 200b. In yet another aspect, when the response data is face tracking data, the computer 200a generates a view image that moves the face of another avatar object corresponding to the user of the computer 200.

In step S1265, the HMD 110a updates the visual field image based on the received visual field image data, and displays the updated visual field image on the monitor 112.

As shown in FIGS. 12 and 13, the time T from the utterance by the user of the computer 200a until the other avatar object corresponding to the user of the computer 200b operates in response to the utterance is the computer 200a and the computer. It depends at least on the communication time with 200b.

Therefore, in step S1270 during the time T, the computer 200a generates a visual field image in which the line of sight of the other avatar object is changed, and outputs the visual field image data corresponding to the visual field image to the HMD 110a. In step S1275, the HMD 110a updates the visual field image based on the received visual field image data, and displays the updated visual field image on the monitor 112. As a result, the line of sight of the other avatar object is changed during the time T.

[Computer control structure]
The control structure of the computer 200 will be described with reference to FIG. FIG. 14 is a flowchart showing detailed processing executed by the processor 10 of the computer 200 in one aspect of an embodiment.

As shown in FIG. 14, in step S1410, the processor 10 determines whether or not the input of the voice data corresponding to the user's utterance is received from the HMD 110.

When the processor 10 does not accept the input of voice data from the HMD 110 (NO in step S1410), the processor 10 ends the process.

On the other hand, when the processor 10 receives the input of voice data from the HMD 110 (YES in step S1410), the processor 10 identifies another avatar object included in the view area 23 in step S1420.

In step S1430, the processor 10 determines whether the hand object points to any object. For example, as shown in the segment (B) of FIG. 8, when the extension direction of the index finger is defined as the roll direction, the processor 10 uses the detection data of the hand movement of the user 190 obtained by the motion sensor 130. Based on this, it is determined whether or not any object is located on the extension line of the axis in the roll direction.

When the hand object does not point to any object (NO in step S1430), the processor 10 generates a field of view image in which the line of sight of the other avatar object is directed to the user 190 side (virtual camera 1 side) in step S1440. do.

In step S1470, the processor 10 outputs the visual field image data corresponding to the visual field image generated in step S1440 to the HMD 110, and ends the process.

On the other hand, when the hand object points to some object (YES in step S1430), the processor 10 identifies the object pointed to by the hand object in step S1450.

In step S1460, the processor 10 generates a visual field image in which the line of sight of the other avatar object is directed to the object identified in step S1430.

In step S1470, the processor 10 outputs the visual field image data corresponding to the visual field image generated in step S1460 to the HMD 110, and ends the process.

As one aspect of the above processing, a mode in which the computer 200 executes each processing step is exemplified, but when the HMD 110 includes a processor, the processor may execute each processing step.

[Presentation of objects in virtual space]
The presentation of the object in the virtual space 2 will be described with reference to FIG. FIG. 15 is a diagram showing an example of presentation of an object in the virtual space 2 according to a certain embodiment. Figure A of FIG. 15 shows the field of view image 1000 displayed on the monitor 112 of the HMD 110. The visual field image 1000 is an image provided by the HMD 110 to the user 190. Figure B of FIG. 15 shows the field of view 23 captured by the virtual camera 1 in the virtual space 2. The field of view area 23 is an area captured by the virtual camera 1 corresponding to the user 190 in the virtual space 2 visible to the user 190. Each of the three axes (u-axis, v-axis, w-axis) of the visual field coordinate system shown in the fractional diagram A is the three axes (x-axis in the horizontal direction and the vertical direction) of the global coordinate system shown in the fractional diagram B. It corresponds to each of the y-axis and the z-axis in the front-back direction).

As shown in the segment B of FIG. 15, in a certain aspect, in the virtual space image 22, the other avatar objects 910, 920, 940, 950, 960, 990 corresponding to each of the six other users and the virtual space. As a virtual vehicle existing in 2, a taxi object 930 representing a taxi and the like are presented. Among these objects, the other avatar objects 910, 920, 990 and the taxi object 930 are presented in the field of view area 23 captured by the virtual camera 1.

As shown in the segment A of FIG. 15, the visual field image 1000 corresponds to the image of the visual field region 23 captured by the virtual camera 1. In a certain aspect, in the field of view image 1000, other avatar objects 910, 920, 990, a taxi object 930, and the like are presented. Further, when the user 190 operates the controller 160, the hand object 970 is presented in the visual field image 1000.

[Changes in objects presented in virtual space]
The change of the object presented in the virtual space 2 will be described with reference to FIG. FIG. 16 is a diagram showing an example of changes in the objects presented in the virtual space 2 according to a certain embodiment. FIG. 16 shows the visual field images 1001 to 1003 displayed on the monitor 112 of the HMD 110. The visual field images 1001 to 1003 are images provided by the HMD 110 to the user 190.

As shown in FIG. 16, in a certain aspect, in the field of view images 1001 to 1003, other avatar objects 910, 920, 990 corresponding to each of the other users and a taxi object 930 are used as objects included in the field of view area 23. Presented. In a certain aspect, when the user 190 speaks using the microphone 119, the visual field image 1001 presents a character image (“sorry”) 820 corresponding to the utterance. In another aspect, even if the user 190 speaks using the microphone 119, the character image 820 corresponding to the utterance may not be presented in the visual field image 1001. The voice data corresponding to the utterance of the user 190 is output to the computer 200 of another user corresponding to all the other avatar objects 910, 920, 990 included in the visual field area 23.

In the visual field image 1001, the lines of sight of the other avatar objects 910, 920, and 990 are directed to the left toward the visual field image 1001. In the field of view image 1002 after the user 190 has spoken, the lines of sight of the other avatar objects 910, 920, and 990 are directed toward the user 190 who has spoken. The update from the visual field image 1001 to the visual field image 1002 is based on the processing of S1270 by the processor 10 shown in FIGS. 12 and 13 and the processing of S1440 by the processor 10 shown in FIG.

When the computer 200 of the user 190 receives the response data from the computer 200 of any other user, the computer 200 updates the field of view image 1002 to the field of view image 1003 based on the response data. The update from the visual field image 1002 to the visual field image 1003 is based on the processing of S1255 by the processor 10 shown in FIGS. 12 and 13. In the field of view image 1003, the bodies of the other avatar objects 910, 920, and 990 are oriented toward the user 190. In a certain aspect, in the field of view image 1003, the mouth of the other avatar object 920 opens. Further, in the visual field image 1003, a character image (“what?”) 830 corresponding to the utterance of another user corresponding to the other avatar object 920 is presented. In another aspect, the mouth of the other avatar object 920 opens, but in the view image 1003, the character image (“what?”) 830 corresponding to the utterance of the other user corresponding to the other avatar object 920 is not presented. You may.

With reference to FIG. 17, the visual field images 1004 to 1006 displayed on the monitor 112 after the visual field image 1003 shown in FIG. 16 will be described. FIG. 17 is a diagram showing an example of changes in the objects presented in the virtual space 2 according to a certain embodiment. FIG. 17 shows the visual field images 1004 to 1006 displayed on the monitor 112 of the HMD 110. The visual field images 1004 to 1006 are images visually recognized by the user 190 wearing the HMD 110.

As shown in FIG. 17, in the field of view images 1004 to 1006, other avatar objects 910, 920, 990 and taxi objects 930 corresponding to each of the other users are presented as objects included in the field of view area 23. When the user 190 speaks using the microphone 119, the visual field image 1004 presents a character image (“is it a taxi?”) 840 corresponding to the utterance. In another aspect, even if the user 190 speaks using the microphone 119, the visual field image 1004 does not have to present the character image 840 corresponding to the utterance. Further, when the user 190 points to the taxi object 930 by operating the controller 160, the pointing direction of the hand object 970 changes to the direction of the taxi object 930.

In the field of view image 1004, the lines of sight of the other avatar objects 910, 920, and 990 are directed toward the user 190. In a certain aspect, in the view image 1005 after the user 190 speaks, the lines of sight of the other avatar objects 910, 920, and 990 are directed to the taxi object 930 pointed to by the hand object 970. The update from the visual field image 1004 to the visual field image 1005 is based on the processing of S1270 by the processor 10 shown in FIGS. 12 and 13 and the processing of S1460 by the processor 10 shown in FIG.

When the computer 200 of the user 190 receives the response data from the computer 200 of another user, the computer 200 updates the field of view image 1005 to the field of view image 1006 based on the response data. The update from the visual field image 1005 to the visual field image 1006 is based on the processing of S1255 by the processor 10 shown in FIGS. 12 and 13. In the field of view image 1006, the bodies of the other avatar objects 910, 920, and 990 are oriented toward the user 190. In a certain aspect, in the field of view image 1006, the mouth of the other avatar object 920 opens. Further, in the visual field image 1006, a character image (“Yes, that's right”) 850 corresponding to the utterance of another user corresponding to the other avatar object 920 is presented. In another aspect, the mouth of the other avatar object 920 opens, but in the view image 1006, the character image ("Yes, that's right") 850 corresponding to the utterance of the other user corresponding to the other avatar object 920 is not presented. May be good.

[Change the line of sight of other avatar objects in other aspects]
In another aspect, the processor 10 of the computer 200 changes the line of sight of the other avatar object included in the conversation area 23a. FIG. 18 is a diagram illustrating an example of presentation of an object in virtual space 2 according to another aspect of one embodiment. Figure B of FIG. 18 shows the visual field image 2000 displayed on the monitor 112 of the HMD 110 in another aspect. Figure B of FIG. 18 shows the field of view 23 and the conversation area 23a captured by the virtual camera 1 in other aspects.

As shown in the segment B of FIG. 18, the visual field area 23 includes the conversation area 23a. The conversation area 23a is an area closer to the user 190 side (virtual camera 1 side) than the field of view area 23. In the example of segment B in FIG. 18, the visual field area 23 includes the other avatar object 990, while the conversation area 23a does not include the other avatar object 990.

The control structure of the computer 200 in another aspect will be described with reference to FIG. FIG. 19 is a flowchart showing detailed processing executed by the processor 10 of the computer 200 in another aspect of an embodiment. In the flowchart shown in FIG. 19, S1420 in the flowchart shown in FIG. 14 is changed to S1420a. In FIG. 19, the same processes as those shown in FIG. 14 are assigned the same step numbers. Therefore, the description of the same process is not repeated.

As shown in FIG. 19, when the processor 10 receives the input of voice data from the HMD 110 (YES in step S1410), the processor 10 identifies another avatar object included in the conversation area 23a in step S1420a. After that, the processor 10 generates a field of view image in which the line of sight of the specified other avatar object is changed in S1440 or S1460. That is, the processor 10 changes the line of sight of the other avatar object included in the conversation area 23a among the other avatar objects included in the field of view area 23.

The presentation of objects in other aspects will be described with reference to FIG. FIG. 20 is a diagram showing an example of changes in an object presented in virtual space 2 according to another aspect of one embodiment. FIG. 20 shows the visual field images 2001-2003 displayed on the monitor 112 of the HMD 110 in another aspect.

As shown in FIG. 20, the field of view images 2001 to 2003 include other avatar objects 910, 920, 990 corresponding to each of the other users and a taxi object 930 as objects included in the field of view area 23. When the user 190 speaks using the microphone 119, the visual field image 2001 is presented with a character image (“sorry”) 820 corresponding to the utterance. The voice data corresponding to the utterance of the user 190 is output to the computer 200 of another user corresponding to the other avatar objects 910 and 920. On the other hand, the voice data corresponding to the utterance of the user 190 is not output to the computer 200 of the other user corresponding to the other avatar object 990.

In the visual field image 2001, the lines of sight of the other avatar objects 910, 920, and 990 are directed to the left toward the visual field image 2001. In the field of view image 2002 after the user 190 has spoken, the lines of sight of the other avatar objects 910 and 920 are directed toward the user 190 who has spoken. On the other hand, in the field of view image 2002, the line of sight of the avatar object 990 does not face the user 190 who spoke.

When the computer 200 of the user 190 receives the response data from the computer 200 of any other user among the other users corresponding to the other avatar objects 910 and 920, the computer of the user 190 is based on the response data. 200 updates the view image 2002 to the view image 2003. In the field of view image 2003, the orientations of the other avatar objects 910 and 920 are toward the user 190. In the view image 2003, the mouth of the avatar object 920 is opened, and a character image (“what?”) 830 corresponding to the utterance of another user corresponding to the avatar object 920 is presented.

[Change the line of sight of other avatar objects in other aspects]
In another aspect, the processor 10 of the computer 200 directs the line of sight of the other avatar object included in the field of view 23 to the object associated with the content of the utterance of the user 190. The control structure of the computer 200 in another aspect will be described with reference to FIG. FIG. 21 is a flowchart showing detailed processing executed by the processor 10 of the computer 200 in another aspect of an embodiment. In the flowchart shown in FIG. 21, S1440 in the flowchart shown in FIG. 14 is changed to S1440a and S1440b. In FIG. 21, the same processes as those shown in FIG. 14 are assigned the same step numbers. Therefore, the description of the same process is not repeated.

As shown in FIG. 21, when the processor 10 accepts the input of audio data from the HMD 110 (YES in step S1410) and the hand object does not point to any object (NO in step S1430), in step S1440a. To identify the object associated with the content of the user 190's utterance. Recognition of the content of the utterance is realized by, for example, a known voice recognition function.

In step S1440b, the processor 10 generates a visual field image in which the line of sight of the other avatar object is directed to the object specified in step S1440a.

In another aspect, the processor 10 directs the line of sight of the other avatar object included in the conversation area 23a shown in the segment B of FIG. 18 to the object related to the content of the utterance of the user 190 in the field of view area 23. May be good.

Further, in another aspect, if the hand object does not point to any object (NO in step S1430) and the object related to the content of the user 190's utterance cannot be identified, the processor 10 of the other avatar object. A field view image with the line of sight directed toward the user 190 may be generated.

[Change the line of sight of other avatar objects in other aspects]
In another aspect, when the processor 10 of the computer 200 detects the movement of the user 190's hand by the motion sensor 130 provided in the controller 160 without accepting the input of the voice data corresponding to the user's utterance, the view is visible. The line of sight of another avatar object included in the area 23 is changed. The control structure of the computer 200 in another aspect will be described with reference to FIG. FIG. 22 is a flowchart showing detailed processing executed by the processor 10 of the computer 200 in another aspect of an embodiment.

As shown in FIG. 22, in step S2210, the processor 10 determines whether or not the input of the hand movement detection data of the user 190 has been accepted from the motion sensor 130.

When the processor 10 does not accept the input of the hand movement detection data of the user 190 from the motion sensor 130 (NO in step S2210), the processor 10 ends the process.

On the other hand, when the processor 10 receives the input of the hand movement detection data of the user 190 from the motion sensor 130 (YES in step S2210), the processor 10 identifies another avatar object included in the view area 23 in step S2220. ..

In step S2230, the processor 10 identifies the object pointed to by the hand object. For example, the processor 10 identifies an object located on an extension of the roll direction axis shown in the segment diagram (B) of FIG. 8 based on the signal obtained by detecting the movement of the user 190's hand by the motion sensor 130.

In step S2240, the processor 10 generates a visual field image in which the line of sight of the other avatar object is directed to the object identified in step S2230.

In step S2250, the processor 10 outputs the visual field image data corresponding to the visual field image generated in step S2250 to the HMD 110, and ends the process.

[Other configurations]
In another aspect, the processor 10 of the computer 200, in addition to moving the line of sight of the other avatar object during the time T from when the user 190 speaks until the other avatar object operates in response to the utterance. , At least a part of the body of the other avatar object may be moved. For example, the processor 10 may move the line of sight of the other avatar object by moving the face of the other avatar object during the time T.

In another aspect, the processor 10 of the computer 200 may direct the line of sight of the other avatar object to the object possessed by the partial object corresponding to the part of the body of the user 190 during the time T.

In another aspect, in step S1430, the processor 10 of the computer 200 determines whether or not any object is located not only on the axis in the roll direction but also on the axis in the yaw direction or the extension line of the axis in the pitch direction. You may. In still another aspect, the processor 10 may determine in step S1430 whether or not the partial object corresponding to a part of the body, not limited to the hand object, points to any object.

In another aspect, the processor 10 of the computer 200 may specify an object located not only on the axis in the roll direction but also on the axis in the yaw direction or the extension line of the axis in the pitch direction in step S2230. In still another aspect, in step S2230, the processor 10 may determine whether or not a partial object corresponding to a part of the body, not limited to the hand object, points to some object.

In another aspect, the processor 10 of the computer 200 may present not only the other avatar object but also the avatar object corresponding to the user 190 who is the user 190 in the virtual space 2.

In another aspect, the processor 10 of the computer 200 may determine the volume of the utterance of the user 190 and change the conversation area 23a according to the determination result. For example, the processor 10 may set the conversation area 23a closer to the user 190 when the utterance volume of the user 190 is louder than when the utterance volume is low.

[Summary of the present disclosure]
The disclosed technical features include, for example, the following configurations.

(Structure 1)
According to certain embodiments, there is provided a method performed by the computer 200 to communicate via the virtual space 2. In this method, the step of defining the virtual space 2 (S1110) and one or more other avatar objects corresponding to one or more other users with whom the user 190 of the computer 200 can communicate are presented to the virtual space 2. A step (S1150), a step of accepting input of voice data corresponding to the speech of the user 190 of the computer 200 (YES in S1220 and S1410), and a step of outputting the voice data to the computer 200 of another user (S1230). , In the step (S1225) of moving the other avatar object according to the response data output from the computer 200 of another user in response to the input of the voice data, and between the speech and the movement of the other avatar object (time T). The step (S1270, S1440, S1440b, S1460) of changing the line of sight of another avatar object is included.

(Structure 2)
According to a certain embodiment, the step of changing the line of sight of the other avatar object is a predetermined conversation area 23a of the view area 23 in the virtual space 2 of the user 190 of the computer 200 communicating via the virtual space 2. Includes changing the line of sight of another avatar object included in (steps S1420a, S1440, S1460).

(Structure 3)
According to a certain embodiment, the step of changing the line of sight of the other avatar object includes directing the line of sight of the other avatar object to the user 190 side of the computer 200 communicating via the virtual space 2 (S1440).

(Structure 4)
According to a certain embodiment, the above method includes a step (S1170) of presenting a hand object 970 corresponding to the hand of the user 190 of the computer 200 communicating via the virtual space 2 to the virtual space 2 and a hand operation. The step of further including the step of detecting (S1165) and the step of moving the hand object 970 based on the movement of the hand (S1180), and the step of changing the line of sight of the other avatar object is the direction in which the hand object 970 points to the line of sight of the other avatar object. Includes pointing to (S1460).

(Structure 5)
According to a certain embodiment, the above method further includes a step (S1150) of presenting an object related to the content of the utterance to the virtual space 2, and a step of changing the line of sight of the other avatar object is the line of sight of the other avatar object. Includes pointing at an object associated with the content of the utterance (S1440b).

(Structure 6)
According to an embodiment, the response data includes eye tracking data showing the result of tracking the line of sight of another user, face tracking data showing the result of tracking the facial movement of another user, and speech of another user. Includes any of the audio data corresponding to.

(Structure 7)
According to other embodiments, there is provided a method performed by the computer 200 to communicate via the virtual space 2. This method includes a step of defining the virtual space 2 (S1110), a step of presenting another avatar object to the virtual space 2 (S1150), a step of changing the line of sight of the other avatar object (S2240), and the virtual space 2. A step (S1770) of presenting the hand object 970 corresponding to the hand of the user 190 of the computer 200 communicating via the hand to the virtual space 2, a step of detecting the movement of the hand (S1165), and a hand object based on the movement of the hand. The step of changing the line of sight of the other avatar object includes the step of moving the 970 (S1180), and the step of changing the line of sight of the other avatar object includes directing the line of sight of the other avatar object in the direction pointed to by the hand object 970 (S2230, S2240).

(Structure 8)
According to certain embodiments, a program is provided that causes a computer to perform any of the above methods.

(Structure 9)
According to certain embodiments, a computer 200 is provided that includes a memory 11 that stores the above program and a processor 10 that executes the program.

As described above, according to a certain embodiment, it is assumed that the time T from the utterance of the user 190 to the operation of the other avatar object corresponding to the other user in response to the utterance becomes longer due to a communication delay or the like. However, the line of sight of the other avatar object is changed during the time T. As a result, the user 190 does not feel uncomfortable about the conversation in the virtual space 2, so that it becomes easy for the user 190 to interact with the plurality of users in the virtual space 2.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and it is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 virtual camera, 2 virtual space, 5 reference line of sight, 10 processor, 11 memory, 12 storage, 13 input / output interface, 14 communication interface, 15 bus, 19 network, 21 center, 22 virtual space image, 23 view area, 23a conversation Area, 30 grips, 31 frames, 32 tops, 33, 34, 36, 37 buttons, 38 analog sticks, 100 systems, 110 HMDs, 112 monitors, 114,120 sensors, 115 speakers, 119 microphones, 130 motion sensors, 140 Gaze sensor, 150 server, 160 controller, 190 user, 200 computer, 220 display control module, 221 virtual camera control module, 222 view area determination module, 223 view image generation module, 224 reference line-of-sight identification module, 225 voice control module, 230 Virtual space control module, 231 virtual space definition module, 232 virtual object generation module, 233 hand object control module, 240 memory module, 241 space information, 242 object information, 243 user information, 250 communication control module, 800 right controller, 810 right hand , 815, 830, 840, 850 character images, 910, 920, 990 other avatar objects, 930 taxi objects, 970 hand objects, 1000, 2000 visibility images.

Claims (4)

A method performed by a computer to communicate over virtual space,
The acquisition step to acquire the voice data related to the utterance of the first user,
A range setting step for setting a range in which the utterance of the first user can be recognized based on the voice data, and
Including
The range is a range in which an avatar object corresponding to a second user different from the first user, who can recognize the utterance of the first user, exists.
When the voice data is acquired in the acquisition step, the avatar object corresponding to the second user existing in the range is set to operate, and the avatar object corresponding to the second user not in the range is operated. The operation setting step to set not to let
A specific step of identifying the object in the virtual space based on the information indicating the object related to the content of the utterance of the first user included in the voice data, and
Including
In the operation setting step, based on the information indicating the object specified in the specific step, the avatar object corresponding to the second user existing in the range is operated so as to pay attention to the object.
Method. The computer
If the object was not identified in the virtual space in the specific step
In the operation setting step, the avatar object corresponding to the second user existing in the range is operated so as to attract the attention of the first user.
The method according to claim 1. The computer
In the range setting step, the range is set based on the volume information regarding the volume of the utterance of the first user included in the voice data.
The method according to claim 1 or 2. The computer
In the range setting step, the range is set to be reduced as the volume increases, based on the volume information regarding the volume of the utterance of the first user included in the voice data.
The method according to any one of claims 1 to 3.

Images