JP7794852B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device.

In MR (Mixed Reality) technology, the real-world environment perceived by the user is augmented by a computer. Using this technology, it is possible, for example, to precisely overlay a virtual space onto the real world perceived through MR glasses worn by the user.

For example, Patent Document 1 discloses an information processing device that displays realistic virtual reality images while ensuring the safety of a user riding in a vehicle. The information processing device is equipped with AR goggles that display images of multiple virtual first objects. The information processing device also detects second objects that exist outside or inside the vehicle. If the information processing device determines that the image of the first object will be superimposed on a visibility assurance area that affects the user's visibility of the second object, it moves the position of the image of the first object, for example, so that the image of the first object does not superimpose on the visibility assurance area.

Japanese Patent Application Laid-Open No. 2020-204984

However, the technology described in Patent Document 1 only defines the positional relationship between a virtual first object and a real second object, and is not able to change the state of the second object depending on which object the user is focusing on.

The present invention aims to solve the problem of changing the state of one of two virtual objects placed in a virtual space when the user's focus changes from a state in which the user is focusing on the other virtual object.

An information processing device according to a preferred aspect of the present invention is an information processing device comprising: an acquisition unit that acquires gaze information indicating a user's gaze; a display control unit that displays a first virtual object and a second virtual object in positions in virtual space where they do not overlap each other as seen by the user; and an operation control unit that transitions the second virtual object from inactive to active after a first state in which the user's gaze indicated by the gaze information intersects with the first virtual object and the first virtual object is active transitions to a second state in which the user's gaze does not intersect with the first virtual object.

According to the present invention, when a user focuses on one of two virtual objects placed in a virtual space and there is a change in the object the user is focusing on, it is possible to change the state of the other virtual object.

1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment. FIG. 1 is a perspective view showing the appearance of MR glasses 20 according to a first embodiment. Schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 20 according to the first embodiment. Schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 20 according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of MR glass 20 according to the first embodiment. FIG. 1 is a block diagram showing an example of the configuration of a terminal device 10 according to a first embodiment. FIG. 4 is an explanatory diagram illustrating an example of the operation of a display control unit 112 according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of a server 30 according to the first embodiment. 5 is a flowchart showing the operation of the server 30 according to the first embodiment. FIG. 10 is an explanatory diagram illustrating an example of the operation of a display control unit 112 according to a modified example.

1: First Embodiment Hereinafter, with reference to FIGS. 1 to 9, a configuration of an information processing system 1 including a terminal device 10 as an information processing device according to a first embodiment of the present invention will be described.

1-1: Configuration of the first embodiment
1-1-1: Overall Configuration Fig. 1 shows the overall configuration of an information processing system 1 according to a first embodiment of the present invention. The information processing system 1 is a system that uses MR technology to provide a virtual space to a user U1 wearing MR glasses 20 (described later).

The information processing system 1 includes a terminal device 10, MR glasses 20, and a server 30. The terminal device 10 is an example of an information processing device. In the information processing system 1, the terminal device 10 and the server 30 are communicatively connected to each other via a communication network NET. The terminal device 10 and the MR glasses 20 are also communicatively connected to each other. Note that in FIG. 1, three pairs of terminal devices 10 and MR glasses 20 are shown: a pair of terminal device 10-1 and MR glasses 20-1, a pair of terminal device 10-2 and MR glasses 20-2, and a pair of terminal device 10-3 and MR glasses 20-3. However, this number of pairs is merely an example, and the information processing system 1 can include any number of pairs of terminal devices 10 and MR glasses 20.

The server 30 provides various data and cloud services to the terminal device 10 via the communication network NET.

The terminal device 10 displays virtual objects placed in a virtual space on the MR glasses 20 worn on the user's head. The virtual space is, for example, a celestial sphere. Examples of the virtual objects include virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, as well as virtual objects representing applications. Examples of text files include memos, source code, diaries, and recipes. Examples of applications include browsers, applications for using SNS, and applications for generating document files. The terminal device 10 is preferably, for example, a mobile terminal device such as a smartphone or tablet.

The MR glasses 20 are a see-through wearable display worn on the user's head. The MR glasses 20 are controlled by the terminal device 10 to display virtual objects on display panels provided in each of the lenses for both eyes. The MR glasses 20 are an example of a display device.

1-1-2: Configuration of MR Glasses Fig. 2 is a perspective view showing the appearance of the MR glasses 20. As shown in Fig. 2, the appearance of the MR glasses 20 is similar to that of ordinary eyeglasses, and includes temples 91 and 92, a bridge 93, frames 94 and 95, and lenses 41L and 41R. An imaging device 26 is provided on the bridge 93. The imaging device 26 captures an image of the outside world. The imaging device 26 also outputs imaging information indicating the captured image.

Each of the lenses 41L and 41R is equipped with a half mirror. The frame 94 is provided with a liquid crystal panel or organic EL panel for the left eye and an optical element that guides light emitted from the left eye display panel to the lens 41L. Hereinafter, the liquid crystal panel or organic EL panel will be collectively referred to as the display panel. The half mirror provided on the lens 41L transmits external light to guide it to the left eye and reflects light guided by the optical element to make it incident on the left eye. The frame 95 is provided with a display panel for the right eye and an optical element that guides light emitted from the right eye display panel to the lens 41R. The half mirror provided on the lens 41R transmits external light to guide it to the right eye and reflects light guided by the optical element to make it incident on the right eye.

The display 28 described below includes a lens 41L, a display panel for the left eye, and an optical element for the left eye, as well as a lens 41R, a display panel for the right eye, and an optical element for the right eye.

In the above configuration, the user can observe the image displayed on the display panel in a see-through state, superimposed on the outside world. Furthermore, by displaying the image for the left eye on the left eye display panel and the image for the right eye on the right eye display panel in the MR glasses 20, of the binocular images with parallax, the user U1 can perceive the displayed image as if it had depth and a three-dimensional effect.

Figures 3 and 4 are schematic diagrams of a virtual space VS provided to user U1 using MR glasses 20. As shown in Figure 3, virtual objects VO1 to VO5 representing various content, such as a browser, cloud services, images, and videos, are placed in the virtual space VS. By moving around a public space while wearing MR glasses 20 displaying virtual objects VO1 to VO5 placed in the virtual space VS, user U1 can experience the virtual space VS as a private space in a public space. As a result, user U1 can act in the public space while enjoying the benefits provided by the virtual objects VO1 to VO5 placed in the virtual space VS.

As shown in Figure 4, it is also possible for multiple users U1 to U3 to share a virtual space VS. By sharing a virtual space VS among multiple users U1 to U3, the multiple users U1 to U3 can share one or more virtual objects VO, and can communicate between users U1 to U3 via the shared virtual objects VO.

In this embodiment, as shown in Figures 3 and 4, among the multiple virtual objects VO1 to VO5, any two virtual objects VO in any combination are displayed so as not to overlap in the virtual space VS as seen by user U1. Details of the display method of the virtual objects VO will be described later.

Figure 5 is a block diagram showing an example configuration of the MR glasses 20. The MR glasses 20 include a processing unit 21, a storage unit 22, a gaze detection unit 23, a GPS unit 24, a motion detection unit 25, an imaging unit 26, a communication unit 27, a display 28, and a speaker 29. Each element of the MR glasses 20 is interconnected by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, or unit.

The processing device 21 is a processor that controls the entire MR glass 20. The processing device 21 is configured, for example, using one or more chips. The processing device 21 is also configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 21 may be realized by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The processing device 21 executes various processes in parallel or sequentially.

The storage device 22 is a recording medium that can be read and written by the processing device 21. The storage device 22 also stores multiple programs, including the control program PR1, which is executed by the processing device 21.

The gaze detection device 23 detects the gaze of user U1. Any method may be used to detect the gaze by the gaze detection device 23. For example, the gaze detection device 23 may detect gaze information based on the position of the inner corner of the eye and the position of the iris. Furthermore, the gaze detection device 23 outputs gaze information indicating the direction of user U1's gaze based on the detection results to the processing device 21 described below. The gaze information output to the processing device 21 is output to the terminal device 10 via the communication device 27.

The GPS device 24 receives radio waves from multiple satellites. The GPS device 24 also generates location information from the received radio waves. The location information indicates the location of the MR glasses 20. The location information may be in any format as long as it can identify the location. The location information indicates, for example, the latitude and longitude of the MR glasses 20. As an example, the location information is obtained from the GPS device 24. However, the MR glasses 20 may obtain the location information by any method. The obtained location information is supplied to the processing device 21. The location information output to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

The motion detection device 25 detects the movement of the MR glasses 20. Examples of the motion detection device 25 include inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration. The acceleration sensor detects acceleration in the orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as the central axes of rotation. The motion detection device 25 can generate attitude information indicating the attitude of the MR glasses 20 based on the output information of the gyro sensor. The motion detection device 25 supplies attitude information related to the attitude of the MR glasses 20 to the processing device 21. The motion detection device 25 also supplies motion information related to the movement of the MR glasses 20 to the processing device 21. The motion information includes acceleration data indicating the acceleration in each of the three axes and angular acceleration data indicating the angular acceleration in each of the three axes. The attitude information and motion information supplied to the processing device 21 are transmitted to the terminal device 10 via the communication device 27.

The imaging device 26 outputs imaging information obtained by capturing an image of the outside world. The imaging device 26 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted by the imaging element into an imaging signal, which is an analog signal. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is supplied to the processing device 21. The imaging information supplied to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

The communication device 27 is hardware that functions as a transmitting/receiving device for communicating with other devices. The communication device 27 is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 27 may have a connector for wired connection and an interface circuit corresponding to the connector. The communication device 27 may also have a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN and Bluetooth (registered trademark), etc.

The display 28 is a device that displays images. The display 28 displays various images under the control of the processing device 21. As described above, the display 28 includes a lens 41L, a display panel for the left eye, and an optical element for the left eye, as well as a lens 41R, a display panel for the right eye, and an optical element for the right eye. Various display panels, such as a liquid crystal display panel and an organic EL display panel, are preferably used as the display panel.

The speaker 29 is a device that emits sound. The speaker 29 emits various sounds under the control of the processing device 21. For example, audio data, which is a digital signal, is converted into an audio signal, which is an analog signal, by a DA converter (not shown). The amplitude of the audio signal is amplified by an amplifier (not shown). The speaker 29 emits the sound represented by the audio signal after the amplitude has been amplified.

The processing device 21 functions as an acquisition unit 211 and a display control unit 212, for example, by reading and executing the control program PR1 from the storage device 22.

The acquisition unit 211 acquires image information indicating an image to be displayed on the MR glasses 20 from the terminal device 10. More specifically, the acquisition unit 211 acquires, for example, the second image information (described below) transmitted from the terminal device 10.

The acquisition unit 211 also acquires gaze information input from the gaze detection device 23, position information input from the GPS device 24, posture information and movement information input from the movement detection device 25, and imaging information input from the imaging device 26. The acquisition unit 211 then outputs the acquired gaze information, position information, posture information, movement information, and imaging information to the communication device 27.

The display control unit 212 displays the image indicated by the second image information on the display 28 based on the second image information acquired from the terminal device 10 by the acquisition unit 211.

1-1-3: Configuration of Terminal Device Fig. 6 is a block diagram showing an example configuration of the terminal device 10. The terminal device 10 includes a processing device 11, a storage device 12, a communication device 13, a display 14, an input device 15, and an inertial sensor 16. The elements of the terminal device 10 are connected to each other by one or more buses for communicating information.

The processing device 11 is a processor that controls the entire terminal device 10. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, etc. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 11 executes various processes in parallel or sequentially.

The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 also stores multiple programs, including the control program PR2, which is executed by the processing device 11.

The communication device 13 is hardware that functions as a transmitting/receiving device for communicating with other devices. The communication device 13 is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN and Bluetooth (registered trademark), etc.

The display 14 is a device that displays images and text information. The display 14 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitable for use as the display 14.

The input device 15 accepts operations from a user U1 wearing the MR glasses 20 on their head. For example, the input device 15 may include a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 15 includes a touch panel, it may also serve as the display 14.

The inertial sensor 16 is a sensor that detects inertial force. The inertial sensor 16 includes, for example, one or more of an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 11 detects the attitude of the terminal device 10 based on the output information of the inertial sensor 16. Furthermore, the processing device 11 accepts the selection of a virtual object VO, the input of text, and the input of instructions in the spherical virtual space VS based on the attitude of the terminal device 10. For example, user U1 operates the input device 15 while pointing the central axis of the terminal device 10 toward a predetermined area in the virtual space VS, thereby selecting a virtual object VO to be placed in the predetermined area. An example of the operation performed by user U1 on the input device 15 is a double tap. In this way, user U1 can operate the terminal device 10 to select a virtual object VO without looking at the input device 15 of the terminal device 10.

The processing device 11 functions as an acquisition unit 111, a display control unit 112, a judgment unit 113, and an operation control unit 114 by reading and executing the control program PR2 from the storage device 12.

The acquisition unit 111 acquires gaze information indicating the line of sight of the user U1. More specifically, the acquisition unit 111 acquires gaze information output from the MR glasses 20 via the communication device 13. The acquisition unit 111 also acquires the first image information (described below) indicating the image displayed on the MR glasses 20 from the server 30 via the communication device 13.

Based on the first image information acquired by the acquisition unit 111, the display control unit 112 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as seen by the user U1. Specifically, the first image information is image information showing images of the first virtual object VO1 itself and the second virtual object VO2 itself. In addition, the display control unit 112 transmits second image information, including the above-mentioned first image information and layout information such that the first virtual object VO1 and the second virtual object VO2 do not overlap each other as seen by the user U1 in the virtual space VS viewed by the user U1 through the MR glasses 20, to the MR glasses 20 via the communication device 13.

Figure 7 is an explanatory diagram of a display method for the first virtual object VO1 and the second virtual object VO2 by the display control unit 112. In the following description, assume that the virtual space VS has an X-axis, Y-axis, and Z-axis that are orthogonal to each other. As an example, the X-axis extends left and right from the user U1. Furthermore, from the perspective of the user U1, the right direction along the X-axis is the positive direction, and the left direction along the X-axis is the negative direction. The Y-axis extends forward and backward from the user U1. Furthermore, from the perspective of the user U1, the forward direction along the Y-axis is the positive direction, and the backward direction along the Y-axis is the negative direction. These X-axis and Y-axis form a horizontal plane. Furthermore, the Z-axis is orthogonal to the XY plane and extends up and down from the user U1. Furthermore, from the perspective of the user U1, the upward direction along the Z-axis is the positive direction, and the downward direction along the Z-axis is the negative direction.

In the virtual space VS, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of user U1, i.e., the coordinates of one end point of user U1's line of sight V, is assumed to be (x, y, z) = (x0, y0, z0). Furthermore, the display control unit 112 displays the first virtual object VO1 at the position (x, y, z) = (x1, y1, z1).

Here, a cylinder Y is assumed to represent the region in which user U1's line of sight V may exist while user U1 is viewing the first virtual object VO1. The cylinder Y has a central axis L1, with the position (x0, y0, z0) of user U1's left eye pupil EL or right eye pupil ER as its first endpoint C1 and the position (x1, y1, z1) of the first virtual object VO1 as its second endpoint C2. Furthermore, the radius R1 of the base of the cylinder Y is a length that allows the base to include the first virtual object VO1. Note that, for ease of explanation, in Figure 7, the position of user U1's left eye pupil EL or right eye pupil ER does not coincide with the position of the first endpoint C1. Similarly, in Figure 7, the position (x1, y1, z1) of the first virtual object VO1 does not coincide with the position of the second endpoint C2. However, for practical purposes of the present invention, these respective positions coincide.

The display control unit 112 displays the second virtual object VO2 in the virtual space VS outside the cylinder Y in an inactive state. Specifically, the display position (x, y, z) = (x2, y2, z2) of the second virtual object VO2 is a position where the entire second virtual object VO2 is outside the range of the cylinder Y. Here, "active" means that the virtual object VO is in operation or has been selected by the user U1. On the other hand, "inactive" means that the operation of the virtual object VO is stopped or that the virtual object VO has not been selected by the user U1. For example, if the virtual object VO corresponds to a video application, "the virtual object VO is in operation" means that the video is being played by the user U1's operation. Furthermore, if the virtual object VO corresponds to an email application, "the virtual object VO is in operation" means that the email is being sent or received, or a text file containing the contents of the email is being opened by the user U1's operation. Furthermore, when the virtual object VO corresponds to a music application, "the virtual object VO is in operation" means that the music is being emitted from the speaker 29 by operation of the user U1.

The display control unit 112 displays the second virtual object VO2 in the virtual space VS in a first state in which the line of sight V of the user U1 intersects with the first virtual object VO1 and the first virtual object VO1 is active. Specifically, the display control unit 112 displays the second virtual object VO2 in its entirety outside the cylinder Y in the first state in which the line of sight V of the user U1 is located entirely inside the cylinder Y and the first virtual object VO1 is active.

The display position of the second virtual object VO2 is an area within a predetermined distance from the first virtual object VO1. Specifically, the second virtual object VO2 is displayed within the field of view of the user U1 while the user U1 is viewing the first virtual object VO1. This processing enables the terminal device 10 to draw the user U1's attention to the second virtual object VO2. The above-mentioned "predetermined distance" is an example of the "first distance."

Returning to the explanation of Figure 6, the determination unit 113 determines whether a first state in which the line of sight V of user U1 intersects with the first virtual object VO1 and the first virtual object VO1 is active has transitioned to a second state in which the line of sight V of user U1 does not intersect with the first virtual object VO1.
The operation control unit 114 transitions the second virtual object VO2 from inactive to active after the determination unit 113 determines that the first state has transitioned to the second state. Specifically, in Fig. 7 , after the line of sight V of the user U1 changes from a state where it fits within the cylinder Y to a state where it does not fit within the cylinder Y, the operation control unit 114 transitions the second virtual object VO2 from inactive to active.

In the second state, it is preferable that the operation control unit 114 transitions the second virtual object VO2 from inactive to active after the user U1's line of sight V remains stationary for a predetermined period of time. Specifically, in the second state, the operation control unit 114 transitions the second virtual object VO2 from inactive to active on the condition that the change per unit time of the user U1's line of sight V remains within a predetermined range for a predetermined period of time. As a result, the terminal device 10 can transition the second virtual object VO2 from inactive to active after the user U1's line of sight V is no longer wandering and is close to a stationary state. The "predetermined range" is an example of a "second range." The "predetermined time" is an example of a "second time."

Furthermore, after transitioning from the first state to the second state, the operation control unit 114 transitions the first virtual object VO1 from active to inactive. This process allows the terminal device 10 to switch between the first virtual object VO1 and the second virtual object VO2 to activate, depending on the position of the user U1's line of sight V. Furthermore, the terminal device 10 can exclusively activate the first virtual object VO1 and the second virtual object VO2. The state transitions of the first virtual object VO1 and the second virtual object VO2 may be executed simultaneously, or one may be executed before the other.

1-1-4: Server Configuration Fig. 8 is a block diagram showing an example configuration of the server 30. The server 30 includes a processing device 31, a storage device 32, a communication device 33, a display 34, and an input device 35. The elements of the server 30 are connected to one another by one or more buses for communicating information.

The processing device 31 is a processor that controls the entire server 30. The processing device 31 is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, etc. Some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 31 executes various processes in parallel or sequentially.

The storage device 32 is a recording medium that can be read and written by the processing device 31. The storage device 32 also stores multiple programs, including the control program PR3, which is executed by the processing device 31.

The communication device 33 is hardware that functions as a transmitting/receiving device for communicating with other devices. The communication device 33 is also referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN and Bluetooth (registered trademark), etc.

The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing device 31. For example, various display panels such as a liquid crystal display panel and an organic EL display panel are suitable for use as the display 34.

The input device 35 is a device that accepts operations by the administrator of the information processing system 1. For example, the input device 35 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 35 includes a touch panel, it may also serve as the display 34.

The processing device 31 functions as an acquisition unit 311 and a generation unit 312, for example, by reading and executing the control program PR3 from the storage device 32.

The acquisition unit 311 acquires various data from the terminal device 10 using the communication device 33. This data includes, for example, data indicating the operation content for the virtual object VO input to the terminal device 10 by a user U1 wearing the MR glasses 20 on their head.

The generation unit 312 generates first image information indicating an image to be displayed on the MR glasses 20. This first image information is transmitted to the terminal device 10 via the communication device 33. The display control unit 112 provided in the terminal device 10 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as seen by the user U1, based on the first image information received via the communication device 13, as described above.

1-2: Operation of the First Embodiment Fig. 9 is a flowchart showing the operation of the terminal device 10 according to the first embodiment. The operation of the terminal device 10 will be described below with reference to Fig. 9.

In step S1, the processing device 11 functions as the display control unit 112. The processing device 11 displays the first virtual object VO1 in the virtual space VS.

In step S2, the processing device 11 functions as an acquisition unit 111. The processing device 11 acquires the line of sight V of the user U1.

In step S3, the processing device 11 functions as the display control unit 112. The processing device 11 displays the second virtual object VO2 in an inactive state in the virtual space VS so that it does not overlap with the first virtual object VO1 as seen by the user U1. Note that the first virtual object VO1 is assumed to be active at the latest at the stage of step S3.

In step S4, the processing device 11 functions as the determination unit 113. The processing device 11 determines whether a transition has occurred from a first state in which the line of sight V of the user U1 intersects with the first virtual object VO1 and the first virtual object VO1 is active to a second state in which the line of sight V of the user U1 does not intersect with the first virtual object VO1. If it is determined that a transition has occurred from the first state to the second state, i.e., if the determination result of step S4 is positive, the processing device 11 executes the processing of step S5. If it is not determined that a transition has occurred from the first state to the second state, i.e., if the determination result of step S4 is negative, the processing device 11 executes the processing of step S4.

In step S5, the processing device 11 functions as the operation control unit 114. The processing device 11 transitions the second virtual object VO2 from inactive to active.

In step S6, the processing device 11 functions as the operation control unit 114. The processing device 11 transitions the first virtual object VO1 from active to inactive. Thereafter, the processing device 11 ends all operations shown in FIG. 9.

1-3: Advantages of the First Embodiment According to the above description, the terminal device 10 as an information processing device includes an acquisition unit 111, a display control unit 112, and an operation control unit 114. The acquisition unit 111 acquires gaze information indicating the line of sight V of the user U1. The display control unit 112 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap with each other as seen by the user U1. The operation control unit 114 transitions from a first state in which the line of sight V of the user U1 indicated by the gaze information intersects with the first virtual object VO1 and the first virtual object VO1 is active to a second state in which the line of sight V of the user U1 does not intersect with the first virtual object VO1, and then transitions the second virtual object VO2 from inactive to active.

By using the above configuration, the terminal device 10 can newly display the second virtual object VO2 so that it does not overlap with the previously displayed first virtual object VO1. Furthermore, the terminal device 10 can switch between the first virtual object VO1 and the second virtual object VO2 to activate depending on the direction of the line of sight V of the user U1.

Furthermore, according to the above description, after transitioning from the first state to the second state, the operation control unit 114 transitions the first virtual object VO1 from active to inactive.

By using the above configuration, the terminal device 10 can exclusively activate the first virtual object VO1 and the second virtual object VO2. For example, if the first virtual object VO1 is a first application and the second virtual object VO2 is a second application, the user U1 can stop the operation of the first application and start the second application by moving their line of sight V away from the first virtual object VO1. In other words, the user U1 can switch the application to be operated simply by moving their line of sight V away from the first virtual object VO1, without having to input an instruction for the second virtual object VO2.

Furthermore, according to the above description, in the first state, the display control unit 112 starts displaying the second virtual object VO2 in the virtual space VS.

By using the above configuration, the terminal device 10 can make the second virtual object VO2 appear in the first state. For example, the terminal device 10 can display the second application, which is the second virtual object VO2, while the first application, which is the first virtual object VO1, is running.

Furthermore, according to the above explanation, the operation control unit 114 transitions the second virtual object VO2 from inactive to active when, in the above-mentioned second state, the change per unit time of the user U1's line of sight V remains within the second range for a second period of time.

By using the above configuration, the terminal device 10 can transition the second virtual object VO2 from inactive to active after the user U1's line of sight V is no longer wandering and is close to being stationary.

Furthermore, according to the above description, the display control unit 112 displays the second virtual object VO2 in an area within the first distance from the first virtual object VO1.

By using the above configuration, the terminal device 10 can increase the degree to which it can attract the user U1's attention to the second virtual object VO2.

Furthermore, according to the above explanation, the display control unit 112 displays the second virtual object VO2 outside a cylinder Y in the virtual space VS, which has a central axis L1 that coincides with the line of sight V of the user U1 and a bottom surface that includes the first virtual object VO1.

By using the above configuration, the terminal device 10 can display the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as seen by the user U1.

2: Modifications The present disclosure is not limited to the above-described exemplary embodiments. Specific modifications are exemplified below. Two or more modifications selected from the following examples may be combined.

2-1: Variation 1
In the terminal device 10 according to the above embodiment, the display control unit 112 displays the second virtual object VO2 in the virtual space VS in the first state. However, the method of displaying the second virtual object VO2 is not limited to this. For example, in the second state, the display control unit 112 may display the second virtual object VO2 at a position intersecting the user U1's line of sight V after the user U1's line of sight V remains stationary for a predetermined period of time. In particular, in the second state, the display control unit 112 may display the second virtual object VO2 at a position intersecting the user U1's line of sight V on the condition that the change per unit time of the user U1's line of sight V remains within a predetermined range for a predetermined period of time. By displaying the second virtual object VO2 ahead of the user U1's line of sight V, the user U1 does not need to search for the second virtual object VO2 in the virtual space VS. The "predetermined time" is an example of the "first time." The "predetermined range" is an example of the "first range."

2-2: Variation 2
The terminal device 10 according to the above embodiment includes a display control unit 112 and an operation control unit 114. The display control unit 112 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as viewed from the user U1. The operation control unit 114 transitions the state from a first state in which the line of sight V of the user U1 indicated by the line of sight information intersects with the first virtual object VO1 and the first virtual object VO1 is active to a second state in which the line of sight V of the user U1 does not intersect with the first virtual object VO1, and then transitions the second virtual object VO2 from inactive to active. However, these operations may be performed by a device other than the terminal device 10. For example, the server 30 may include components similar to the display control unit 112 and the operation control unit 114. In particular, in a case where the communication speed between the terminal device 10 and the server 30 is faster than in a normal information processing system and the amount of data transmitted from the server 30 to the terminal device 10 is large, it is preferable for the server 30 to perform these operations. In this modified example 2, the server 30 is an example of an information processing device.

2-3: Variation 3
In the terminal device 10 according to the above embodiment, a cylinder Y is assumed as the area in which the line of sight V of the user U1 may exist while the user U1 is viewing the first virtual object VO1. However, the shape of the area is not limited to a cylinder.

Figure 10 is an explanatory diagram illustrating an example of the operation of the display control unit 112 according to Variation 3. As shown in Figure 10, the shape of the above-mentioned area may be, for example, a cone N. The cone N has a central axis L2 with the position (x0, y0, z0) of the pupil EL of the left eyeball or the pupil ER of the right eyeball of user U1 as its first endpoint C1 and the position (x1, y1, z1) of the first virtual object VO1 as its second endpoint C2. The first endpoint C1 is the apex of the cone N. Furthermore, the radius R2 of the base of the cone N is a length that allows the base to include the first virtual object VO1. Note that, for ease of explanation, in Figure 10, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of user U1 does not coincide with the position of the first endpoint C1. 10, the position (x1, y1, z1) of the first virtual object VO1 and the position of the second endpoint C2 do not match, but in practical use of the present invention, the respective positions in these pairs match.

2-4: Variation 4
In the information processing system 1 according to the above embodiment, the terminal device 10 and the MR glasses 20 are realized as separate entities. However, the method for realizing the terminal device 10 and the MR glasses 20 in the embodiment of the present invention is not limited to this. For example, the terminal device 10 and the MR glasses 20 may be realized in a single housing by the MR glasses 20 having the same functions as the terminal device 10.

2-5: Variation 5
The information processing system 1 according to the above embodiment includes MR glasses 20. However, the information processing system 1 may include any one of an HMD (Head Mounted Display) employing VR (Virtual Reality) technology, an HMD employing AR (Augmented Reality) technology, and AR glasses employing AR technology, instead of the MR glasses 20. Alternatively, the information processing system 1 may include any one of a regular smartphone and tablet equipped with an imaging device, instead of the MR glasses 20. These HMDs, AR glasses, smartphones, and tablets are examples of display devices.

2-6: Variation 6
In the information processing system 1 according to the above embodiment, the generation unit 312 included in the server 30 generates the first image information indicating the image to be displayed on the MR glasses 20. However, the device that generates the first image information is not limited to the server 30. For example, the terminal device 10 may generate the above-mentioned first image information. In this case, the information processing system 1 does not need to include the server 30 as an essential component.

3: Others (1) In the above-described embodiment, storage device 12, storage device 22, and storage device 32 are exemplified by ROM and RAM, but they may also be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The programs may also be transmitted from a network via telecommunications lines. The programs may also be transmitted from a communications network (NET) via telecommunications lines.

(2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

(3) In the above-described embodiments, input and output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. Input and output information, etc. may be overwritten, updated, or added to. Output information, etc. may be deleted. Input information, etc. may be sent to another device.

(4) In the above-described embodiments, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values (e.g., comparison with a predetermined value).

(5) The order of the processing procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an exemplary order and are not limited to the particular order presented.

(6) Each function illustrated in Figures 1 to 10 is realized by any combination of hardware and/or software. Furthermore, there are no particular limitations on the method for realizing each functional block. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., using a wired or wireless connection) and these multiple devices. A functional block may also be realized by combining software with the single device or multiple devices.

(7) The programs exemplified in the above embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), these wired and/or wireless technologies are included within the definition of transmission media.

(8) In each of the above forms, the terms "system" and "network" are used interchangeably.

(9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a specified value, or corresponding other information.

(10) In the above-described embodiments, the terminal device 10 and the server 30 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. Furthermore, in this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

(11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

(12) In the above-described embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

(13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), and ascertaining something that is considered to be a "determining." Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and other actions that are considered to be a "determining." Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

(14) In the above embodiments, when the terms "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

(15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the nouns following these articles being plural.

(16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." In addition, the term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

(17) Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, notification of specified information (e.g., notification that "X is true") is not limited to explicit notification, but may also be implicit (e.g., not notifying the specified information).

Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and explanatory and does not have any limiting meaning on the present disclosure.

1...information processing system, 10...terminal device, 11...processing device, 12...storage device, 13...communication device, 14...display, 15...input device, 16...inertial sensor, 20...MR glasses, 21...processing device, 22...storage device, 23...gaze detection device, 24...GPS device, 25...motion detection device, 26...imaging device, 27...communication device, 28...display, 30...server, 31...processing device, 32...storage device, 33...communication device, 34...display, 35...input device, 41L, 41R ...lens, 91, 92...temples, 93...bridge, 94, 95...frame, 111...acquisition unit, 112...display control unit, 113...determination unit, 114...operation control unit, 211...acquisition unit, 212...display control unit, 311...acquisition unit, 312...generation unit, C1, C2...end point, L1, L2...center axis, PR1, PR2, PR3...control program, R1, R2...radius, U1, U2, U3...user, VO...virtual object, VO1...first virtual object, VO2...second virtual object

Claims (7)

an acquisition unit that acquires line-of-sight information indicating a line of sight of a user;
a display control unit that displays the first virtual object and the second virtual object at positions in the virtual space where they do not overlap each other, as seen by the user;
an operation control unit that, after a first state in which the line of sight of the user indicated by the line of sight information intersects with the first virtual object and the first virtual object is active transitions to a second state in which the line of sight of the user does not intersect with the first virtual object, transitions the second virtual object from inactive to active even if the line of sight of the user indicated by the line of sight information does not intersect with the second virtual object;
An information processing device comprising: the first virtual object is operated by a first application;
the second virtual object is operated by a second application;
The first virtual object being active means that the first virtual object is being operated by the first application;
transitioning the second virtual object from inactive to active means switching from a state in which the second virtual object is not in operation to a state in which the second virtual object is in operation by the second application;
the operation control unit transitions the first virtual object from active to inactive after the first state transitions to the second state;
The information processing device according to claim 1 . The information processing device according to claim 1 or 2, wherein the display control unit starts displaying the second virtual object in the virtual space in the first state. 3. The information processing device according to claim 1, wherein the display control unit starts displaying the second virtual object at a position that intersects with the user's line of sight, on the condition that, in the second state, a change per unit time of the user's line of sight falls within a first range for a first period of time. 3. The information processing device according to claim 1, wherein the operation control unit transitions the second virtual object from inactive to active on the condition that a change per unit time of the user's line of sight falls within a second range for a second period of time in the second state . The information processing device according to claim 1 , wherein the display control unit causes the second virtual object to be displayed in an area within a first distance from the first virtual object. 3 . The information processing device according to claim 1 , wherein the display control unit displays the second virtual object outside a cylinder or cone in the virtual space, the cylinder or cone having a central axis that coincides with the user's line of sight and a bottom surface that includes the first virtual object.