JP7829680B2 - display device - Google Patents

Description

This invention relates to a display device.

In recent years, technological developments related to virtual spaces where virtual objects are placed have been progressing. Display devices that provide virtual spaces to users display a portion of the virtual space visible from the user's viewpoint. If the virtual object the user wishes to use is not present within the display area of the display device, technologies have been developed to guide the user's gaze to the location of that virtual object.

Patent Document 1 discloses a program that displays a controller object in a virtual space that indicates the position of a game controller in real space. This program displays a directional guide object in the virtual space that indicates the direction of the controller object when the controller object is not within the imaging range of a virtual stereo camera.

Japanese Patent Publication No. 2015-232783

However, placing a directional guide object in the center of the display area of a device obstructs the user's field of vision. As a result, conventional technologies suffer from problems such as reduced visibility in the virtual space.

This disclosure aims to provide a display device that guides the user's gaze towards virtual objects located outside the display area of the display device, while suppressing a decrease in the visibility of the virtual space.

A first preferred embodiment of the present disclosure is a display device that displays a portion of an augmented reality space or mixed reality space, in which virtual objects arranged in a virtual space are superimposed on a real space, in a display area, and comprises: an image generation unit that generates a guidance image that guides the user's gaze to the position of the virtual object based on the position of the virtual object arranged in the virtual space when the virtual object is not displayed in the display area; and a display control unit that displays the guidance image at a position adjacent to the outer edge of the display area.

According to the present invention, compared to the prior art, it is possible to guide the user's gaze towards virtual objects located outside the display area of the display device without reducing the visibility of the virtual space.

A block diagram showing the overall configuration of information processing system 1. A schematic diagram of a virtual object VO and a guide image GP that indicates the location of the virtual object VO. A block diagram showing an example configuration of XR Glasses 10-k. A diagram showing the mutually orthogonal X, Y, and Z axes assumed in a virtual space. An example of the position of the display area AR in the rθφ coordinate system. Examples of display positions for the GP (Graphics Program) guide image. Examples of display positions for the GP (Graphics Program) guide image. A block diagram showing an example configuration of the management server 50. A flowchart illustrating the display process. A flowchart illustrating the display process. A schematic diagram of a virtual object VO and a guide image GP that directs the user's gaze to the location of the virtual object VO. An example of the appearance of a GP (General Purpose) guide image. An example of the appearance of a GP (General Purpose) guide image.

1: First Embodiment Hereinafter, the information processing system 1 according to the first embodiment will be described with reference to Figures 1 to 8.

1-1: Configuration of the First Embodiment 1-1-1: Overall Configuration Figure 1 is a block diagram showing the overall configuration of the information processing system 1. As shown in Figure 1, the information processing system 1 comprises XR glasses 10-1, 10-2, ... 10-k, ... 10-j, and a management server 50 for managing data related to augmented reality space or mixed reality space. j is any integer greater than or equal to 1. k is any integer greater than or equal to 1 and less than or equal to j. In this embodiment, the XR glasses 10-1, 10-2, ... 10-k, ... 10-j have the same configuration. However, the information processing system 1 may include XR glasses with different configurations.

In the information processing system 1, the management server 50 and the XR glasses 10-k are connected to each other via a communication network NET, enabling communication between them. In Figure 1, user U[k] uses the XR glasses 10-k. The same applies to users U[1], U[2], ... U[k-1], U[k+1], ... U[j].

The XR Glasses 10-k are so-called through-the-glasses display glasses and are an example of a display device that displays an augmented reality space or a mixed reality space that includes virtual objects. Augmented reality refers to a virtual space in which virtual objects can be superimposed on real objects in the real world. Mixed reality refers to a virtual space in which real objects and virtual objects interact with each other. In augmented reality, virtual objects are simply displayed superimposed on real objects. In contrast, in mixed reality, the interaction between virtual and real objects can be represented by using a physical model of the virtual object. For example, if a user throws a virtual ball object at a wall, which is a real object, the ball bouncing off the wall is represented.

User U[k] perceives virtual objects while viewing the real-world view through the XR glasses 10-k. The virtual objects are placed in the virtual space, corresponding to their positions in the real space. By using the XR glasses 10-k, user U[k] perceives an augmented reality or mixed reality space, which is a combination of the real and virtual spaces.

Figure 2 is a schematic diagram of a virtual object VO existing in an augmented reality or mixed reality space, which is recognized by user U[k] using XR glasses 10-k, and a guidance image GP that guides the user's gaze to the location of the virtual object VO.

In the following explanation, the area visible to user U[k] through the XR glasses 10-k is referred to as the display area AR of the XR glasses 10-k. As an example, as shown in Figure 2, the display area AR of the XR glasses 10-k is assumed to be a rectangle with points A to D as its four vertices. Furthermore, the virtual object VO is located outside the range of the display area AR, and the centroid F of the virtual object VO is located in the direction of vertex B when viewed from the center E of the display area AR. In this case, the guidance image GP has an L-shape that includes a first rectangle P1 with vertex B as its endpoint and part of side AB as its long side, and a second rectangle P2 with vertex B as its endpoint and part of side BD as its long side. By displaying the L-shaped guidance image GP, which is tangent to the outer edge of the display area AR, at a position including vertex B, user U[k] can recognize that the virtual object VO, which is not displayed within the range of the display area AR, exists in the direction of vertex B when viewed from the center E of the display area AR.

Details regarding the changes in the position and shape of the guide image GP in accordance with the position of the virtual object VO will be described later.

1-1-2: Configuration of XR Glasses Figure 3 is a block diagram showing an example configuration of XR glasses 10-k. XR glasses 10-k comprises a processing unit 11, a storage device 12, an imaging device 13, a sound collection device 14, a detection device 15, a communication device 16, and a projection device 17. Each element of XR glasses 10-k is interconnected by one or more buses for communicating information.

The processing unit 11 is a processor that controls the entire XR glasses 10-k. The processing unit 11 is configured using, for example, one or more chips. The processing unit 11 is also configured using a central processing unit (CPU) that includes, for example, interfaces with peripheral devices, arithmetic units, and registers. Some or all of the functions of the processing unit 11 may be implemented by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The processing unit 11 executes various processes in parallel or sequentially.

The storage device 12 is a recording medium that can be read from and written to by the processing device 11. The storage device 12 also stores multiple programs, including the control program PR1 executed by the processing device 11. The storage device 12 functions as a work area for the processing device 11. Furthermore, the storage device 12 stores data used by the image generation unit 111 (described later) to render the virtual object VO, the guidance image GP, and the virtual space in which the virtual object VO resides.

The imaging device 13 outputs an image obtained by imaging the external environment. The imaging device 13 also includes, for example, a lens, an image sensor, an amplifier, and an AD converter. Light focused through the lens is converted into an analog imaging signal by the image sensor. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified analog imaging signal into digital imaging information. The converted imaging information is output to the processing device 11. The image output to the processing device 11 is then output to the management server 50 via the communication device 16.

The XR Glasses 10-k have a shape similar to that of ordinary eyeglasses. The XR Glasses 10-k have a left lens, a right lens, and a frame that supports the left and right lenses. The frame has a bridge between the left and right lenses and a pair of temples that rest on the left and right ears of the wearer. The imaging device 13 is mounted on the bridge. Therefore, the imaging device 13 images the external world in the direction that the user U[k]'s face is facing.

The sound-collecting device 14 is a microphone equipped with a sound-collecting section that collects sounds from the surroundings of the XR glasses 10-k and converts the collected sounds into electrical signals. The sound-collecting section can be any configuration that collects sound, for example, a windproof structure. The sound-collecting device 14 collects the voice of user U[k] as ambient sound.

The detection device 15 detects the state of the XR glasses 10-k. The detection device 15 includes, for example, inertial sensors such as an acceleration sensor for detecting acceleration and a gyroscope sensor for detecting angular acceleration, a geomagnetic sensor for detecting direction, and a positioning device such as a GPS device for detecting the position of the XR glasses 10-k in real space. The acceleration sensor detects acceleration along the orthogonal X, Y, and Z axes. The gyroscope sensor detects angular acceleration with the X, Y, and Z axes as the central axes of rotation. Based on the output information from the gyroscope sensor, the detection device 15 can generate attitude information regarding the orientation of the XR glasses 10-k. The motion information includes acceleration information indicating the acceleration of each of the three axes and angular acceleration information indicating the angular acceleration of each of the three axes. The detection device 15 also outputs attitude information regarding the orientation of the XR glasses 10-k, motion information regarding the movement of the XR glasses 10-k, direction information regarding the direction of the XR glasses 10-k, and position information regarding the position of the XR glasses 10-k to the processing device 11.

The communication device 16 is hardware acting as a transmitting and receiving device for communicating with other devices. The communication device 16 is also referred to as, for example, a network device, network controller, network card, and communication module. The communication device 16 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connections include products compliant with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products compliant with wireless LAN and Bluetooth®.

The projection device 17 is a device that displays images. The projection device 17 displays various images under the control of the processing device 11. The left temple of the XR glasses 10-k is provided with a display panel for the left eye and an optical element that guides the light emitted from the left eye display panel to the lens. The half mirror provided on the left lens transmits light from the outside world and guides it to the left eye, and after reflecting the light guided by the optical element, it is incident on the left eye. The right temple and right lens are configured similarly. The projection device 17 includes a left lens, a right lens, a display panel for the left eye, a display panel for the right eye, an optical element for the left eye, and an optical element for the right eye. In the above configuration, the user U[k] can observe the image displayed on the display panel in a see-through state superimposed on the outside world. Furthermore, the XR glasses 10-k displays the left eye image on the left eye display panel and the right eye image on the right eye display panel, among the binocular images with parallax. Therefore, the XR glasses 10-k cause the user U[k] to perceive the displayed image as if it had depth and three-dimensionality.

In the above configuration, the processing unit 11 reads the control program PR1 from the storage device 12 and executes the read control program PR1, thereby functioning as the image generation unit 111, the display control unit 112, the detection unit 113, and the speech recognition unit 114.

The image generation unit 111 generates an image showing the virtual space where the virtual object VO is placed. Furthermore, if the virtual object VO is not displayed in the display area AR, the image generation unit 111 generates a guidance image GP that guides the user's gaze to the location of the virtual object VO, based on the position of the virtual object VO in the virtual space. Both the image showing the virtual object VO and the guidance image GP may be two-dimensional or three-dimensional images. The image generation unit 111 generates the guidance image GP in response to user U[k] input to the XR glasses 10-k.

The display control unit 112 causes the projection device 17 to display an image showing the virtual space where the virtual object VO is placed. As a result, a part of the augmented reality space or mixed reality space, in which the real space and the virtual space displayed by the display control unit 112 are superimposed, is displayed in the display area of the XR glasses 10-k.

Figure 4 shows the mutually orthogonal X, Y, and Z axes assumed in the virtual space. The X, Y, and Z axes are common to all figures exemplified in the following explanation. As exemplified in Figure 2, one direction along the X axis from an arbitrary point is denoted as the X1 direction, and the direction opposite to the X1 direction is denoted as the X2 direction. The X-axis direction includes both the X1 and X2 directions. Similarly, the mutually opposite directions along the Y axis from an arbitrary point are denoted as the Y1 and Y2 directions. The Y-axis direction includes both the Y1 and Y2 directions. In addition, the mutually opposite directions along the Z axis from an arbitrary point are denoted as the Z1 and Z2 directions. The Z-axis direction includes both the Z1 and Z2 directions. In the XYZ coordinate system, the XR glass 10-k is located at the origin O. Furthermore, the depth direction of the XR glasses 10-k is defined as the Y1 direction, and the direction towards the user U[k] facing the XR glasses 10-k is defined as the Y2 direction. Also, the direction to the right of the user U[k] facing the XR glasses 10-k is defined as the X1 direction, and the direction to the left of the user U[k] facing the XR glasses 10-k is defined as the Z1 direction, and the direction downwards is defined as the Z2 direction.

Furthermore, in this virtual space, we assume a three-dimensional polar coordinate system, the rθφ coordinate system, superimposed on the XYZ coordinate system. In the rθφ coordinate system, the position of any point P is given by the distance r from the origin O and two angular parameters θ and φ. θ is the angle between the Z1 direction and the line OP, where 0 ≤ θ < π. If the foot of the perpendicular from point P to the XY plane is point Q, then φ is the angle between the X1 direction and the line OQ, where 0 ≤ φ < 2π.

Figure 5 shows an example of the position of the display area AR in the rθφ coordinate system. In the display area AR illustrated in Figure 5, the position of vertex A is (r,θ,φ) = ( _r1 , _θ1 , _φ2 ). The position of vertex B is (r,θ,φ) = ( _r1 , _θ1 , _φ1 ). The position of vertex C is (r,θ,φ) = ( _r1 , _θ2 , _φ2 ). The position of vertex D is (r,θ,φ) = ( _r1 , _θ2 , _φ1 ). The display area AR is a part of a sphere centered at the origin O with radius _r1 , and these vertices A to D are considered the four vertices.

Furthermore, in the virtual space VS shown in Figure 5, let α be the region where 0 ≤ θ < _θ₁ and _φ₂ ≤ φ < _2π . Let β be the region where 0 ≤ θ < _θ₁ and φ₁ ≤ φ < _φ₂ . Let γ be the region where 0 ≤ θ < _θ₁ and 0 ≤ φ < _φ₁ . Let δ be the region where _θ₁ ≤ θ < _θ₂ and _φ₂ ≤ φ < 2π. Let ε be the region where _θ₁ ≤ θ < _θ₂ and _φ₁ ≤ φ < _φ₂ . Let ζ be the region where _θ₁ ≤ θ < _θ₂ and 0 ≤ φ < _φ₁ . Let η be the region where _θ₂ ≤ θ < π and _φ₂ ≤ φ < 2π. Let ι be the region where _θ₂ ≤ θ < π and _φ₁ ≤ φ < _φ₂ . Let κ be the region where θ ₂ ≤ θ < π and 0 ≤ φ < φ ₁ .

Returning to Figure 3, the display control unit 112 displays the guidance image GP at a position adjacent to the outer edge of the display area AR of the XR glasses 10-k. Figures 6A and 6B are examples of the display position of the guidance image GP. The operation of the display control unit 112 will be explained below with reference to Figures 6A and 6B, as well as Figure 2 described above.

As shown in Figure 6A, when the centroid F of the virtual object VO is located within region β, the display control unit 112 displays a rectangle as a guide image GP within the display region AR, which is tangent to side AB and whose longer side is in the direction of side AB. This rectangle corresponds to the first rectangle P1 of the guide image GP shown in Figure 2. If the foot of the perpendicular from the centroid F of the virtual object VO to side _AB is point FAB, then the center of the longer side of the first rectangle P1 contained within side AB is point _FAB . Furthermore, the length L1 in the direction of the longer side of the first rectangle P1 changes depending on the distance from the origin O to the centroid F. For example, the longer the distance from the origin O to the centroid F, the longer L1 may be. Alternatively, the shorter the distance from the origin O to the centroid F, the longer L1 may be. As an example, in an augmented reality or mixed reality space where a virtual space VS and a real space are superimposed, if the distance from the origin O to the centroid F is less than 1 m, the display control unit 112 sets L1 = _l11 . If the distance from the origin O to the centroid F is 1 m or more and less than 3 m, the display control unit 112 sets L1 = _l12 . If the distance from the origin O to the centroid F is 3 m or more, the display control unit 112 sets L1 = _l13 . Here, _l11 < _l12 < _l13 is also possible, and _l11 > _l12 > _l13 is also possible. Furthermore, if the position of point _FAB is too close to vertex A, and the distance between point _FAB and vertex A becomes shorter than half the length L1 in the long side direction of the first rectangle P1, the display control unit 112 displays the first rectangle P1 at a position tangent to vertex A while maintaining the length L1 in the long side direction. Similarly, if the position of point _FAB is too close to vertex B, and the distance between point _FAB and vertex B becomes shorter than half the length L1 of the long side of the first rectangle P1, the display control unit 112 will display the first rectangle P1 at a position tangent to vertex B while maintaining the length L1 of the long side.

Furthermore, although not shown in the diagram, if the centroid F of the virtual object VO is located within region ι, the display control unit 112 similarly displays a rectangle as a guide image GP within the display region AR, which is tangent to side CD and whose longer side is aligned with side CD.

As shown in Figure 6B, when the centroid F of the virtual object VO is located within region ζ, the display control unit 112 displays a rectangle as a guide image GP within the display region AR, which is tangent to side BD and whose longer side is in the direction of side BD. This rectangle corresponds to the second rectangle P2 of the guide image GP shown in Figure 2. If the foot of the perpendicular from the centroid F of the virtual object VO to side BD is point _FBD , then the center of the longer side of the second rectangle P2 contained within side BD is point _FBD . Furthermore, the length L2 in the direction of the longer side of the second rectangle P2 changes depending on the distance from the origin O to the centroid F. For example, the longer the distance from the origin O to the centroid F, the longer L2 may be. Alternatively, the shorter the distance from the origin O to the centroid F, the longer L2 may be. As an example, in an augmented reality or mixed reality space where a virtual space VS and a real space are superimposed, if the distance from the origin O to the centroid F is less than 1 m, the display control unit 112 sets L2 = _l21 . If the distance from the origin O to the centroid F is 1 m or more and less than 3 m, the display control unit 112 sets L2 = _l22 . If the distance from the origin O to the centroid F is 3 m or more, the display control unit 112 sets L2 = _l23 . Here, _l21 < _l22 < _l23 is also acceptable, and _l21 > _l22 > _l23 is also acceptable. Furthermore, if the position of point _FBD is too close to vertex B, and the distance between point _FBD and vertex B becomes shorter than half the length L2 of the long side of the second rectangle P2, the display control unit 112 displays the second rectangle P2 at a position tangent to vertex B while maintaining the length L2 of the long side. Similarly, if the position of point _FBD is too close to vertex D, and the distance between point _FBD and vertex D becomes shorter than half the length L2 of the long side of the second rectangle P2, the display control unit 112 displays the second rectangle P2 at a position tangent to vertex D while maintaining the length L2 of the long side.

Furthermore, although not shown in the diagram, if the centroid F of the virtual object VO is located within region δ, the display control unit 112 similarly displays a rectangle as a guide image GP within the display region AR, which is tangent to side AC and whose longer side is aligned with side AC.

As shown in Figure 2, when the centroid F of the virtual object VO is located within region γ, the display control unit 112 displays a first rectangle P1 as a guide image GP within the display region AR. This rectangle has vertex B as its endpoint, is tangent to side AB, and its longer side is in the direction of side AB. The display control unit 112 also displays a second rectangle P2 as a guide image GP within the display region AR. This rectangle has vertex B as its endpoint, is tangent to side BD, and its longer side is in the direction of side BD. In other words, when the centroid F of the virtual object VO is located within region γ, the display control unit 112 displays an L-shaped guide image GP in which the first rectangle P1 and the second rectangle P2 are superimposed. The length L1 of the longer side of the first rectangle P1 varies depending on the distance from the origin O to the centroid F. For example, the longer the distance from the origin O to the centroid F, the longer L1 may be. Alternatively, L1 may be longer the shorter the distance from the origin O to the centroid F. For example, in an augmented reality space or mixed reality space where a virtual space VS and a real space are superimposed, if the distance from the origin O to the centroid F is less than 1 m, the display control unit 112 sets L1 = _l11 . If the distance from the origin O to the centroid F is 1 m or more and less than 3 m, the display control unit 112 sets L1 = _l12 . If the distance from the origin O to the centroid F is 3 m or more, the display control unit 112 sets L1 = _l13 . Here, _l11 < _l12 < _l13 may also be true, and _l11 > _l12 > _l13 may also be true. Similarly, the length L2 in the direction of the longer side of the second rectangle P2 changes depending on the distance from the origin O to the centroid F. For example, if the distance from the origin O to the centroid F is long, L2 may be long. Alternatively, L2 may be longer the shorter the distance from the origin O to the centroid F. For example, in an augmented reality space or mixed reality space where a virtual space VS and a real space are superimposed, if the distance from the origin O to the centroid F is less than 1 m, the display control unit 112 sets L2 = _l21 . If the distance from the origin O to the centroid F is 1 m or more and less than 3 m, the display control unit 112 sets L2 = _l22 . If the distance from the origin O to the centroid F is 3 m or more, the display control unit 112 sets L2 = _l23 . Here, _l21 < _l22 < _l23 is also acceptable, and _l21 > _l22 > _l23 is also acceptable.

Furthermore, although not shown in the diagram, if the centroid F of the virtual object VO is located within region α, the display control unit 112 will similarly display an L-shaped guide image GP including vertex A as the guide image GP. Similarly, if the centroid F of the virtual object VO is located within region η, the display control unit 112 will similarly display an L-shaped guide image GP including vertex C as the guide image GP. Similarly, if the centroid F of the virtual object VO is located within region κ, the display control unit 112 will similarly display an L-shaped guide image GP including vertex D as the guide image GP.

As a result, the XR glasses 10-k can guide the user U[k]'s gaze towards a virtual object VO located outside the display area AR without reducing the visibility of the virtual space VS, compared to conventional technology. Furthermore, user U[k] can perceive the distance from the XR glasses 10-k to the virtual object VO.

In the above example, the display control unit 112 changed the length of the long side of the first rectangle P1 and the second rectangle P2 included in the guide image GP according to the distance from the origin O to the centroid F. However, the display control unit 112 may change the shape of the guide image GP in other ways. For example, the display control unit 112 may change the length of the short side of the first rectangle P1 and the second rectangle P2 included in the guide image GP according to the distance from the origin O to the centroid F. Alternatively, the outline of the guide image GP may be changed to either a rectangle or a triangle, with one side coinciding with any of the four sides of the display area AR, according to the distance from the origin O to the centroid F.

Furthermore, the display control unit 112 may change at least one of the color and brightness of the guidance image GP according to the distance from the origin O to the centroid F. For example, in an augmented reality space or mixed reality space in which a virtual space VS and a real space are superimposed, if the distance from the origin O to the centroid F is less than 1 m, the display control unit 112 sets the color of the guidance image GP to red. If the distance from the origin O to the centroid F is 1 m or more but less than 3 m, the display control unit 112 sets the color of the guidance image GP to yellow. If the distance from the origin O to the centroid F is 3 m or more, the display control unit 112 sets the color of the guidance image GP to blue. Alternatively, if the distance from the origin O to the centroid F is less than 1 m, the display control unit 112 sets the brightness of the guidance image GP to xu _1. If the distance from the origin O to the centroid F is 1 m or more but less than 3 m, the display control unit 112 sets the brightness of the guidance image GP to xu ₂ . If the distance from the origin O to the centroid F is 3m or more, the display control unit 112 sets the brightness of the guide image GP to xu _3. Here, xu ₁ < xu ₂ < xu ₃ may also be the case, and xu ₁ > xu ₂ > xu ₃ may also be the case.

As a result, user U[k] can perceive the distance from the XR glasses 10-k to the virtual object VO.

Furthermore, the display control unit 112 may change the appearance of the guidance image GP in the display area AR in accordance with changes in the real space over which the virtual space VS is superimposed. For example, if the real space is a bright space, the display control unit 112 may make the guidance image GP a low-brightness image, while if the real space is a dark space, the display control unit 112 may make the guidance image GP a high-brightness image.

As a result, the contrast between the real world and the guidance image GP in the AR display area is increased, maintaining high visibility of the guidance image GP for the user U[k].

Furthermore, the display control unit 112 changes the appearance of the guidance image GP in real time by using the above method in response to the movement of the virtual object VO.

As a result, user U[k] can recognize the change in position associated with the movement of the virtual object VO in real time.

Returning to the explanation in Figure 3, the detection unit 113 detects the body movements of user U[k]. Specifically, the detection unit 113 acquires user U[k]'s gestures based on the image of user U[k] acquired from the imaging device 13. For example, the detection unit 113 acquires specific body movements as gestures of user U[k], such as shaking their head or waving their hands to search for the virtual object VO. The image generation unit 111 takes the specific body movements of user U[k] detected by the detection unit 113 as input to user U[k], and generates a guidance image GP in response to this input.

As a result, the XR glasses 10-k can display a guidance image GP in the AR display area of the XR glasses 10-k when the user U[k] makes a specific gesture. For example, when the user U[k] shakes their head to search for a virtual object VO in an augmented reality or mixed reality space, the XR glasses 10-k can display a guidance image GP in the AR display area.

The speech recognition unit 114 recognizes the voice of user U[k] captured by the sound collection device 14. Specifically, when user U[k] pronounces the name of a virtual object VO, the speech recognition unit 114 recognizes that name as the voice of user U[k]. For example, if the virtual object VO is a dog, the speech recognition unit 114 recognizes the name of the dog pronounced by user U[k]. The image generation unit 111 takes the recognition result of the voice in which user U[k] pronounced the name of the virtual object VO, as recognized by the speech recognition unit 114, as input to user U[k], and generates a guidance image GP in accordance with that input.

As a result, the XR glasses 10-k can display a guidance image GP when user U[k] pronounces a specific name. For example, if user U[k] pronounces the name of a dog in order to find a virtual object VO in an augmented reality or mixed reality space, the XR glasses 10-k can display a guidance image GP in the display area AR.

1-1-3: Configuration of the Management Server Figure 7 is a block diagram showing an example configuration of the management server 50. The management server 50 comprises a processing unit 51, a storage device 52, an input device 53, a communication device 54, and a display 55. Each element of the management server 50 is interconnected by one or more buses for communicating information.

The processing unit 51 is a processor that controls the entire management server 50. The processing unit 51 is configured, for example, using one or more chips. Furthermore, the processing unit 51 is configured using a central processing unit (CPU), which includes, for example, interfaces with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing unit 51 may be implemented by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The processing unit 51 executes various processes in parallel or sequentially.

The storage device 52 is a recording medium that can be read from and written to by the processing device 51. The storage device 52 also stores multiple programs, including the control program PR2 executed by the processing device 51. The storage device 52 functions as a work area for the processing device 51. Furthermore, the storage device 52 stores data used by the XR glasses 10-k to render virtual objects VO and the virtual space VS in which the virtual objects VO reside.

The input device 53 receives operations from the administrator of the management server 50. For example, the input device 53 is configured to include a keyboard, touchpad, touch panel, or pointing device such as a mouse. Here, if the input device 53 is configured to include a touch panel, it may also function as the display 55.

The communication device 54 is hardware acting as a transmitting and receiving device for communicating with other devices. The communication device 54 is also referred to as, for example, a network device, network controller, network card, or communication module. The communication device 54 may be equipped with a connector for wired connection. The communication device 54 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connection include products compliant with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products compliant with wireless LAN and Bluetooth®.

The display 55 is a device that displays images. The display 55 displays various images under the control of the processing unit 51.

In the above configuration, the processing unit 51 functions as a communication control unit 511 and an acquisition unit 512 by reading the control program PR2 from the storage device 52 and executing the read control program PR2.

The communication control unit 511 causes the communication device 54 to receive from the XR glasses 10-k the following information: attitude information regarding the orientation of the XR glasses 10-k, motion information regarding the movement of the XR glasses 10-k, orientation information regarding the direction of the XR glasses 10-k, and position information regarding the position of the XR glasses 10-k.

Furthermore, the communication control unit 511 uses the communication device 54 to transmit to the XR glasses 10-k the data acquired by the acquisition unit 512 (described later) that the XR glasses 10-k uses to render the virtual object VO and the augmented reality space or mixed reality space containing the virtual object VO.

The acquisition unit 512 acquires data from the storage device 52 that the XR glasses 10-k uses to render the virtual object VO and the augmented reality space or mixed reality space containing the virtual object VO. Specifically, the acquisition unit 512 acquires the above data according to the orientation information regarding the attitude of the XR glasses 10-k, the motion information regarding the movement of the XR glasses 10-k, the orientation information regarding the direction of the XR glasses 10-k, and the position information regarding the location of the XR glasses 10-k, which the communication control unit 511 has received from the communication device 54.

1-2: Operation of the Embodiment The following describes the display process by which the XR glasses 10-k displays the guidance image GP. Figure 8 is a flowchart showing the contents of the display process by which the XR glasses 10-k according to the first embodiment displays the guidance image GP in the display area AR.

In step S10, if the virtual object VO is not displayed in the display area AR, the processing unit 11 generates a guidance image GP that guides the user's gaze to the position of the virtual object VO, based on the position of the virtual object VO placed in the virtual space VS, in response to the input of user U[k].

In step S11, the processing unit 11 displays the guide image GP at a position adjacent to the outer edge of the display area AR.

In the above display process, the processing unit 11 functions as an image generation unit 111 in step S10. Furthermore, the processing unit 11 functions as a display control unit 112 in step S11.

1-3: Effects of the Embodiment According to the above description, the XR glasses 10-k display a part of an augmented reality space or mixed reality space, in which a virtual object VO placed in a virtual space VS is superimposed on the real space, in the display area AR. The XR glasses 10-k comprises an image generation unit 111 and a display control unit 112. When the virtual object VO is not displayed in the display area AR, the image generation unit 111 generates a guidance image GP that guides the user's gaze to the position of the virtual object VO, based on the position of the virtual object VO placed in the virtual space VS. The display control unit 112 displays the guidance image GP at a position adjacent to the outer edge of the display area AR.

Since the XR glasses 10-k have the above configuration, compared to the conventional technology, it is possible to guide the user U[k]'s gaze in the direction of a virtual object VO located outside the display area AR without reducing the visibility of the virtual space VS. Here, it is preferable that the image generation unit 111 generates a guidance image GP in response to the input of user U[k] to the XR glasses 10A-k. Rather than generating the guidance image GP at all times, generating the guidance image GP in response to the input of user U[k] reduces the load on the processing unit 11.

Furthermore, since the XR glasses 10-k generate a guidance image GP in response to user U[k] input, the guidance image GP can be displayed only when necessary for user U[k] to guide their gaze towards the virtual object VO.

Furthermore, the XR glasses 10-k are further equipped with a detection unit 113. The input from user U[k] described above is a specific body movement of user U[k] detected by the detection unit 113.

Since the XR glasses 10-k have the above configuration, it is possible to display the guidance image GP in the display area AR when the user U[k] makes a specific gesture. For example, when the user U[k] shakes their head to search for a virtual object VO in an augmented reality space or mixed reality space, the XR glasses 10-k can display the guidance image GP in the display area AR.

The XR glasses 10-k also include a speech recognition unit 114. The input from user U[k] is the voice representing the name of the virtual object VO, as recognized by the speech recognition unit 114 and pronounced by user U[k].

Since the XR glasses 10-k have the above configuration, when user U[k] pronounces a specific name, it becomes possible to display a guidance image GP in the display area AR. For example, when user U[k] pronounces the name of a virtual object VO in order to find the virtual object VO in an augmented reality space or mixed reality space, the XR glasses 10-k can display a guidance image GP in the display area AR.

Furthermore, in the XR glasses 10-k, the display control unit 112 changes at least one of the color and brightness of the guidance image GP according to the distance from the XR glasses 10-k to the virtual object VO.

Since the XR glasses 10-k have the above configuration, user U[k] can perceive the distance from the XR glasses 10-k to the virtual object VO.

Furthermore, in the XR glasses 10-k, the display control unit 112 changes the shape of the guidance image GP according to the distance from the XR glasses 10-k to the virtual object VO.

Since the XR glasses 10-k have the above configuration, user U[k] can perceive the distance from the XR glasses 10-k to the virtual object VO.

Furthermore, in the XR glasses 10-k, the display control unit 112 changes the shape of the guide image GP in accordance with the movement of the virtual object VO.

Since the XR glasses 10-k have the above configuration, the user U[k] can recognize the change in position accompanying the movement of the virtual object VO in real time.

Furthermore, in the XR glasses 10-k, the display control unit 112 changes the appearance of the guidance image GP in response to changes in the real space displayed on the XR glasses 10-k.

Since the XR Glasses 10-k has the above configuration, it can maintain high visibility of the guidance image GP for the user U[k].

2: Second Embodiment The information processing system 1A according to the second embodiment will be described below. In the following description, for the sake of simplicity, the same reference numerals will be used for components of the information processing system 1A that are the same as those of the information processing system 1, and the explanation of their functions will be omitted in principle.

2-1: Configuration of the Second Embodiment 2-1-1: Overall Configuration The overall configuration of the information processing system 1A according to this embodiment is the same as the overall configuration of the information processing system 1 shown in Figure 1, so its illustration is omitted.

Information processing system 1A is equipped with XR glasses 10A-1, 10A-2, ... 10A-k, ... 10A-j instead of the XR glasses 10-1, 10-2, ... 10-k, ... 10-j provided in information processing system 1.

2-1-2: Terminal Device Configuration The XR Glasses 10A-k is equipped with a processing unit 11A instead of the processing unit 11 and a storage device 12A instead of the storage device 12 that are present in the XR Glasses 10-k. Note that the configuration of the XR Glasses 10A-k is the same as the configuration of the XR Glasses 10-k shown in Figure 3, so its illustration is omitted.

The storage device 12A stores control program PR1A instead of control program PR1 stored in storage device 12. The processing device 11A reads control program PR1A from storage device 12A and executes the read control program PR1A, thereby functioning as an image generation unit 111A and a display control unit 112A, in addition to the same detection unit 113 and voice recognition unit 114 as the XR glasses 10-k according to the first embodiment.

The image generation unit 111A generates an image showing the virtual space VS where the virtual object VO is placed. Furthermore, if the virtual object VO is not displayed in the display area AR, the image generation unit 111A generates a guidance image GP that guides the user's gaze to the location of the virtual object VO, based on the position of the virtual object VO in the virtual space VS.

In the first embodiment, the image generation unit 111 generated a guidance image GP in response to user U[k] input to the XR glasses 10-k. On the other hand, the image generation unit 111A in this embodiment generates a guidance image GP independently of user U[k] input to the XR glasses 10-k.

The display control unit 112A displays an image in the display area AR showing the virtual space VS where the virtual object VO is located, in response to input from user U[k] to the XR glasses 10-k. Here, "input from user U[k]" may be a specific body movement of user U[k] detected by the detection unit 113, similar to the input from user U[k] in the first embodiment. Alternatively, "input from user U[k]" may be the recognition result of the voice recognition unit 114 recognizing the voice in which user U[k] pronounces the name of the virtual object VO, similar to the input from user U[k] in the first embodiment.

In other words, in the first embodiment, the image generation unit 111 generated a guide image GP in response to user U[k] input, and the display control unit 112 displayed the guide image GP in the display area AR. On the other hand, in this embodiment, the image generation unit 111A generates a guide image GP regardless of whether or not user U[k] input is present, and the display control unit 112A displays the guide image GP in the display area AR in response to user U[k] input.

2-2: Operation of the Embodiment The following describes the display process by which the XR glasses 10A-k displays the guidance image GP. Figure 9 is a flowchart showing the contents of the display process by which the XR glasses 10A-k according to the second embodiment displays the guidance image GP in the display area AR.

In step S20, if the virtual object VO is not displayed in the display area AR, the processing unit 11A generates a guidance image GP that guides the user's gaze to the position of the virtual object VO, based on the position of the virtual object VO placed in the virtual space VS.

In step S21, the processing unit 11 displays the guidance image GP at a position adjacent to the outer edge of the display area AR, in response to the input of user U[k].

In the above display process, the processing unit 11A functions as an image generation unit 111A in step S20. Furthermore, the processing unit 11A functions as a display control unit 112A in step S21.

2-3: Effects of the Embodiment According to the above description, the XR glasses 10A-k display a part of an augmented reality space or mixed reality space, in which a virtual object VO placed in a virtual space VS is superimposed on the real space, in the display area AR. The XR glasses 10A-k comprises an image generation unit 111A and a display control unit 112A. When the virtual object VO is not displayed in the display area AR, the image generation unit 111A generates a guidance image GP that guides the user's gaze to the position of the virtual object VO, based on the position of the virtual object VO placed in the virtual space VS. The display control unit 112A causes the guidance image GP to be displayed at a position adjacent to the outer edge of the display area AR. The display control unit 112A displays the guidance image GP in response to user U[k] input to the XR glasses 10A-k.

Since the XR glasses 10A-k have the above configuration, compared to conventional technology, it is possible to guide the user U[k]'s gaze towards the virtual object VO located outside the display area AR without reducing the visibility of the virtual space VS.

Furthermore, since the XR glasses 10A-k display a guidance image GP in response to user U[k] input, the guidance image GP can be displayed only when necessary for user U[k] to guide their gaze towards the virtual object VO.

3. Modifications This disclosure is not limited to the embodiments illustrated above. Specific examples of modifications are given below. Two or more embodiments may be arbitrarily selected from the following examples and combined.

3-1: Variation 1
In the first embodiment, the display control unit 112 changed one or more of the lengths of the long sides, the short sides, and the outlines of the first rectangle P1 and the second rectangle P2 included in the guide image GP according to the distance from the origin O to the centroid F. However, the display control unit 112 may change one or more of the lengths of the long sides, the short sides, and the outlines of the first rectangle P1 and the second rectangle P2 included in the guide image GP based on other parameters.

As described above, the display area AR is a part of a sphere centered at the origin O with radius _r1 . The display control unit 112 may, for example, change one or more of the long side length, short side length, outline, color, and brightness of the first rectangle P1 and the second rectangle P2 included in the guide image GP based on the distance between the point where the line connecting the origin O and the centroid F of the virtual object VO intersects the sphere and the display area AR. In other words, the display control unit 112 may change the appearance of the guide image GP based on the distance between the point obtained by projecting the centroid F toward the origin O onto the sphere and the display area AR.

Figure 10, similar to Figure 2, is a schematic diagram of a virtual object VO existing in an augmented reality or mixed reality space, which is recognized by user U[k] using XR glasses 10-k, and a guidance image GP that guides the user's gaze to the location of the virtual object VO. In Figure 2, D1 is the distance between vertex B of the display area AR and point F' obtained by projecting the centroid F onto the sphere. On the other hand, in Figure 10, D2 is the distance between vertex B of the display area AR and point F' obtained by projecting the centroid F onto the sphere. Comparing Figure 2 and Figure 10, D2 is longer than D1. Accordingly, the length L1 in the long side direction of the first rectangle P1 in Figure 10 is longer than the length L1 in the long side direction of the first rectangle P1 in Figure 2. Similarly, the length L2 in the long side direction of the second rectangle P2 in Figure 10 is longer than the length L2 in the long side direction of the second rectangle P2 in Figure 2.

As a result, user U[k] can determine how close the virtual object VO is to the display area AR.

The same applies to the display control unit 112A according to the second embodiment.

3-2: Modification 2
In the first embodiment, the display control unit 112 displayed a guide image GP that included at least one of a first rectangle P1 and a second rectangle P2, with a side portion of the outer edge of the display area AR, according to the distance from the origin O to the centroid F. However, the appearance of the guide image GP displayed by the display control unit 112 is not limited to this.

Figure 11 shows an example of the appearance of another guide image GP. In the example shown in Figure 11, the guide image GP is a rectangular frame that includes the outer edge of the display area AR. As shown in Figure 11, the display control unit 112 may make at least one of the color and brightness of the area Q1 in the guide image GP that corresponds to the direction in which the virtual object VO exists in the virtual space VS differ from at least one of the color and brightness of other areas.

As a result, user U[k] can recognize the direction of the virtual object VO as seen from the XR glasses 10-k without reducing the visibility of the virtual space VS compared to conventional technology.

The same applies to the display control unit 112A according to the second embodiment.

3-3: Variation 3
In the first embodiment, the display control unit 112 displayed a guide image GP that included at least one of a first rectangle P1 and a second rectangle P2, with a side portion of the outer edge of the display area AR, according to the distance from the origin O to the centroid F. The display control unit 112 may also display other images. For example, consider a virtual object VO projected perpendicularly onto the first rectangle P1 and the second rectangle P2 toward the display area AR. In this case, the display control unit 112 may make at least one of the color and brightness of the portion of the virtual object VO that overlaps the first rectangle P1 and the second rectangle P2 different from at least one of the color and brightness of other portions of the first rectangle P1 and the second rectangle P2.

Figure 12 shows an example of the appearance of the guide image GP according to this modified example. In Figure 12, it is assumed that the virtual object VO is projected perpendicularly onto the first rectangle P1. The display control unit 112 makes at least one of the color and brightness of the area Q2 where the virtual object VO overlaps with the first rectangle P1 different from at least one of the color and brightness of other areas of the first rectangle P1.

As a result, user U[k] can perceive the approximate size of the virtual object VO before actually viewing it.

The same applies to the display control unit 112A according to the second embodiment.

In the first embodiment described above, the information processing system 1 comprises XR glasses 10-1, 10-2, ... 10-k, ... 10-j, and a management server 50. However, the disclosure is not limited thereto. In addition to the XR glasses 10, the information processing system 1 may also include terminal devices connected to the XR glasses 10. If the information processing system 1 includes terminal devices, the terminal devices may perform some or all of the image generation unit 111, display control unit 112, detection unit 113, and voice recognition unit 114.

Furthermore, the XR glasses 10 is an example of a display device that displays a portion of an augmented reality or mixed reality space, in which virtual objects placed in a virtual space are superimposed on the real space, within its display area, but is not limited to this. The display device may also be a terminal device that displays an augmented reality space within its display area. Examples of such terminal devices include smartphones and tablets.

The same applies to the information processing system 1A according to the second embodiment.

4. Other (1) In the embodiments described above, the storage device 12, storage device 12A, and storage device 52 are exemplified by ROM and RAM, but they can also be flexible disks, magneto-optical disks (e.g., compact disks, digital multipurpose disks, Blu-ray® disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy® disks, magnetic strips, databases, servers, and other suitable storage media. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communication network NET via a telecommunications line.

(2) In the embodiments described above, the information, signals, etc. may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

(3) In the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

(4) In the embodiments described above, the determination may be made by a value represented using one bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

(5) The processing procedures, sequences, flowcharts, etc., exemplified in the embodiments described above may be rearranged in order, as long as there is no contradiction. For example, the methods described in this disclosure present various step elements using an exemplary order and are not limited to the specific order presented.

(6) Each function illustrated in Figures 1 to 12 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each function block is not particularly limited. That is, each function block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired, wireless, etc.). A function block may also be realized by combining software with the one or more of the above devices.

(7) The programs illustrated in the embodiments described above should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages or by other names.

Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technologies (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

(8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.

(9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a given value, or other corresponding information.

(10) In the embodiments described above, the XR glasses 10-1 to 10-j, XR glasses 10A-1 to 10A-j, and the management server 50 may be a Mobile Station (MS). A Mobile Station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms. In this disclosure, terms such as “Mobile Station,” “User Terminal,” “User Equipment (UE),” and “Terminal” may be used interchangeably.

(11) In the embodiments described above, the terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, “connection” may be reinterpreted as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain and optical (both visible and invisible) domain.

(12) In the embodiments described above, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on".

(13) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, or inquiring (e.g., searching in a table, database, or other data structure), or ascertaining. “Determining” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), inputting, outputting, or accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

(14) Where the terms “include,” “including,” and variations thereof are used in the embodiments described above, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

(15) In the present disclosure, if articles are added by translation, such as a, an, and the in English, the present disclosure may include the fact that the noun following these articles is plural.

(16) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

(17) Each aspect/embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of the specified information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

(18) In this disclosure, the term “at least one of A and B” or “at least one of A or B” means “A), (B), or (A and B).” That is, the term “at least one of A and B” is equivalent to “one or more of A and B” or “at least one selected from the group of A and B.”

Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are illustrative and not restrictive in any way.

1, 1A... Information processing system, 10, 10A... XR glasses, 11, 11A... Processing unit, 12, 12A... Storage device, 13... Imaging device, 14... Sound collection device, 15... Detection device, 16... Communication device, 17... Display device, 50... Management server, 51... Processing unit, 52... Storage device, 53... Input device, 54... Communication device, 55... Display, 111, 111A... Image generation unit, 112, 112A... Display control unit, 113... Detection unit, 114... Voice recognition unit, 511... Communication control unit, 512... Acquisition unit, P1... First rectangle, P2... Second rectangle, PR1, PR1A, PR2... Control program.

Claims (6)

A display device that displays a portion of an augmented reality space or mixed reality space, in which virtual objects placed in a virtual space are superimposed on the real space, in a display area,
If the virtual object is not displayed in the display area, an image generation unit generates a guidance image that guides the user's gaze to the location of the virtual object based on the position of the virtual object placed in the virtual space.
A display control unit that displays the aforementioned guidance image at a position touching the outer edge of the display area,
Equipped with,
The display control unit is a display device that makes at least one of the color and brightness of the part of the guidance image corresponding to the direction in which the virtual object exists in the virtual space different from at least one of the color and brightness of other parts. The system further includes a detection unit that detects the user's body movements,
The display device according to claim 1, wherein the user input to the display device is a specific movement of the user's body detected by the detection unit. The system further includes a voice recognition unit that recognizes the user's voice,
The display device according to claim 1, wherein the user input to the display device is a voice representing the name of the virtual object pronounced by the user and recognized by the voice recognition unit. The display control unit changes the shape of the guidance image according to the distance from the display device to the virtual object, as described in claim 1. The display control unit changes the appearance of the guide image in accordance with the movement of the virtual object, as described in claim 1. The display device according to claim 1, wherein the display control unit changes the appearance of the guidance image in accordance with changes in the real space displayed in the display area.