JP7496403B2 - Display terminal - Google Patents

Description

The present invention relates to a technology for displaying virtual objects on a display terminal.

There is a technology for improving the visibility of an Augmented Reality (AR) image that adds computer-generated information to an object in real space and is displayed on a display terminal such as an HMD (Head Mounted Display). For example, Patent Document 1 discloses a head-mounted display device with a display unit that can see through the outside world, which includes "a superimposed image display control unit that causes a predetermined image to be displayed on the display unit by superimposing it on the outside world seen through the display unit, an imaging unit that captures at least a predetermined range of the outside world seen through the display unit, a partial image identification unit that identifies a partial image of a predetermined range whose position corresponds to the predetermined image from the captured image obtained by the imaging unit, and a visibility correction unit that corrects the appearance of the predetermined image displayed by the superimposed image display control unit according to color information about the identified partial image (summary excerpt)."

JP 2016-142887 A

Virtual objects displayed in AR images can be moved, rotated, transformed, and so on, by giving instructions to operation points set in advance around the virtual object. However, depending on the display position of the virtual object, part of the virtual object may be behind a surrounding structure, resulting in hidden surface elimination, and as a result, the necessary operation points may not be displayed. For this reason, high operability is not necessarily achieved. On the other hand, if the virtual object is displayed as is, regardless of the position of the virtual object after operation, a sense of reality cannot be achieved.

The present invention was made in consideration of the above circumstances, and aims to provide a virtual object display technology that achieves high operability while maintaining a sense of realism regardless of the display position of the virtual object.

The present invention is a display terminal comprising a display, a color image camera that acquires a color image of a predetermined shooting range, a distance image camera that acquires a distance image of the shooting range, a detection unit that detects the gaze direction of a user of the display terminal, a spatial recognition unit that generates a three-dimensional map of structures within the shooting range using the color image and the distance image, and a control unit that controls the display to display a virtual object, wherein the control unit controls the display to display an operation object for the user of the display terminal to operate the virtual object in a position different from the virtual object when a rear area in which the virtual object is behind the structure in relation to the gaze direction detected by the detection unit is identified based on the three-dimensional map and placement position data of the virtual object in real space.

The present invention provides a virtual object display technology that maintains a sense of realism and achieves high operability regardless of the display position of the virtual object. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiment.

1A to 1C are explanatory diagrams illustrating an overview of a virtual object display process according to an embodiment of the present invention. FIG. 1A is a diagram showing the hardware configuration of an HMD according to an embodiment of the present invention, and FIG. FIG. 2 is a functional block diagram of a controller of the HMD according to the embodiment of the present invention. 1A and 1B are explanatory diagrams illustrating an overview of a virtual object display process according to an embodiment of the present invention. 11 is a flowchart of a virtual object display process according to the embodiment of the present invention. 1A and 1B are diagrams illustrating a display according to an embodiment of the present invention. 1A to 1C are explanatory diagrams illustrating an example of a virtual object display mode according to an embodiment of the present invention. 13A to 13C are explanatory diagrams illustrating another example of a virtual object display mode according to an embodiment of the present invention. 13A and 13B are diagrams illustrating a first modified example of a virtual object display mode according to an embodiment of the present invention. FIG. 11 is an explanatory diagram for explaining a modified example 2 of the aspect of virtual object display according to the embodiment of the present invention. FIG. 11 is an explanatory diagram for explaining a modified example 3 of the aspect of virtual object display according to the embodiment of the present invention. FIG. 13 is an explanatory diagram for explaining a fourth modified example of a virtual object display mode according to an embodiment of the present invention. FIG. 13 is an explanatory diagram for explaining a modified example 5 of the aspect of virtual object display according to the embodiment of the present invention. FIG. 13 is an explanatory diagram for explaining a sixth modified example of a virtual object display mode according to an embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the drawings. In the following description, unless otherwise specified, parts having the same function are given the same reference numerals and repeated explanations are omitted. Note that the present invention is not limited to the embodiment described here.

In this embodiment, the display terminal is an HMD (Head Mounted Display) that is worn on the head of the user (wearer), and a see-through (see-through) HMD that has a transparent display and allows both the outside world and the displayed image to be viewed will be used as an example.

First, an overview of this embodiment will be described with reference to Figs. 1(a) to 1(c). In this embodiment, as shown in Fig. 1(a), a user 201 wears an HMD 200 indoors, and displays and operates a virtual object 100 on the display of the HMD 200.

The HMD 200 has distance information (depth information) of the real space and the virtual object 100. Conventionally, as shown in FIG. 1(b), the display of the HMD 200 does not display the part of the virtual object 100 that is located behind a structure 300 such as a wall in the real space. Therefore, in such a case, the user 201 cannot grasp the entirety of the virtual object 100.

Generally, the virtual object 100 is moved, rotated, transformed, and so on, by manipulating an operation point (transform controller) 110 that is set on or near the periphery of the virtual object 100. Therefore, when displayed as in FIG. 1(b), the operation point 110 behind the structure 300 is not displayed and cannot be operated.

To solve this problem, in this embodiment, as shown in FIG. 1(c), even if a structure 300 is present, the virtual object 100 displays all operation points 110. The virtual object 100 itself may also be displayed in its entirety. In this case, an additional object 400 may also be displayed so that it is clear that even parts that would not normally be displayed are being displayed.

The following describes the HMD 200 of this embodiment, which realizes such display control.

[Hardware configuration diagram]
Fig. 2A is a diagram showing the hardware configuration of the HMD 200. Fig. 2B is a diagram showing the external appearance of the HMD 200 of this embodiment.

The HMD 200 of this embodiment basically has the same configuration as a general-purpose computer (information processing device). That is, as shown in this figure, the HMD 200 includes a controller 210, a camera 214, a display 215, an audio interface (I/F) 216, a communication I/F 217, a sensor 218, a bus 219 that electrically connects each part, and a gaze detection device 214c. Furthermore, the HMD 200 of this embodiment includes a frame 241 that supports each part and is worn by the user 201.

The controller 210 performs various processes according to a predetermined program. In this embodiment, the controller 210 displays the virtual object 100 at a predetermined position on the display 215, for example.

The controller 210 of this embodiment includes a CPU 211, a RAM 212, and a ROM 213. The CPU 211 loads programs stored in advance in the ROM 213 into the RAM 212 and executes them to realize various functions. Note that the RAM 212 and the ROM 213 are collectively referred to as the storage device 230 (see FIG. 3) unless there is a particular need to distinguish between them. Note that the controller 210 is disposed on, for example, a frame 241.

The camera 214 includes a color image camera 214a and a distance image camera 214b. The color image camera 214a captures an image of a shooting range including the visual field range of the user 201, and acquires a color image. The distance image camera 214b acquires a distance image of a shooting range that is approximately the same as that of the color image camera 214a. The camera 214 (color image camera 214a and distance image camera 214b) is disposed, for example, at a position at the very front of the frame 241 (closest to the display 215) where the above-mentioned shooting range can be captured.

The display 215 is a device that displays images acquired by the camera 214 and display data generated within the HMD 200. The display 215 is composed of, for example, a transmissive liquid crystal device, an organic EL device, or an optical scanning device using MEMS (micro electro mechanical systems). However, the device is not limited to this, and any device can be used as long as it can realize a transmissive display structure that displays an image on the display 215 while allowing the viewer to see through to the other side of the display 215.

In the HMD 200 of this embodiment, a transmissive display 215 is supported in front of one or both eyes of the user 201. The display 215 can have any shape. The display 215 may include right and left display panels, and one or more UI objects of a graphical user I/F may be displayed on the display 215.

The audio I/F 216 is, for example, an audio output device such as a microphone, speaker, or buzzer. The audio I/F 216 inputs external sounds and outputs sounds such as sounds created within the HMD 200 and sounds such as voice and music sent via the communication I/F 217. In this embodiment, the audio I/F 216 does not necessarily have to be provided.

The communication I/F 217 includes an encoding circuit, a decoding circuit, an antenna, etc., and transmits and receives data (data communication) with other devices via a network. In this embodiment, the communication I/F 217 is an I/F that connects to the network via an access point or the like (not shown), or via a base station or the like of a mobile telephone communication network (not shown). Via the communication I/F 217, the HMD 200 transmits and receives data with each server connected to the network.

The connection between the HMD 200 and the access point is established by, for example, a wireless communication method such as Wi-Fi (registered trademark) or other communication method. The connection between the HMD 200 and the base station of the mobile telephone communication network is established by, for example, a W-CDMA (registered trademark) (Wideband Code Division Multiple Access) method, a GSM (registered trademark) (Global System for Mobile communications) method, a LTE (Long Term Evolution) method, or other communication method. Note that the HMD 200 of this embodiment does not need to include a communication I/F 217.

The sensor 218 detects the current position, inclination, speed, and operation by the user 201 of the HMD 200. The HMD 200 includes, as the sensor 218, a position information acquisition sensor such as a GPS receiver 218a, a gyro sensor 218b, an acceleration sensor 218c, a geomagnetic sensor 218d, a touch sensor 218e, and the like. Note that it is not necessary to include all of the sensors 218.

The gaze detection device 214c detects the gaze direction of the user 201. The gaze detection device 214c is realized, for example, by a gaze detection camera that detects the gaze direction of the user 201. The gaze detection camera is attached so that the iris, pupil, etc. of the user 201's eye are included in the shooting range.

The frame 241 supports the components of the HMD 200, such as the display 215, the camera 214, and the controller 210.

[Function block]
Next, functions related to virtual object display realized by the controller 210 of this embodiment will be described. Fig. 3 is a functional block diagram of functions related to virtual object display processing of the HMD 200 of this embodiment. As shown in this figure, the controller 210 of this embodiment realizes the functions of an image acquisition unit 228, a display control unit 220, and an audio output control unit 229. In addition, the display control unit 220 includes a space recognition unit 221, an instruction acceptance unit 222, a display data generation unit 223, and a display correction unit 224.

Each function is realized by the CPU 211 loading a program stored in the ROM 213 into the RAM 212 and executing it.

The storage device 230 also stores color image data 231, distance image data 232, spatial recognition data 233, virtual object data (virtual OJT data) 234, additional object data (additional OJT data) 235, and audio data 236.

The color image data 231 is an image acquired by the color image camera 214a. The distance image data 232 is an image acquired by the distance image camera 214b.

The image acquisition unit 228 stores the color image and distance image acquired by the color image camera 214a and the distance image camera 214b, respectively, in the storage device 230 as color image data 231 and distance image data 232. In this embodiment, the color image and distance image are acquired approximately synchronously.

The spatial recognition unit 221 recognizes the surrounding real space and stores the result in the storage device 230 as spatial recognition data 233. The recognition is performed using color image data 231 and distance image data 232 acquired approximately simultaneously.

The spatial recognition unit 221 generates spatial recognition data 233, which is three-dimensional data (three-dimensional map) of the structure 300 in the shooting range, from each image data at a predetermined time interval in accordance with the scanning operation of the user 201, and stores it in the storage device 230. The surroundings are scanned by the user 201 as an initial setting, for example, immediately after startup.

The spatial recognition data 233 is created, for example, in a world coordinate system that defines the entire three-dimensional space. As an example, the origin and each axis direction of this world coordinate system are used as the origin and each axis direction of the local coordinate system of the HMD 200, which is specified by the position and orientation (initial posture) of the HMD 200 body when an instruction to start spatial recognition is received. In this coordinate system, for example, in the initial posture of the HMD 200 body, a specific position on the display 215 of the HMD 200 is set as the origin, the interior of the display 215 is set as the xy plane, and the z-axis direction is set as perpendicular to the xy plane (the surface of the display 215).

The amount of displacement and rotation of the local coordinate system of the HMD 200 relative to the world coordinate system due to the scanning action of the user 201 is calculated using data obtained by various sensors 218.

The spatial recognition is performed using, for example, existing technology such as Spatial Mapping. That is, the HMD 200 of this embodiment scans the surroundings using the color image camera 214a and the distance image camera 214b. Then, using the results, the spatial recognition unit 221 generates three-dimensional data using an application such as Spatial Mapping. The spatial recognition data 233 is stored, for example, as mesh data.

At this time, the spatial recognition unit 221 may be configured to simultaneously perform not only the three-dimensional data but also spatial understanding to recognize the type of structure 300. By spatial understanding, the spatial recognition unit 221 can recognize the material and type of the structure 300 within the shooting range. In other words, it can recognize whether the structure 300 is, for example, a "wall," a "floor," or a "ceiling." The spatial recognition unit 221 of this embodiment stores these recognition results in the storage device 230 as attribute data of the spatial recognition data 233.

The instruction receiving unit 222 receives display instructions and operation instructions from the user 201 for the virtual object 100 displayed on the display 215. The display instructions and operation instructions may be, for example, instructions based on the line of sight (gaze) or finger movements (gestures).

The gaze direction information used for gaze is detected, for example, using the gaze detection device 214c.

Gestures include, for example, a click event (air tap), tap and hold, bloom, etc. on the operation point 110 of the virtual object 100. The instruction receiving unit 222 detects the movement of fingers within a gesture frame provided within the shooting range of the color image camera 214a and the distance image camera 214b, and detects display instructions, operation instructions, etc.

For example, when a display instruction is received, the instruction receiving unit 222 extracts data of the specified virtual object 100 from the virtual object data 234, and causes the display data generating unit 223 described below to generate the display data.

In addition, when an operation instruction is received, the instruction receiving unit 222 detects the operation and notifies the display data generating unit 223 described below.

The display data generating unit 223 generates display data from the virtual object data 234 in accordance with an instruction from the user 201 via the instruction receiving unit 222, for displaying the specified virtual object 100 in a specified shape at a specified position on the display 215. By displaying the display data generated in accordance with the instruction from the instruction receiving unit 222 on the display 215, the virtual object 100 is displayed to move, rotate, and deform in accordance with the instruction from the user 201.

At this time, the display data generation unit 223 generates display data in which the area of the virtual object 100 behind the structure 300 in the line of sight of the user 201 (calculated from the spatial coordinate position of the HMD 200 worn by the user 201 and information on the up, down, left and right orientations) is identified as the rear area 101. The rear area 101 (Figure 4) is identified based on the three-dimensional map (spatial recognition data 233) generated by the spatial recognition unit 221 and the placement position data in real space of the virtual object 100 to be displayed. Note that the placement position data in real space is registered in the same coordinate system as the coordinate system used by the spatial recognition unit 221 for spatial recognition.

The placement position data of the virtual object 100 in real space is obtained from the virtual object data 234 of the virtual object 100 to be displayed. The virtual object data 234 includes the size, shape, and initial placement position information of each virtual object 100. The initial placement position information is registered, for example, for a placement position point, an operation point 110, etc. that is set in advance for each virtual object 100. The placement position point is, for example, the three-dimensional center of gravity position of the virtual object 100. Also, the operation point 110 is a point that receives an instruction to change the display shape of the virtual object 100, as described above.

The display data generating unit 223 obtains the latest placement position information of the virtual object data 234 corresponding to the virtual object 100 to be displayed by reflecting the instruction from the instruction receiving unit 222 in the current placement position information. The display data generating unit 223 then obtains placement position data of the virtual object 100 in real space using the latest placement position information of the placement position point and information such as size and shape.

The display data generating unit 223 further uses the latest placement position data in real space to identify the back region 101 of the virtual object 100 to be displayed. For example, based on the placement position data of the virtual object 100 in real space, the display data generating unit 223 identifies the virtual object 100 as depth information from the coordinate position in the space of the HMD 200 worn by the user 201, the distance to the virtual object calculated from the information on the up, down, left, and right orientations, the shape data of the virtual object 100, and the coordinate position of the structure 300.

The area (part) of the virtual object 100 and the operation point 110 that is located relatively further back than the depth information of the structure 300 is defined as the rear area 101.

In normal processing, the rear region 101 is an area where hidden surface processing is performed. For example, as shown in FIG. 4(a), when a part of the virtual object 100 is placed behind a structure 300 such as a wall or furniture, the display data generating unit 223 conventionally performs hidden surface processing on this rear region 101 of the virtual object 100 and generates display data so as not to display it.

However, in this embodiment, in order to aim for a more user-friendly display, even in such a case, the display correction unit 224 corrects the display so that the entire virtual object 100 and the associated operation point 110 are displayed, as shown in FIG. 4(b). This processing is performed by the display correction unit 224.

When the virtual object data 234 to be displayed has a back area 101, the display correction unit 224 corrects the display data of the virtual object data 234.

Specifically, the display correction unit 224 cancels the hidden surface processing for the rear region 101. Then, the display data is corrected so that the rear region 101 is also displayed as if the structure 300 does not exist. In other words, even if a part of the virtual object data 234 is behind the structure 300, the display correction unit 224 corrects the display data so that the part is also displayed.

Furthermore, at this time, the display correction unit 224 displays an additional object 400 to indicate that the virtual object 100 is not in its original display mode. The additional object 400 is an object for making the display of the virtual object 100 appear more natural and realistic, for example, by showing the virtual object 100 breaking through the structure 300 or making a hole in the structure 300.

The data of the additional object 400 to be displayed is prepared in advance as additional object data 235 in the storage device 230. The additional object data 235 is registered in association with information such as the display mode of the additional object 400, its initial size, and its initial display position relative to the display position of the virtual object 100.

When the display direction of the virtual object 100 relative to the line of sight of the user 201 is changed by an operation of the user 201, the display correction unit 224 may change the display shape of the additional object 400 accordingly. Furthermore, when the display direction of the virtual object 100 is changed due to a movement of the line of sight of the user 201, the display correction unit 224 may change the display shape of the additional object 400 in a similar manner. The deformation of the additional object 400 is performed, for example, according to the same program as the program used for the process of deforming the display of the virtual object 100 in accordance with a change in the line of sight.

Next, the flow of the display process of the virtual object 100 by the controller 210 of this embodiment will be described. FIG. 5 is a process flow of the virtual object display process of this embodiment. It is assumed that the spatial recognition process has already been performed as an initial process. It is also assumed that the virtual object 100 is displayed at a position where the entire object is displayed.

The display data generation unit 223 and the display correction unit 224 repeat the following process until an end instruction is received from the user 201 via the instruction receiving unit 222 (step S1101).

When a movement operation on the virtual object 100 is received from the user 201 via the instruction receiving unit 222 (step S1102), the display data generating unit 223 generates display data (step S1103).

In step S1103, the display data generation unit 223 first calculates the display position of the virtual object 100 on the display 215. The display data generation unit 223 then identifies the line of sight direction, and calculates depth information according to the corresponding virtual object data 234 of the virtual object 100. The display data generation unit 223 also calculates depth information of the structure 300. The display data generation unit 223 then identifies the back area 101 on the display data.

Next, the display data generation unit 223 determines whether the virtual object 100 to be displayed has a back area 101 (step S1104).

If there is no back area 101 (S1104; No), the display data generation unit 223 notifies the display correction unit 224 of that fact. The display correction unit 224 then displays the display data of the virtual object 100 at the position calculated in step S1103 on the display 215 (step S1112). The controller 210 then waits for the next operation instruction.

If there is a rear area 101 (S1104; Yes), the display data generation unit 223 determines whether the entire virtual object 100 is the rear area 101 (step S1105).

If the entire area is the back area 101 (S1105; Yes), the display data generation unit 223 notifies the display correction unit 224 of that fact. The display correction unit 224 then erases the display of the virtual object 100 (step S1111) and ends the process.

On the other hand, if the entire area is not the rear area 101 (S1105; No), that is, if the rear area 101 is part of the virtual object 100, the display data generation unit 223 notifies the display correction unit 224 of that effect. Then, the display correction unit 224 corrects the display data of the virtual object 100 using the method described above (step S1106). Here, the display correction unit 224 corrects the display data so as to display all the operation points 110 of the virtual object 100 and the parts of the virtual object 100 operated by the operation points 110. Then, the display correction unit 224 displays the corrected display data on the display 215 at the position calculated in step S1103 (step S1107).

Then, the display correction unit 224 obtains the additional object data 235 from the storage device 230 and displays it superimposed on the back area 101 of the virtual object 100 (step S1108). Then, the controller 210 waits for the next operation instruction.

The above process will be explained using a concrete example. Note that on the display 215, as shown in FIG. 6(a), the virtual object 100 and the additional object 400 are displayed within a frame 215a defined by the display 215. Here, the structure 300 is a real object in real space. In the following explanation of the display mode, the frame of the display 215 will be omitted, and only the real objects visible through the display 215 and the display data such as the virtual object 100 will be described and explained, as shown in FIG. 6(b).

For example, assume that a virtual object 100 is displayed at the position shown in FIG. 7(a). In response, a user 201 performs an operation instruction (gesture) in real space by pushing with his/her finger 202 in the direction of arrow 203. By continuing this, the display position of the virtual object 100 moves relative to a structure 300, which is a real object in the real space.

According to this embodiment, as shown in FIG. 7(b), even if the display position of a portion of the virtual object 100 is behind the structure 300, the entire virtual object 100 is displayed together with its operation point 110. In addition, at this time, the additional object 400 is displayed as if a hole had been made in the wall of the structure 300.

Furthermore, at this time, as shown in FIG. 7(c), the audio output control unit 229 may be configured to output audio. When the display correction unit 224 displays the additional object 400, it notifies the audio output control unit 229 accordingly. Upon receiving the notification from the display correction unit 224, the audio output control unit 229 extracts audio from the audio data 236 and outputs it from the audio I/F 216.

Note that different data may be registered as the audio data 236 in association with the material of the structure 300. The material of the structure 300 is recognized, for example, by spatial understanding performed by the spatial recognition unit 221 described above.

At this time, an onomatopoeia expressing the output sound data 236 as a character string may be displayed. For example, "Bari" as shown in FIG. 7(c). In this case, onomatopoeia data 237 expressing the sound data 236 as a character string is registered in advance in the storage device 230 in association with the sound data 236. Then, the display correction unit 224 generates display data so that the onomatopoeia data 237 registered in association with the output sound data 236 is displayed near the additional object 400.

A speech bubble may also be displayed, and the onomatopoeia data 237 may be displayed in the speech bubble. Also, only the onomatopoeia data 237 may be displayed without outputting the voice data 236.

The display correction unit 224 also changes the shape of the additional object 400 accordingly as the virtual object 100 sinks into the structure 300. For example, as shown in FIG. 8(a), a case will be described in which the virtual object 100 has a shape formed by combining two rectangular parallelepipeds with different horizontal widths. Here, the virtual object 100 is pushed into the structure 300 in the direction of the arrow 203, starting from the rectangular parallelepiped with the smaller horizontal width.

In this case, the display correction unit 224 processes the size of the additional object 400 according to the horizontal width of the virtual object 100 and displays it, as shown in FIG. 8(b). Specifically, as shown in FIG. 8(a), when a rectangular parallelepiped region with a small horizontal width is virtually embedded in the structure 300, the additional object 400 is displayed at a small size. On the other hand, when a region with a larger horizontal width than that in FIG. 8(a) is virtually embedded in the structure 300, the additional object 400 is displayed at a larger size than that in FIG. 8(a), as shown in FIG. 8(b).

In addition, the line of sight direction changes as the user 201 changes his/her standing position. For example, as shown in FIG. 8(c), when the user 201 faces a wall, which is a structure 300, the display mode of the virtual object 100 changes from the cases of FIG. 8(a) and FIG. 8(b) even if the display position of the virtual object 100 in the world coordinate system is the same. Accordingly, the display shape of the virtual object 100 also changes.

In this case, the display correction unit 224 may also change the display shape of the additional object 400 to follow the change in the display shape of the virtual object 100, as shown in FIG. 8(c).

As described above, the HMD 200 of this embodiment includes a color image camera 214a that acquires a color image of a predetermined shooting range, a distance image camera 214b that acquires a distance image of approximately the same shooting range, a display 215, and a display control unit 220 that displays the virtual object 100 on the display 215. The display control unit 220 includes a space recognition unit 221 that generates a three-dimensional map of the structure 300 within the shooting range using the color image and the distance image, a display data generation unit 223 that generates display data that identifies the area of the virtual object 100 behind the structure 300 in the line of sight as the back area 101 based on the three-dimensional map and the placement position data in the real space of the virtual object 100 to be displayed, and a display correction unit 224 that corrects the display data and displays it on the display 215. The display correction unit 224 also corrects the display data to display the operation point 110 in the back area 101. The operation point 110 is a point through which an operation instruction for the virtual object 100 is received.

In this way, according to this embodiment, the operation point 110 of the back area 101 of the virtual object 100, which is behind the structure 300 and would not normally be displayed, is also displayed. Therefore, even if the virtual object 100 is moved by the operation of the user 201, the user can perform an operation on the virtual object 100 after the movement.

In addition, when displaying the operation point 110 in the rear area 101, the display correction unit 224 also displays the rear area 101 and displays the additional object 400 around the virtual object 100. Therefore, according to this embodiment, regardless of the display position of the virtual object 100 on the display 215, it is possible to achieve high operability while maintaining a sense of realism without sacrificing a natural appearance.

[Modification 1]
Note that the display of the virtual object 100 having a part of the back region 101 is not limited to the above. For example, the display correction unit 224 may display the virtual object 100 so that the back region 101 and the other region (hereinafter referred to as a front region) 102 can be distinguished.

For example, as shown in FIG. 9(a), the display correction unit 224 corrects the display data so that a line 121 is displayed between the rear region 101 and the front region 102. Also, as shown in FIG. 9(b), the display correction unit 224 corrects the display data so that the rear region 101 is displayed with a surface appearance 122 that is different from the original surface of the virtual object 100. The correction appearances in FIG. 9(a) and FIG. 9(b) may be combined.

[Modification 2]
Moreover, the additional object 400 may be displayed in an emphasized manner. An example of the additional object 400 displayed in an emphasized manner is shown in Fig. 10. The emphasized display mode of the additional object 400 may be registered in advance in the additional object data 235. Moreover, the display correction unit 224 may process the registered additional object 400 into an emphasized display mode using various image processing software, and display it.

By highlighting the additional object 400, the user 201 can easily understand that the virtual object 100 is displayed differently from the original display mode. In other words, the user 201 can more intuitively understand that an area that would normally be invisible is being displayed.

[Modification 3]
Furthermore, the additional object data 235 may have different textures prepared according to the material and/or type of the structure 300 into which the virtual object 100 is pushed. In this case, the display correction unit 224 refers to the space recognition data 233 from the space recognition unit 221 and identifies the material and/or type of the structure 300. Then, the display correction unit 224 extracts the additional object data 235 having a texture according to the identified material and/or type, and displays it as the additional object 400.

For example, if the structure 300 is made of a hard material such as a wall, an additional object 400 representing a state in which the wall is cracked, as shown in FIG. 10, is prepared as additional object data 235. On the other hand, if the structure 300 is made of a soft material such as a cushion, an additional object 402 representing a state in which the virtual object 100 is sinking into the structure 300, as shown in FIG. 11, is prepared and displayed.

Furthermore, at this time, the manner of highlighting may be configured to change depending on the material and/or type of the structure 300, or the texture of the additional object 400.

[Modification 4]
Furthermore, the display correction unit 224 may cut out a part of the color image data 231 and use it as the additional object 400. In particular, as shown in Fig. 12, an image of the same texture as a structure 301 behind a structure 300 into which the virtual object 100 is pushed may be used. In this case, the display correction unit 224 processes the additional object data 235 using the space recognition data 233 according to the arrangement position of the additional object 400. Then, the display correction unit 224 arranges the processed additional object data 235.

In this way, by using the same texture data as the background image for the additional object 400, the user 201 can perceive the display mode of the virtual object 100 of this embodiment more naturally.

[Modification 5]
In the above embodiment, when the virtual object 100 is moved to a position where the entire object is in the rear area 101, the display of the virtual object 100 is erased, but this is not limited to the above. Even when the entire object is in the rear area 101, the virtual object 100 and/or the operation point 110 may be displayed.

For example, as shown in FIG. 13, the display correction unit 224 corrects the display data so that the virtual object 100 is displayed at the calculated position. Furthermore, the display correction unit 224 may display a miniature virtual object 401 for operation as an additional object in the foreground. In this case, the display correction unit 224 generates the miniature virtual object 401 from the virtual object 100 and displays it.

As a result, even if the entire virtual object 100 is placed further in the depth direction than the structure 300, the user 201 can perform operations while maintaining a sense of reality.

[Modification 6]
Furthermore, the display 215 may be a non-transmissive type. In this case, as shown in Fig. 14, the display correction unit 224 displays the virtual object 100 and the additional object 400 in a superimposed manner on the color image (through image 500) acquired by the color image camera 214a.

In this case, for example, the display terminal is not limited to the HMD 200 equipped with a transmissive display. For example, it may be an HMD equipped with a non-transmissive display, a portable or mobile information processing device such as a mobile terminal, etc.

[Modification 7]
Furthermore, input of operation instructions is not limited to gaze and gestures, but may also be performed using voice, a motion controller, or the like.

[Modification 8]
In the above embodiment, the display correction unit 224 corrects the display of the virtual object 100 so that all the operation points 110 of the virtual object 100 are displayed, but this is not limited to the above. The operation points 110 to be displayed may be some of the operation points 110 in the rear area 101. In other words, at least one operation point 110 in the rear area 101 may be displayed.

The same applies to the shape of the virtual object 100; it is not necessary to display the entire shape. A portion of the back area 101 may be displayed.

The present invention is not limited to the above-described embodiments and modifications, but includes various modifications. For example, the above-described embodiments and modifications have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is also possible to replace part of the configuration of one embodiment or modification with the configuration of another embodiment or modification. It is also possible to add the configuration of another embodiment or modification to the configuration of one embodiment or modification. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment or modification with other configurations.

The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. The above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory unit, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

In addition, the control lines and information lines shown are those considered necessary for the explanation, and not all control lines and information lines on the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected.

100: Virtual object, 101: Back area, 102: Front area, 110: Operation point, 121: Line, 122: Surface aspect,
200: HMD, 201: user, 202: finger, 203: arrow, 210: controller, 211: CPU, 212: RAM, 213: ROM, 214: camera, 214a: color image camera, 214b: distance image camera, 214c: gaze detection device, 215: display, 215a: inside frame, 216: audio I/F, 217: communication I/F, 218: sensor, 218a: GPS receiver, 218b: gyro sensor, 218c: acceleration sensor, 218d: geomagnetic sensor, 218e: touch sensor, 219: bus,
220: display control unit, 221: space recognition unit, 222: instruction reception unit, 223: display data generation unit, 224: display correction unit, 228: image acquisition unit, 229: audio output control unit,
230: storage device, 231: color image data, 232: distance image data, 233: space recognition data, 234: virtual object data, 235: additional object data, 236: voice data, 237: onomatopoeia data,
241: frame,
300: structure, 301: structure, 400: additional object, 401: miniature virtual object, 402: additional object, 500: through image

Claims (8)

A display terminal,
A display and
a color image camera for acquiring a color image of a predetermined shooting range;
a distance image camera for acquiring a distance image of the shooting range;
A detection unit that detects a line of sight direction of a user of the display terminal;
a spatial recognition unit that generates a three-dimensional map of structures within the shooting range using the color image and the distance image;
A control unit that controls the display to display a virtual object ,
the control unit, when a rear area in which the virtual object is behind the structure in the line of sight direction detected by the detection unit is identified based on the three-dimensional map and position data of the virtual object in real space, controls so that an operation object for a user of the display terminal to operate the virtual object is displayed in a position different from the virtual object . A display terminal,
A display and
a color image camera for acquiring a color image of a predetermined shooting range;
a distance image camera for acquiring a distance image of the shooting range;
A detection unit that detects a gaze direction of a user of the display terminal;
A three-dimensional map of structures within the imaging range is generated using the color image and the range image.
a spatial recognition unit that
A control unit that controls the display to display a virtual object,
The control unit acquires data on the three-dimensional map and the position of the virtual object in a real space.
Based on the above, before the object is behind the structure in the line of sight direction detected by the detection unit.
When displaying the back area identified as the area of the virtual object, the structure is cracked.
or the virtual object sinks and the structure changes.
The additional object is displayed around the virtual object, and the display terminal
A manipulation object for a user to manipulate the virtual object is displayed on the display.
A display terminal characterized by controlling the display to be displayed on the display device . A display terminal,
A display and
a color image camera for acquiring a color image of a predetermined shooting range;
a distance image camera for acquiring a distance image of the shooting range;
A detection unit that detects a line of sight direction of a user of the display terminal;
a spatial recognition unit that generates a three-dimensional map of structures within the shooting range using the color image and the distance image;
A control unit that controls the display to display a virtual object,
When displaying a rear area specified as a region of the virtual object that is behind the structure in relation to the line of sight detected by the detection unit based on the three-dimensional map and placement position data of the virtual object in real space, the control unit
Adding an additional object to the virtual structure to show that the structure has been torn or has a hole in it
and controlling the display so that an operation object for a user of the display terminal to operate the virtual object is displayed around the virtual object on the display. The display terminal according to claim 2 or 3 further comprises:
A storage unit for storing a plurality of additional objects,
The control unit performs the above-mentioned storage in accordance with the type or material of the structure recognized by the space recognition unit.
A display terminal that displays an additional object extracted from the portion . A display terminal according to claim 2 or 3 ,
The control unit detects that a second structure larger than the structure is within the shooting range.
a color image of the second structure being used to display the additional object when the second structure is behind the additional object . A display terminal according to any one of claims 1 to 3 ,
A display terminal , wherein the operation object is a miniature object of the virtual object . A display terminal according to any one of claims 1 to 3 ,
The display terminal is a head mounted display . A display terminal according to any one of claims 1 to 3 ,
The display terminal is a portable information processing device .

Images