JP7802161B2 - Display control system and wearable device - Google Patents

Description

The present invention relates to a display control system and a wearable device.

XR glasses that apply XR (Extended Reality) technology, such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality), have become widespread. XR glasses may be equipped with an imaging device that captures images of the area in front of the user. Images captured by the imaging device are used, for example, to assist the user wearing the XR glasses in performing tasks. Alternatively, the captured images are used for recording purposes, such as for retrospective evaluation of whether the user performed the task appropriately.

For example, Patent Document 1 below discloses a work support system that uses markers placed in real space to align virtual objects placed in virtual space with real-space objects. Specifically, the work support system includes a display unit, an imaging unit, a position information detection unit, and a display control unit. The display unit displays an image of the virtual space corresponding to the real space. The imaging unit captures an image of an area including the center of the user's field of view. The position information detection unit detects the positional relationship between the marker and the imaging unit based on the captured image of the marker. The display control unit controls the display of the image of the virtual space based on the detection result of the image of the marker detected while the user's gaze is on the marker and the detection result of mechanically detecting the user's posture.

Japanese Patent Application Laid-Open No. 2020-170249

Generally, there is a discrepancy between the imaging range of the imaging device installed in the XR glasses and the user's field of view. As a result, the imaging range expected by the user may not match the actual imaging range, and the required image may not be obtained. Furthermore, for example, when a user works while receiving advice from an assistant in a remote location, the user may not be able to grasp the area the assistant wants them to focus on, i.e., the area the assistant wants the imaging device to capture, which may result in reduced work efficiency.

The object of the present invention is to efficiently capture images that depict a specific range of real space.

A display control system according to one aspect of the present invention is a display control system comprising a wearable device and a terminal device, wherein the wearable device comprises a first display control unit that controls a transparent first display device positioned in front of a first user's eye, and a first communication control unit that transmits to the terminal device a captured image of real space included in the first user's field of view. The terminal device comprises a second communication control unit that receives the captured image, a second display control unit that displays the captured image on a second display device, and a reception unit that receives from the second user a designation of a first area in the captured image displayed on the second display device. The second communication control unit transmits area information that identifies the range of the first area in the captured image to the wearable device, and the first display control unit displays on the first display device a first virtual object that indicates a range in the real space that corresponds to the first area based on the area information.

A wearable device according to one aspect of the present invention comprises a first display control unit that controls a transparent first display device located in front of the first user's eye, a transmission unit that transmits to a terminal device a captured image of real space included in the first user's field of view, and a reception unit that receives area information that identifies the range of a first area specified in the captured image, and the first display control unit displays on the first display device a first virtual object that indicates a range of the real space that corresponds to the first area based on the area information.

According to one aspect of the present invention, it is possible to efficiently capture an image that captures a specific range of real space.

1 is a block diagram showing a configuration of a display control system 1 according to an embodiment. 1A and 1B are diagrams illustrating a wearable device 10 in use; FIG. 2 is a diagram schematically illustrating a state in which a terminal device 20 is used. FIG. 1 is a block diagram showing the configuration of a wearable device 10. FIG. 2 is a block diagram showing the configuration of a terminal device 20. FIG. 10 is an explanatory diagram schematically showing a range related to the visual field range R1 of a first user U1. FIG. 2 is a diagram showing an example of a captured image displayed on a display device 201. FIG. 2 is a diagram showing an example of a captured image displayed on a display device 201. FIG. 2 is a diagram showing an example of a display range R3 of the transmissive display device 101. FIG. 2 is a diagram showing an example of a display range R3 of the transmissive display device 101. FIG. 10 is a diagram showing another example of the display range R3 of the transmissive display device 101. FIG. 10 is a diagram showing an example of a display when a part of the first imaginary frame V1-1 is outside the display range R3. FIG. 10 is a diagram showing an example of a display when the proportion of the range RY in the display range R3 is equal to or less than a predetermined value.

A. Embodiment A-1. System Configuration Fig. 1 is a block diagram showing the configuration of a display control system 1 according to an embodiment. The display control system 1 includes a wearable device 10 and a terminal device 20. The wearable device 10 and the terminal device 20 are connected via a communication network N.

Figure 2 is a schematic diagram showing the wearable device 10 in use. As shown in Figure 2, the wearable device 10 is a head-mounted display worn on the head of a first user U1, who is an operator. The wearable device 10 includes a transparent display device 101 (see Figure 4) located in front of the eyes of the first user U1 wearing the wearable device 10. The transparent display device 101 is an example of a first display device. The wearable device 10 has a shape similar to that of ordinary eyeglasses, for example. The wearable device 10 has lenses L and a frame F that supports the lenses L. The lenses L include a left lens positioned in front of the first user U1's left eye and a right lens positioned in front of the user's right eye. The frame F has a bridge between the left and right lenses and a pair of temples that rest on the left and right ears. As described below, the transparent display device 101 displays various images, including virtual objects, on the lenses L. The lenses L are the display range of the transparent display device 101.

In this embodiment, a first user U1 performs wiring work between multiple devices DV1 to DV3 stored in a rack RA while wearing a wearable device 10. The wearable device 10 includes an imaging device 105 (see FIG. 4). The wearable device 10 transmits images captured by the imaging device 105 to the terminal device 20.

Figure 3 is a schematic diagram showing the usage state of the terminal device 20. The terminal device 20 is used by a second user U2 who is an assistant for the work. The terminal device 20 is an information processing terminal such as a personal computer, laptop computer, tablet terminal, or smartphone. The terminal device 20 is equipped with a display device 201. The display device 201 is an example of a second display device. The terminal device 20 displays a captured image transmitted from the wearable device 10 on the display device 201. The captured image shows at least a portion of the field of view R1 (see Figure 6) of the first user U1. Specifically, the captured image shows the space in which the devices DV1 to DV3 are arranged and the first user U1 performing work on the devices DV1 to DV3. By displaying such a captured image on the terminal device 20, the second user U2 can remotely advise and monitor the work performed by the first user U1. Furthermore, for example, by saving the captured images on the terminal device 20 or the wearable device 10, it is possible to verify after the fact whether the work procedures and work contents of the first user U1 were correct, or to investigate the cause of any malfunctions that may occur.

A-2. Wearable device 10
4 is a block diagram showing the configuration of the wearable device 10. The wearable device 10 includes a transparent display device 101, a speaker 102, a microphone 103, a communication device 104, an image capture device 105, a storage device 106, a processing device 107, and a bus 108. At least a part of the configuration shown in FIG. 4 is stored in, for example, a frame F. The transparent display device 101, the speaker 102, the microphone 103, the communication device 104, the image capture device 105, the storage device 106, and the processing device 107 are interconnected by a bus 108 for communicating information. The bus 108 may be configured using a single bus, or may be configured using different buses between elements of the device, etc.

The transmissive display device 101 includes left and right lenses L, a display panel, and optical members. The display panel and optical members are housed in, for example, a frame F. A pair of display panels and optical members may be provided, one on each side, corresponding to the left and right lenses L. The transmissive display device 101 displays a projected image corresponding to a virtual object on the display panel based on control from the processing device 107. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical members guide light emitted from the display panel to the left and right lenses L.

The left and right lenses L each have a half mirror. The half mirrors on the left and right lenses L transmit light representing real space, thereby guiding the light representing real space to the eyes of the first user U1. The half mirrors on the left and right lenses L also reflect light representing virtual objects guided by the optical elements toward the eyes of the first user U1. When the light of real space that has passed through the half mirror and the light representing virtual objects that has been reflected by the half mirror are incident on the eyes of the first user U1, the first user U1 perceives the virtual objects as being located in real space.

The speaker 102 plays audio data and outputs audio corresponding to the audio data. The audio data played by the speaker 102 is, for example, audio data indicating the speech of the second user U2 picked up by the microphone 204 of the terminal device 20. The microphone 103 picks up surrounding audio and generates audio data. The audio picked up by the microphone 103 is, for example, the speech of the first user U1 wearing the wearable device 10. The speaker 102 and the microphone 103 are arranged, for example, in frame F. Alternatively, the speaker 102 and the microphone 103 may not be included in the wearable device 10 but may be separate from the wearable device 10.

The communication device 104 has a communication interface for communicating with other devices. The communication device 104 connects to the communication network N using wireless communication or wired communication and communicates with the terminal device 20 via the communication network N.

The imaging device 105 has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens. The imaging lens is provided, for example, on the bridge described above. Therefore, the imaging device 105 captures an image of the outside world in the direction in which the face of the first user U1 is facing. The imaging optical system may have various optical elements, such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

The imaging device 105 captures an image of a subject and generates captured image data corresponding to the captured image (hereinafter referred to as the "captured image"). As will be described later with reference to FIG. 6, in this embodiment, the imaging range R2 (see FIG. 6) of the imaging device 105 includes the range of the first user U1's field of view R1 that is visible through the lens L, i.e., the display range R3 (see FIG. 6) of the transmissive display device 101. The correspondence between each pixel of the captured image captured by the imaging device 105 and each pixel of the transmissive display device 101 is calibrated in advance. Therefore, for example, taking object A in the captured image as an example, when the first user U1 views real space through the transmissive display device 101, the area that object A occupies within the display range R3 of the transmissive display device 101 is known. Therefore, the first display control unit 111 (described later) can project a virtual object, the display position of which is determined based on the captured image, onto the transmissive display device 101.

The captured image data generated by the imaging device 105 is transmitted to the terminal device 20 via the communication device 104. The imaging device 105 repeatedly captures images at a predetermined interval and transmits the generated captured image data to the terminal device 20 each time an image is captured. In this embodiment, the captured image data is assumed to be video data.

The storage device 106 is a recording medium readable by the processing device 107. The storage device 106 includes, for example, non-volatile memory and volatile memory. The non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, RAM (Random Access Memory). The storage device 106 stores the program PG1. The program PG1 is a program for operating the wearable device 10.

The processing device 107 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

The processing device 107 reads the program PG1 from the storage device 106. By executing the program PG1, the processing device 107 functions as a first display control unit 111 and a first communication control unit 112. At least one of the first display control unit 111 and the first communication control unit 112 may be configured with circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). Details of the first display control unit 111 and the first communication control unit 112 will be described later.

A-3. Terminal device 20
5 is a block diagram showing the configuration of terminal device 20. Terminal device 20 includes a display device 201, an input device 202, a speaker 203, a microphone 204, a communication device 205, a storage device 206, a processing device 207, and a bus 208. Display device 201, input device 202, speaker 203, microphone 204, communication device 205, storage device 206, and processing device 207 are interconnected by bus 208 for communicating information. Bus 208 may be configured using a single bus, or may be configured using different buses for each device.

The display device 201 is a display device (e.g., various display panels such as a liquid crystal panel or an organic EL panel) that displays information to the outside. The input device 202 is an input device (e.g., a keyboard, mouse, switch, button, sensor, etc.) that accepts input from the outside. The display device 201 and the input device 202 may also be integrated into one structure (e.g., a touch panel).

The speaker 203 plays audio data and outputs audio corresponding to the audio data. The audio data played by the speaker 203 is, for example, audio data indicating the speech of the first user U1 picked up by the microphone 103 of the wearable device 10. The microphone 204 picks up surrounding audio and generates audio data. The audio picked up by the microphone 204 is, for example, the speech of the second user U2 using the terminal device 20. The speaker 203 and the microphone 204 may not be included in the terminal device 20 and may be separate from the terminal device 20.

The communication device 205 has a communication interface for communicating with other devices. The communication device 104 connects to the communication network N using wireless or wired communication and communicates with the wearable device 10 via the communication network N.

The storage device 206 is a recording medium readable by the processing device 207. The storage device 206 includes, for example, non-volatile memory and volatile memory. The non-volatile memory is, for example, ROM, EPROM, and EEPROM. The volatile memory is, for example, RAM. The storage device 206 stores the program PG2. The program PG2 is a program for operating the terminal device 20.

The processing unit 207 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

The processing device 207 reads the program PG2 from the storage device 206. By executing the program PG2, the processing device 207 functions as a second communication control unit 211, a second display control unit 212, and a reception unit 213. At least one of the second communication control unit 211, the second display control unit 212, and the reception unit 213 may be configured by a circuit such as a DSP, ASIC, PLD, or FPGA. Details of the second communication control unit 211, the second display control unit 212, and the reception unit 213 will be described later.

A-4. Details of the Processing Device 107 and the Processing Device 207 The processing device 107 and the processing device 207 will now be described in detail. Prior to this, the relationship between the field of view R1 of the first user U1, the imaging range R2 of the imaging device 105, and the display range R3 of the transmissive display device 101 will be described. FIG. 6 is an explanatory diagram schematically showing ranges related to the field of view R1 of the first user U1. Note that the ranges R1 to R3 shown in FIG. 6 are intended to show the relative size of each range and differ from the actual field of view angle. In FIG. 6, the symbol E schematically indicates the eyes of the first user U1. It is generally known that the field of view of a human, i.e., the field of view R1 of the first user U1, is approximately 200° horizontally (left-right) and approximately 130° vertically (up-down).

In contrast, the imaging range R2 of the imaging device 105 is approximately 100° horizontally and 80° vertically when the imaging lens of the imaging device 105 is a wide-angle lens. Therefore, the imaging range R2 of the imaging device 105 is narrower than the field of view R1 of the first user U1. Note that this does not apply, for example, if a special lens such as a fisheye lens is used as the imaging lens of the imaging device 105. Furthermore, strictly speaking, the starting point of the imaging range R2 is at a different position from the eye E of the first user U1. However, as described above, the imaging lens of the imaging device 105 is provided on the bridge, and the starting point of the imaging range R2 and the eye E of the first user U1 can be considered to be at approximately the same position. Furthermore, because the wearable device 10 is worn on the head of the first user U1, the range of real space captured in the captured image changes depending on the movement of the first user U1. Therefore, the captured image of the imaging device 105 captures the real space included in the field of view R1 of the first user U1.

The display range R3 of the transmissive display device 101 is the range of the field of view R1 of the first user U1 that is visible through the lens L. As described above, the transmissive display device 101 allows the first user U1 to perceive a virtual object by allowing light from real space that has passed through a half mirror provided in the lens L and light representing the virtual object that has been reflected by the half mirror to enter the user's eyes. Therefore, the range of real space in which the transmissive display device 101 can display a virtual object is the range that the first user U1 can see through the lens L. In this embodiment, the display range R3 of the transmissive display device 101 is narrower than the field of view R1 of the first user U1. However, it is also possible to make the display range R3 of the transmissive display device 101 equal to the field of view R1 of the first user U1 by, for example, curving the shape of the lens L. In addition, in this embodiment, the display range R3 of the transmissive display device 101 is narrower than the imaging range R2 of the imaging device 105, but as described above, if the display range R3 of the transmissive display device 101 is made wider, it is also possible to make the display range R3 of the transmissive display device 101 larger than the imaging range R2 of the imaging device 105.

Next, we will explain the processing device 107 of the wearable device 10. As described above, the processing device 107 functions as the first display control unit 111 and the first communication control unit 112. The first display control unit 111 controls the transparent display device 101 located in front of the eyes of the first user U1.

The first communication control unit 112 controls the transmission and reception of data by the communication device 104. The first communication control unit 112 is an example of a transmitting unit and a receiving unit. The first communication control unit 112 transmits, for example, a captured image of the real space included in the field of view of the first user U1 (more specifically, captured image data corresponding to the captured image) to the terminal device 20.

The processing device 207 of the terminal device 20 functions as a second communication control unit 211, a second display control unit 212, and a reception unit 213. The second communication control unit 211 controls the transmission and reception of data by the communication device 205. The second communication control unit 211 receives, for example, a captured image (more specifically, captured image data corresponding to the captured image) from the wearable device 10.

The second display control unit 212 controls the display device 201. For example, the second display control unit 212 displays a captured image on the display device 201. The reception unit 213 receives designation of a specific area RX in the captured image displayed on the display device 201. The specific area RX is an example of a first area. The specific area RX may be any area, but in this embodiment, it is a particularly important area in the captured image that is desired to be continuously included in the captured image. The reception unit 213 receives designation of the specific area RX, for example, by receiving drawing of a frame indicating the outer edge of the specific area RX for the captured image displayed on the display device 201. This frame is drawn using, for example, the input device 202.

7A and 7B are diagrams showing an example of a captured image displayed on the display device 201. The captured image M1 shown in FIG. 7A shows an area in real space that includes devices DV1 and DV2. For example, assume that the second user U2 needs to monitor the status of device DV1 (e.g., the meter display and connector connection status) while the first user U1 is working. Therefore, the second user U2 designates the area in the captured image M1 that includes device DV1 as a specific area RX. Specifically, the reception unit 213 displays an area designation button B1 superimposed on the captured image M1. When the second user U2 presses the area designation button B1, the reception unit 213 enters an area designation mode that accepts the designation of the specific area RX, and displays a cursor C resembling a pen superimposed on the captured image M1, as shown in FIG. 7B. The second user U2 moves the cursor C using, for example, a mouse, which is an example of the input device 202, to draw a frame W that surrounds device DV1. When the second user U2 presses the OK button B2, the reception unit 213 receives the area inside the frame W as the specific area RX. The reception unit 213 generates area information that identifies the specific area RX. The area information is, for example, position information (e.g., coordinate values) of each point of the frame W on the captured image M1. The second communication control unit 211 transmits the area information that identifies the position of the specific area RX in the captured image M1 to the wearable device 10.

When the wearable device 10 receives area information from the terminal device 20, the first display control unit 111 displays a first virtual object V1 indicating a range in real space that corresponds to a specific area RX on the transmissive display device 101 based on the area information. The first virtual object V1 includes, for example, a first virtual frame V1-1 and a message image V1-2. The first virtual frame V1-1 is an example of a frame and a first frame. The first virtual frame V1-1 indicates a position in real space that corresponds to the outer edge of the specific area RX. The message image V1-2 includes a message urging the first user U1 to maintain a state in which the first virtual frame V1-1 is included in the display range R3 of the transmissive display device 101. The message image V1-2 is an example of a first message. Note that only the first virtual frame V1-1 may be displayed as the first virtual object V1.

Figures 8A and 8B are diagrams showing an example of the display range R3 of the transmissive display device 101. As described above, the display range R3 of the transmissive display device 101 is the range of the field of view R1 of the first user U1 that is visible through the lens L. Furthermore, the display range R3 of the transmissive display device 101 is narrower than the imaging range R2 of the imaging device 105. In the example of Figures 8A and 8B, the display range R3 of the transmissive display device 101 includes the entire device DV1 and a portion of the device DV2.

Until the second user U2 specifies a specific area RX, only real space is visible in the display range R3, as shown in Figure 8A. Once the second user U2 specifies a specific area RX, a first virtual frame V1-1, which is an example of the first virtual object V1, is displayed on the lens L, as shown in Figure 8B. The first virtual frame V1-1 is a frame of the same shape as the frame W drawn by the second user U2. The first virtual frame V1-1 indicates the outer edge of the range RY in real space that corresponds to the specific area RX in the captured image.

As described above, the correspondence between each pixel of the captured image captured by the imaging device 105 and each pixel of the transmissive display device 101 has been calibrated in advance. Therefore, the first display control unit 111 can display on the transmissive display device 101 a first virtual frame V1-1 that corresponds to the frame W drawn on the captured image.

Figure 9 is a diagram showing another example of the display range R3 of the transmissive display device 101. Figure 8B shows the display range R3 when the first user U1 is positioned directly in front of the device DV1. Figure 9 shows the display range R3 when the first user U1 is positioned diagonally to the side of the device DV1. As described above, the first virtual frame V1-1 is positioned in real space on the outer edge of the range RY that corresponds to the specific region RX of the captured image. Therefore, as shown in Figure 9, when the first user U1 is looking at the device DV1 from diagonally to the side, the first virtual frame V1-1 is also drawn as if looking at it from diagonally to the side.

Also, as shown in FIG. 8B, the first display control unit 111 displays a message image V1-2 as the first virtual object V1. The message image V1-2 includes a message urging the first user U1 to adjust their posture so that the entire first virtual frame V1-1 is within the display range R3. If the entire first virtual frame V1-1 is included in the display range R3, the specific region RX is included in the image captured by the imaging device 105. On the other hand, if part or all of the first virtual frame V1-1 is outside the display range R3, the image captured by the imaging device 105 may not include part or all of the specific region RX. The first user U1 performs work while adjusting the orientation of their face so that the first virtual frame V1-1 always fits within the display range R3. Therefore, the specific region RX is always included in the image captured by the imaging device 105. In addition to displaying the first virtual object V1, audio may be output from the speaker 102 to inform the first user U1 that the specific region RX has been set.

Figure 10 is a diagram showing an example of a display when part of the first virtual frame V1-1 falls outside the display range R3. For example, the first user U1 may be so focused on their work that they do not notice that the first virtual frame V1-1 has fallen outside the display range R3. In the example of Figure 10, the top edge of the rectangular first virtual frame V1-1 falls outside the display range R3. When at least part of the first virtual object V1 falls outside the display range R3 of the transmissive display device 101 in this way, the first display control unit 111 further displays a second virtual object V2 to encourage the first user U1 to move.

In the example of FIG. 10, the second virtual object V2 includes a message image V2-1 and an arrow image V2-2. The message image V2-1 includes a message urging the first user U1 to turn their face upwards so that the entire frame (first virtual frame V1-1) is within the lens L. The arrow image V2-2 indicates the direction in which the first user U1 should move their face. The arrow indicated by the arrow image V2-2 is an example of a symbol indicating the direction in which the first user should move. Note that the second virtual object V2 does not necessarily have to display both the message image V2-1 and the arrow image V2-2; it is sufficient for either one of them to be displayed. In other words, the second virtual object V2 displays at least one of the message image V2-1 urging the first user U1 to move so that the first virtual frame V1-1 is included in the display range R3 of the transmissive display device 101, and the arrow image V2-2 indicating the direction in which the first user U1 should move.

Along with the second virtual object V2, audio alerting the first user U1 may be output from the speaker 102. It is predicted that the first user U1, upon viewing the second virtual object V2, will move his or her face so that the entire first virtual frame V1-1 is within the lens L. Therefore, the imaging range of the imaging device 105 can be adjusted so that the entire specific area RX is captured in the captured image.

Note that, as shown in FIG. 6, in this embodiment, the imaging range R2 of the imaging device 105 is wider than the display range R3 of the transmissive display device 101. Therefore, even if part of the first virtual frame V1-1 falls outside the display range R3, there is a possibility that the specific area RX will not be missing from the captured image. On the other hand, by displaying the second virtual object V2 at the point when part of the first virtual frame V1-1 falls outside the display range R3, it is possible to prevent the specific area RX from being missing from the captured image. Therefore, the entire specific area RX can be captured more reliably.

Furthermore, even if the entire first virtual frame V1-1 is within the display range R3, if the first user U1 is located far from the device DV1, the range of the specific area RX shown in the captured image will be smaller. In this case, the second user U2 may have difficulty viewing the specific area RX and may not be able to obtain the necessary information. For this reason, the first display control unit 111 further displays a third virtual object V3 urging the first user U1 to move when the proportion of the range RY in real space corresponding to the specific area RX in the display range R3 falls below a predetermined value. The above-mentioned predetermined value can also be considered the proportion of the specific area RX that occupies the captured image. The predetermined value may be determined by the second user U2 or the first user U1, or by the wearable device 10 (e.g., the first display control unit 111).

Figure 11 is a diagram showing an example of a display when the proportion of range RY in display range R3 is equal to or less than a predetermined value. The first display control unit 111 displays a first virtual object V1 and a third virtual object V3, for example, as shown in Figure 11. The first virtual object V1 is a first virtual frame V1-1. The first virtual frame V1-1 indicates a position in real space corresponding to the outer edge of a specific area RX. The first virtual frame V1-1 is displayed. The third virtual object V3 includes a second virtual frame V3-1 that surrounds the first virtual frame V1-1, and a message image V3-2 that includes a message urging the first user U1 to move so that the second virtual frame V3-1 is outside the display range R3 of the transmissive display device 101. The second virtual frame V3-1 is an example of a second frame. The message image V3-2 is an example of a second message.

The second virtual frame V3-1 is displayed at a predetermined distance outside the first virtual frame V1-1. The area outside the second virtual frame V3-1 is, so to speak, an area that should not be visible to the first user U1. The second virtual frame V3-1 is displayed, for example, in a different display mode (for example, a different color or a different line type) than the frame of the first virtual frame V1-1. The message image V3-2 displays a message urging the first user U1 to move (forward) until the dotted frame (second virtual frame V3-1) is no longer visible.

It is not necessary for all of the second virtual frame V3-1, message image V3-2, and arrow image V3-3 to be displayed as the third virtual object; it is sufficient for at least one of these to be displayed. Furthermore, for example, instead of the second virtual frame V3-1, an image that fills in (masks) the outside of the second virtual frame V3-1 may be displayed. Furthermore, along with the display of the third virtual object V3, audio may be output from the speaker 102 to alert the first user U1.

A-5. Summary of the Embodiment As described above, according to the embodiment, the display control system 1 transmits a captured image captured by the wearable device 10 to the terminal device 20, and accepts designation of a specific area RX on the captured image at the terminal device 20. Furthermore, the display control system 1 uses the transmissive display device 101 of the wearable device 10 to display a first virtual object V1 indicating a range in real space that corresponds to the specific area RX in the field of view R1 of the first user U1. Thus, the first user U1 can grasp the range in real space that corresponds to the specific area RX, and can efficiently capture an image that depicts the range in real space that corresponds to the specific area RX.

The display control system 1 also displays, as the first virtual object V1, a first virtual frame V1-1 indicating a position in real space corresponding to the outer edge of the specific region RX, and a message image V1-2 including a message urging the first user U1 to maintain the first virtual frame V1-1 within the display range R3 of the transmissive display device 101. Therefore, the first user U1 can intuitively grasp the position in real space corresponding to the outer edge of the specific region RX and appropriately maintain their own posture. Furthermore, by visually recognizing the message image V1-2, the first user U1 can understand that the first virtual frame V1-1 should be maintained within the display range R3. Therefore, for example, even if the first user U1 is unfamiliar with using the wearable device 10 or if many virtual objects are displayed within the display range R3, they can easily capture an image depicting the range in real space corresponding to the specific region RX.

In addition, the display control system 1 further displays a second virtual object V2 that prompts the first user U1 to move when at least a portion of the first virtual frame V1-1 falls outside the display range R3. Thus, the first user U1 can notice that a portion of the first virtual frame V1-1 has fallen outside the display range R3, even if, for example, he or she is concentrating on work.

In addition, the display control system 1 displays, as the second virtual object V2, at least one of a message image V2-1 containing a message urging the first user U1 to move so that the first virtual frame V1-1 is included in the display range R3, or an arrow image V2-2 indicating the direction in which the first user U1 should move. Therefore, the first user U1 can specifically grasp the movements he or she should make, allowing him or her to work efficiently.

Furthermore, the display control system 1 further displays a third virtual object V3 that prompts the first user U1 to move when the proportion of the range RY in real space corresponding to the specific area RX in the display range R3 falls below a predetermined value. This prevents the specific area RX in the captured image from becoming too small, allowing the second user U2 to more efficiently assist with their work.

The display control system 1 also displays, as the third virtual object V3, a second virtual frame V3-1 surrounding the first virtual frame V1-1 and a message image V3-2 containing a message urging the first user U1 to move so that the second virtual frame V3-1 is outside the display range R3. Therefore, the first user U1 can specifically grasp the actions he or she should take and can work efficiently.

B: Modifications Modifications of the above-described embodiment are shown below. Two or more of the following modifications may be arbitrarily selected and combined as long as they are not mutually contradictory.

B1: First Modification In the above-described embodiment, the wearable device 10 is configured only with an eyeglass-type terminal device. However, the wearable device 10 may also include an eyeglass-type terminal device and a portable terminal device such as a smartphone, a tablet device, or a laptop computer. In this case, the eyeglass-type terminal device and the portable terminal device are connected to each other. When the wearable device 10 includes a portable terminal device, the portable terminal device may be responsible for some or all of the first display control unit 111 and the first communication control unit 112.

C: Other (1) Each function illustrated in Figure 4 or Figure 5 is realized by any combination of hardware and software. There are no particular limitations on how each function is realized. Each function may be realized using a single device that is physically or logically coupled, or may be realized using a device that is configured by connecting two or more physically or logically separated devices directly or indirectly (for example, using wires, wirelessly, etc.). Each function may be realized by combining software with the single device or the multiple devices.

(2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

(3) In each of the embodiments and the first variant, the storage device 106 and the storage device 206 may be configured by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. The program may also be transmitted from a network via a telecommunications line.

(4) Each of the embodiment and the first variant example is applicable to LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (x is, for example, an integer or a decimal point), FRA (Future Radio The present invention may be applied to at least one of systems utilizing IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wideband), Bluetooth, other suitable systems, and next generation systems that are extended, modified, created, or defined based on these. In addition, multiple systems may be applied in combination (for example, a combination of at least one of LTE and LTE-A with 5G, etc.).

(5) The order of the process procedures, sequences, flowcharts, etc. illustrated in each of the embodiments and the first variant may be changed as long as it is not inconsistent. For example, the methods described herein present elements of various steps in an exemplary order and are not limited to the particular order presented.

(6) In each of the embodiments and the first variant, input and output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. Input and output information, etc. may be overwritten, updated, or added to. Output information, etc. may be deleted. Input information, etc. may be transmitted to another device.

(7) In each of the embodiments and the first variant, the determination may be based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values (e.g., a comparison with a predetermined value).

(8) The programs exemplified in each of the embodiments and the first variant should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names. Furthermore, software, instructions, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared, microwave), at least one of these wired technology and wireless technology is included within the definition of a transmission medium.

(9) The information described in each of the embodiments and the first variant may be represented using any of a variety of different technologies. For example, data, information, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(10) In each of the embodiments and the first variant, the terms "system" and "network" are used interchangeably.

(11) In each of the embodiments and the first variant, the terminal device 20 may be a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

(12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, etc. A mobile station may be a device mounted on a mobile body, or the mobile body itself, etc. A mobile body refers to an object that can move. A mobile body can move at any speed. A mobile body can stop. Examples of mobile bodies include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile body may also be a mobile body that moves autonomously based on operation commands. The mobile object may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). The mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

(13) In each of the embodiments and the first variant, the term "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), and considering something ascertained to be "determined." Furthermore, "determining" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, or accessing (e.g., accessing data in memory) as having been "judgment" or "decided." Furthermore, "determining" may include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "decided." In other words, "determining" may include regarding some action as having been "decided." Furthermore, "determining" may be interpreted as "assuming," "expecting," "considering," etc.

(14) In each of the embodiments and the first variant, the term "connected," or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

(15) In each of the embodiments and the first variant, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

(16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

(17) When the words "include," "including," and variations thereof are used in this specification or the claims in each of the embodiments and the first variant, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this specification or the claims is not intended to be an exclusive or.

(18) Throughout this application, where articles are added by translation, such as a, an, and the in English, the disclosure may include the plural form of the noun following these articles.

(19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the descriptions in this specification are for illustrative purposes only and do not have any limiting meaning on the present invention. Furthermore, multiple aspects selected from the aspects exemplified in this specification may be combined.

1...display control system, 10...wearable device, 20...terminal device, 101...transmissive display device, 104...communication device, 105...imaging device, 106...storage device, 107...processing device, 111...first display control unit, 112...first communication control unit, 201...display device, 202...input device, 205...communication device, 206...storage device, 207...processing device, 211...second communication control unit, 212...second display control unit, 213...reception unit, F...frame, L...lens, N...communication network.

Claims (6)

A display control system including a wearable device and a terminal device,
The wearable device
a first display control unit that controls a first transmissive display device located in front of the first user;
a first communication control unit that transmits to the terminal device a captured image of a real space included in a visual field of the first user;
The terminal device
a second communication control unit that receives the captured image;
a second display control unit that displays the captured image on a second display device;
a receiving unit that receives, from a second user, a designation of a first area in the captured image displayed on the second display device,
the second communication control unit transmits area information that identifies a range of the first area in the captured image to the wearable device;
the first display control unit displays, on the first display device, a first virtual object indicating a range in the real space corresponding to the first region, based on the region information;
a range of the real space that is captured in the captured image changes depending on a movement of the first user;
the first display control unit further displays a third virtual object that prompts the first user to move when a ratio of a range in the real space corresponding to the first area to a display range of the first display device becomes equal to or less than a predetermined value.
Display control system. the first virtual object is a first frame,
the first frame indicates a position in the real space corresponding to an outer edge of the first area,
the third virtual object includes a second frame surrounding the first frame, and a second message prompting the first user to move the second frame out of a display range of the first display device;
The display control system according to claim 1 . the first virtual object includes a frame and a first message;
the frame indicates a position in the real space corresponding to an outer edge of the first area,
the message prompts the first user to maintain a state in which the frame is included in a display range of the first display device.
The display control system according to claim 1 . the first display control unit further displays a second virtual object that prompts the first user to move when at least a part of the first virtual object is out of a display range of the first display device.
The display control system according to claim 1 . the second virtual object includes at least one of a message prompting the first user to move so that the first virtual object is included in a display range of the first display device and a symbol indicating a direction in which the first user should move;
The display control system according to claim 4 . a first display control unit that controls a first transmissive display device located in front of the first user;
a transmitting unit that transmits to a terminal device a captured image of a real space included in a field of view of the first user;
a receiving unit that receives area information that identifies a range of a first area designated in the captured image,
the first display control unit displays, on the first display device, a first virtual object indicating a range in the real space corresponding to the first region, based on the region information;
a range of the real space that is captured in the captured image changes depending on a movement of the first user;
the first display control unit further displays a third virtual object that prompts the first user to move when a ratio of a range in the real space corresponding to the first area to a display range of the first display device becomes equal to or less than a predetermined value.
Wearable devices.