JP7646403B2 - IMAGE DISPLAY CONTROL DEVICE, IMAGE DISPLAY CONTROL SYSTEM, AND IMAGE DISPLAY CONTROL METHOD - Google Patents

Description

The present invention relates to an image display control device, an image display control system, and an image display control method.

A method for sharing gaze information in a system including multiple information input/output terminals connected via a communication line is known in which a first terminal captures an image of the field of view in front of a user using the first terminal, and a second terminal displays the image captured by the first terminal (Patent Document 1).

The technology in Patent Document 1 detects the gaze direction of the user of the second terminal with respect to the image displayed on the second terminal to determine the gaze position, calculates the display position of a marker indicating the gaze position in the forward field of view of the user of the first terminal, and displays the marker superimposed at the calculated display position.

JP 2018-182570 A

In the technology of Patent Document 1, the visual range of the user of the first terminal with respect to the outside world is the same as the visual range of the user of the second terminal with respect to the same outside world, so the gaze position of the user of the second terminal can be displayed in the visual range of the user of the first terminal. However, if the visual range of the user of the first terminal with respect to the outside world is different from the visual range of the user of the second terminal with respect to the same outside world, the gaze position of the user of the second terminal cannot be displayed in the visual range of the user of the first terminal, resulting in a problem in that accurate communication cannot be carried out.

The problem that the present invention aims to solve is to provide an image display control device, an image display control system, and an image display control method that enable accurate communication between users of information input/output terminals even when the visual range of the user of an information input/output terminal with respect to the outside world differs from the visual range of a user of another information input/output terminal with respect to the same outside world.

The present invention solves the above problem by acquiring vehicle-side information about the interior and/or exterior of a vehicle, transmitting a control signal to a first display device for displaying a vehicle-side image representing the interior and/or exterior of the vehicle, generated based on the acquired vehicle-side information, to a first user outside the vehicle via a first display device, acquiring first gaze point position information of a first gaze point at which the first user is gazing in the vehicle-side image, generating first gaze point display information for displaying the first gaze point in the vehicle-side image to a second user based on the first gaze point position information when the field of view when the first user is gazing differs from the field of view when a second user in a different space from the first user is gazing, and transmitting a control signal to the second display device for displaying the generated first gaze point display information to the second user via a second display device.

According to the present invention, even if the visual range of a user of an information input/output terminal with respect to the outside world differs from the visual range of a user of another information input/output terminal with respect to the same outside world, it is possible for users of the information input/output terminals to communicate accurately with each other.

1 is a block diagram showing a configuration of an image display control system according to a first embodiment. 11 is a diagram showing an example of a first gaze point and a field of view range of a first user in a vehicle-side image. FIG. 11 is a diagram showing an example of the scenery inside and outside the vehicle interior that is seen in the field of view of a second user. 11 is a diagram showing an example of the scenery inside and outside the vehicle interior that is seen in the field of view of a second user. 4 is a flowchart showing the procedure of an image display control method according to the present embodiment. 10 is a flowchart showing an example of a procedure for acquiring first gaze point position information. FIG. 11 is a block diagram showing a configuration of an image display control system according to a second embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an image display control system including an image display control device according to this embodiment. The image display control system 1 includes an image display control device 10, a first output device 11, and a second output device 12. The image display control device 10 is a device that controls the input and output of image information and audio information in the second output device 12 provided in the cabin of the vehicle 2 and the first output device 11 provided in the space outside the cabin of the vehicle 2. Specifically, the image display control device 10 acquires vehicle-side information inside and/or outside the cabin of the vehicle 2, and outputs the vehicle-side information to a first user via the first output device 11. The first user is outside the cabin of the vehicle 2 and uses the first output device 11. The first user can see the scenery inside and outside the cabin of the vehicle 2 by outputting the vehicle-side image from the first output device 11. For example, the vehicle-side image is a virtual reality (VR) image that represents a virtual space corresponding to the vehicle space of the vehicle 2. The image display control device 10 presents a virtual reality image on the display screen of a first output device 11, such as VR goggles worn by the first user. In other words, the first user is presented with a vehicle-side image that corresponds to the line of sight of the first user in accordance with the movement of the head, and is presented with a virtual reality as if the first user were inside the cabin of the vehicle 2.

The image display control device 10 also outputs first gaze point information related to the first gaze point that the first user is gazing at in the vehicle-side image to the second user via the second output device 12. This allows the second user to know where the first user is looking in the scenery inside and outside the vehicle 2 displayed as an image. The first output device 11 and the second output device 12 each have a microphone and a speaker, and acquire the user's voice information through the microphone and output the other user's voice information through the speaker. The image display control device 10 acquires voice information from each of the first output device and the second output device, and outputs the voice information to each of the first output device and the second output device, so that the first user and the second user can communicate with each other. In this embodiment, the second user is a passenger in the vehicle 2, but this is not limited to this, and the second user may be a user who is outside the vehicle 2 and shares the vehicle-side image via the second output device 12. The second output device may be a device installed outside the cabin of the vehicle 2, or a mobile terminal of the second user. In this embodiment, the image display control device 10 may be a device installed outside the cabin of the vehicle 2, or a device installed inside the cabin of the vehicle 2.

The image display control device 10 includes a controller 100, a candidate gaze point storage device 120, and a communication device 110. The controller 100 includes a computer having hardware and software, which includes a ROM (Read Only Memory) that stores a program, a CPU (Central Processing Unit) that executes the program stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. As the operating circuit, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. can be used instead of or in addition to the CPU. The controller 100 includes at least a vehicle-side information acquisition unit 101, a vehicle-side image display control unit 102, a remote information acquisition unit 103, and a presentation information output control unit 104 as functional blocks. In this embodiment, the functions of the controller 100 are divided into four blocks, and the functions of each functional block are explained; however, the functions of the controller 100 do not necessarily have to be divided into four blocks, and may be divided into three or less functional blocks, or five or more functional blocks.

The vehicle-side information acquisition unit 101 acquires vehicle interior information inside the vehicle cabin of the vehicle 2 and vehicle exterior information outside the vehicle cabin as vehicle-side information. The vehicle interior information includes a vehicle interior image representing the vehicle interior captured by the in-vehicle camera 15. The vehicle interior image includes an image of the second user riding in the vehicle 2 and an image of the in-vehicle device 13. The in-vehicle device 13 includes an IVI (In Vehicle Infotainment), meters, air conditioning, etc. mounted on the vehicle 2. The vehicle-side information acquisition unit 101 also acquires information regarding the posture of the second user estimated from the image of the second user by the posture detection device 14. The vehicle-side information acquisition unit 101 may also acquire information regarding the operating status of the in-vehicle device 13 from the in-vehicle device 13. The vehicle exterior information includes a vehicle exterior image representing the outside of the vehicle cabin captured by the exterior camera 16. The vehicle-side information acquisition unit 101 may also acquire vehicle model information of the vehicle 2, for example, information regarding the interior of the vehicle 2. The vehicle-side image display control unit 102 generates a vehicle-side image representing the inside and/or outside of the vehicle based on the vehicle-side information, and transmits a control signal to display the vehicle-side image to the first user via the first output device 11. Specifically, the vehicle-side image display control unit 102 first generates such a control signal. For example, the vehicle-side image display control unit 102 generates a control signal to generate the vehicle-side image as an image representing a virtual reality space corresponding to the vehicle space. Then, the vehicle-side image display control unit 102 transmits the vehicle-side information and the generated control signal to the first output device 11. In this embodiment, the vehicle-side image display control unit 102 may generate a vehicle-side image based on the vehicle-side information, and transmit a control signal to display the generated vehicle-side image to the first user via the first output device 11.

The remote information acquisition unit 103 acquires, from the first output device 11, first gaze point position information of a first gaze point at which the first user is gazing in the vehicle-side image. The first gaze point position information is information of position coordinates on the vehicle-side image. The remote information acquisition unit 103 also acquires voice information of the first user from the first output device. The remote information acquisition unit 103 also acquires information on a specific part of the first user's body and shared information displayed by the first user's operation. The information on the specific part of the first user's body includes position information on the specific part of the first user's body and body part information indicating the specific part of the body. The shared information also includes image information of a virtual object displayed in the virtual space. The virtual object is an object handled by the first user, and represents, for example, an operation manual or a map.

The presentation information output control unit 104 generates first gaze point display information for displaying the first gaze point to the second user based on the first gaze point position information. The first gaze point display information is information that visually represents the position of the first gaze point in the vehicle-side image. First, the presentation information output control unit 104 acquires a predetermined range from the first gaze point of the first user as a field of view range, and acquires a predetermined range from the gaze point of the second user as a field of view range. The field of view range is the range of scenery visible to the user's eyes when the user is gazing at the gaze point. The field of view range is a range that includes the gaze point. The predetermined range is set to, for example, the range of a general human field of view. Next, the presentation information output control unit 104 compares the field of view range of the first user with the field of view range of the second user. In this embodiment, the field of view range for the outside world, that is, the visible scenery, changes when the first user and the second user each move the direction of their heads. The second user who is aboard the vehicle 2 can freely see inside and outside the vehicle by moving the direction of his or her head. In addition, since the image of the virtual space of the vehicle 2 is displayed via the first output device 11, the first user can freely view the interior and exterior of the vehicle 2. Therefore, the first user and the second user may view different scenery from the scenery inside and outside the vehicle. That is, the first user and the second user may have different viewing ranges for the scenery inside and outside the vehicle. For example, when the first user is gazing at the scenery ahead of the vehicle 2 and the second user is gazing at the scenery to the side of the vehicle 2, the viewing range of the first user and the viewing range of the second user are different. The presentation information output control unit 104 compares the viewing range of the first user with the viewing range of the second user, and generates first gaze point display information when the viewing range when the first user is gazing is different from the viewing range when the second user is gazing. Specifically, the case where the visual field range when the first user is gazing is different from the visual field range when the coordinate information of the visual field range of the first user is compared with the coordinate information of the visual field range of the second user, and the ratio of the overlapping portion of the visual field range of the first user and the visual field range of the second user is less than a predetermined value with respect to the visual field range of the first user or the visual field range of the second user. In addition, the case where the visual field range of the first user and the visual field range of the second user do not overlap (the ratio of overlapping portion is 0%) is also included as a case where the visual field range of the first user is different from the visual field range of the second user.

Then, the presentation information output control unit 104 generates a first gaze point image, which is an image of the peripheral area of the first gaze point position extracted from the vehicle-side image based on the first gaze point position information and the vehicle-side image, as the first gaze point display information. For example, the first gaze point image is a vehicle-side image within a predetermined range from the first gaze point position. Also, the presentation information output control unit 104 generates an indication image showing the peripheral area of the first gaze point of the first user in the vehicle-side image based on the first gaze point position information. The indication image is an image showing the direction of the first gaze point position relative to the display position of the indication image. For example, when the indication image is displayed on the display of the second output device 12, the presentation information output control unit 104 calculates the direction of the first gaze point position relative to the position of the second output device 12 based on the position information of the second output device 12 and the first gaze point position information, and generates an image of an arrow showing the direction of the first gaze point position or an image of text information showing the direction of the first gaze point position as the indication image. The instruction image is displayed in the field of view of the second user via the second output device 12, or is superimposed on the vehicle-side image. In this embodiment, the first output device 11 transmits first gaze point position information to the image display control device 10, and the image display control device 10 generates the first gaze point image from the vehicle-side image, so that image information with a large amount of data does not need to be transmitted from the first output device.

The presentation information output control unit 104 may generate a first gaze point display image in which an image of a predetermined part of the first user's body is superimposed on the first gaze point image. The predetermined part of the first user's body is, for example, an arm or a hand. The presentation information output control unit 104 first determines whether or not the predetermined part of the first user's body is within the field of view of the first user based on the position information of the predetermined part of the first user's body acquired from the posture information of the first user and the first gaze point position information. If it is determined that the predetermined part of the first user's body is within the field of view, the presentation information output control unit 104 generates a first gaze point image by superimposing a virtual object of a hand on the position of the predetermined part of the first user's body in the first gaze point image. This allows the gestures made by the first user's body to be shared with the second user, so that the second user can understand what the first user is pointing to. Furthermore, when a virtual object of an object handled by the first user is displayed within the field of view of the first user, the presentation information output control unit 104 generates a first gaze point image by superimposing an image of the virtual object at the position of the virtual object in the first gaze point image.

The presentation information output control unit 104 transmits a control signal to the second output device 12 to display the generated first gaze point display information to the second user via the second output device 12. Specifically, the presentation information output control unit 104 generates such a control signal and transmits the generated control signal to the second output device 12. In addition, when the second output device 12 is a plurality of monitors in the cabin of the vehicle 2, the presentation information output control unit 104 determines the second output device 12 that displays the first gaze point display information among the plurality of second output devices 12, and displays the first gaze point display information on the determined second output device 12. For example, the presentation information output control unit 104 acquires the line of sight of the second user and selects the second output device 12 that is closest to the line of sight of the second user. This allows the second user to see the second output device 12 without changing the line of sight. In addition, the presentation information output control unit 104 selects the second output device 12 that is closest to the first gaze point. This allows the second user to immediately see the actual scenery reflected at the position of the first gaze point after seeing the first gaze point position image displayed on the second output device 12. In addition, the presentation information output control unit 104 transmits a control signal to the second output device 12 to output the voice information of the first user acquired by the remote information acquisition unit 103 to the second user via the second output device 12.

The candidate gaze point memory device 120 pre-stores candidate gaze points that are candidates for the first gaze point. The candidate gaze points are positions at which there is an object in the vehicle-side image that may be gazed upon by the user. The candidate gaze points include an indoor candidate gaze point located inside the vehicle cabin and an outdoor candidate gaze point located outside the vehicle cabin. For example, the indoor candidate gaze point is the position of an in-vehicle device such as a meter screen on an instrument panel inside the vehicle cabin. For example, in the vehicle cabin, the user often directs his or her gaze to the vehicle speed meter or the screen of the car navigation system, etc., so the candidate gaze point memory device 120 pre-stores such objects. In addition, for example, the outdoor candidate gaze point is the position of a landmark that is a POI (Point of Interest). A POI is information on a specific location on a map, such as the location of a building such as a public facility, a restaurant, or a tourist attraction. For example, outside the vehicle cabin, the user often directs his or her gaze to an object such as a landmark building, so the candidate gaze point memory device 120 pre-stores such objects. The candidate gaze points stored in the candidate gaze point storage unit 120 are referenced when the presentation information output control unit 104 corrects the position of the first gaze point. In addition, the communication device 110 communicates with the first output device 11 and the second output device 12 to exchange information.

The first output device 11 is an output device used by the first user outside the cabin of the vehicle 2. The first output device 11 is, for example, a virtual reality (VR) display device such as VR goggles worn on the head of the first user, and displays a virtual space constructed by the vehicle space of the vehicle 2. The first output device 11 includes a virtual vehicle environment generation unit 111, an image display unit 112, an audio acquisition unit 113, an audio output unit 114, a gaze information acquisition unit 115, a shared information acquisition unit 116, and an equipment communication unit 117. The virtual vehicle environment generation unit 111 generates a vehicle side image representing the inside and outside of the vehicle 2 based on the vehicle side information transmitted from the image display control device 10. Specifically, the virtual vehicle environment generation unit 111 first generates a virtual space image of the inside of the vehicle based on the inside of the vehicle image. The virtual vehicle environment generation unit 111 may also generate a virtual space image of the inside of the vehicle based on the interior information and information on the operating state of the in-vehicle equipment 13. Next, the virtual vehicle environment generation unit 111 generates an avatar image of the second user based on the posture information of the second user, and superimposes it on the position of the second user in the virtual space image. Then, the virtual vehicle environment generation unit 111 superimposes the vehicle exterior image on the position of the vehicle window shown in the virtual space image. For example, when the vehicle exterior image is captured by a 360-degree camera, the virtual vehicle environment generation unit 111 superimposes the vehicle exterior image as a celestial sphere image. In this way, the virtual vehicle environment generation unit 111 generates a vehicle-side image that represents the interior and exterior of the vehicle 2. The virtual space image is updated in real time according to changes in the vehicle space.

The image display unit 112 is a display of the first output device 11, and displays the vehicle-side image to the first user. Specifically, the image display unit 112 displays the vehicle-side image of the field of view corresponding to the line of sight direction according to the line of sight direction of the first user calculated from the first user posture information. The voice acquisition unit 113 acquires the voice emitted by the first user. For example, the voice acquisition unit 113 is a microphone. The voice output unit 114 is, for example, a speaker, and outputs the voice information of the second user. The gaze information acquisition unit 115 acquires the body posture and head position of the first user. For example, the gaze information acquisition unit 115 detects the head movement of the first user by the acceleration sensor and gyro sensor of the first output device 11. Then, the gaze information acquisition unit 115 acquires the first gaze point at which the first user is gazing in the vehicle-side image based on the gaze direction calculated from the head movement of the first user. Specifically, the gaze information acquisition unit 115 acquires information on the position coordinates of the first gaze point in the vehicle-side image. Furthermore, the line-of-sight information acquisition unit 115 may acquire posture information of the first user from an image captured by a camera installed in the space in which the first user is present, and detect the movement of the first user's head from the posture information of the first user.

2 is a diagram showing an example of the position and field of view of the first gaze point in this embodiment. In FIG. 2, a virtual space constructed based on vehicle-side information of the vehicle 2 is shown. In the virtual space, an image of the interior of the vehicle including the interior of the vehicle and an image of the exterior of the vehicle are displayed. For ease of explanation, the vehicle interior image and the vehicle exterior image are displayed in a schematic manner in FIG. 2. Also, for example, in FIG. 2, only the steering wheel, the dashboard, and the seats are displayed in a schematic manner as the vehicle interior image, but in the embodiment, other vehicle interior images are also displayed. Also, an avatar of the second user is displayed in the driver's seat of the vehicle. The first user is displayed a vehicle-side image corresponding to the field of view in the line of sight from the position of the first user according to the movement of the first user's head. The first gaze point is set as the position of the point on the vehicle-side image that the first user is gazing at, based on the position of the first user in the virtual space and the line of sight of the first user.

The shared information acquisition unit 116 also acquires information on the virtual object of the shared object that does not exist in the real vehicle space and is reflected in the field of view of the first user in the virtual space. The shared object is, for example, a predetermined part of the body of the first user. The body of the first user does not exist in the real vehicle space, but may be displayed as a virtual object in the field of view of the first user in the virtual space. Such an object needs to be shared with the second user because it is not visible to the second user. Specifically, the shared information acquisition unit 116 is position information of a predetermined part of the body of the first user and body part information indicating the predetermined part of the body. First, the shared information acquisition unit 116 acquires posture information of the first user from an image of the first user captured by an in-space camera installed in the space. Next, the shared information acquisition unit 116 acquires position information of a predetermined part of the body of the first user in the virtual space from the posture information of the first user. Furthermore, the body part information is information for identifying a predetermined part of the body of the first user. For example, the body part information includes information on the arms and hands. The shared information acquisition unit 116 also acquires, as the shared information, information on the virtual object of the object handled by the first user in the virtual space. The shared information acquisition unit 116 also acquires, as the shared information, information on the virtual sound generated by the first user in the virtual space. When the first user generates a sound in the virtual space by a virtual object, the shared information acquisition unit 116 acquires such a sound in the virtual space. For example, when the first user performs an action such as hitting a virtual object, a sound may be generated in the virtual space in response to the action. In such a case, the shared information acquisition unit 116 acquires audio information related to the virtual sound generated by the first user in the virtual space. The device communication unit 117 transmits the first gaze point position information, information related to a specific part of the first user's body, the shared information, and the audio information to the image display control device 10.

The second output device 12 is a device provided in the vehicle cabin and used by the second user. Examples of the second output device 12 include a terminal equipped with a monitor screen, or an augmented reality (AR) device worn by the second user on the head. The second output device 12 includes an image display unit 121, an audio acquisition unit 122, and an audio output unit 123. The image display unit 121 is a monitor and displays the first gaze point display information to the second user. For example, the image display unit 121 displays the first gaze point image to the second user. As a result, even if the field of view of the first user and the second user is different, the image of the field of view of the first user is displayed in the field of view of the second user, so that the second user can see the scenery seen by the first user without changing the field of view by moving the direction of the head. The second output device 12 may also be an AR device. When the image display unit 121 is a transparent AR display, the first gaze point display information is displayed on the field of view of the second user. When the image display unit 121 is a non-transparent AR display, the first gaze point display information is superimposed on the vehicle-side image displayed on the display. The voice acquisition unit 122 acquires the voice uttered by the second user. For example, the voice acquisition unit 122 is a microphone. The voice output unit 123 is, for example, a speaker, and outputs the voice information of the first user.

Next, an example of image display according to this embodiment will be described with reference to FIG. 3A and FIG. 3B. FIG. 3A and FIG. 3B are diagrams showing the scenery inside and outside the vehicle interior that is reflected in the field of view of the second user sitting in the driver's seat of the vehicle. In the example of FIG. 3A, the first user is looking at a landmark building reflected in the scenery to the side of the vehicle from the passenger seat position next to the second user in a virtual space corresponding to the vehicle space. In such a scene, since the landmark reflected in the vehicle-side image has a first gaze point, the surrounding area of the first gaze point, for example, an area including the landmark and its surrounding buildings, is extracted from the vehicle-side image to generate a first gaze point image. Then, as shown in FIG. 3A, the vehicle-side image including the landmark and its surrounding buildings is displayed as the first gaze point image on the display of the second output device 12. Also, the example of FIG. 3B is also a scene in which the first user is looking at a landmark building reflected in the scenery to the side of the vehicle from the passenger seat position next to the second user in a virtual space corresponding to the vehicle space. In such a case, instead of displaying the first gaze point image as in FIG. 3A, an indication image may be displayed on the display of the second output device 12 as shown in FIG. 3B. In FIG. 3B, when the landmark that the first user is looking at is on the left side as viewed from the position of the second user, an indication image indicating the direction of the first gaze point position is displayed in the field of view of the second user. Note that the indication image may be only one of either an arrow image or an image showing text information.

The in-vehicle device 13 outputs information on the operation status and display contents of the in-vehicle device 13 to the image display control device 10. The in-vehicle device 13 includes an IVI, a meter, and an air conditioner. For example, the in-vehicle device 13 outputs information on meters such as vehicle speed displayed on the instrument panel, the temperature of the air conditioner, and the display contents of the car navigation system to the image display control device 10. The posture detection device 14 detects the posture and position of the head, arms, and hands of the second user. For example, the posture detection device 14 captures an image of the second user using an in-vehicle camera 15, such as a depth camera, provided in the cabin of the vehicle 2, and estimates the skeleton of the second user from the captured image to obtain the posture of the second user. In addition, when the second user is wearing AR goggles, the posture detection device 14 detects the posture of the head of the second user using an acceleration sensor, a gyro sensor, etc. provided in the AR goggles. In addition, the in-vehicle camera 15 captures an image of the inside of the cabin including the second user. The exterior camera 16 is, for example, a 360-degree camera that captures the scenery around the vehicle 2. The exterior camera 16 may also be multiple cameras installed on the front, rear, left and right sides of the vehicle 2.

Next, an example of the control procedure of the image display control method will be described with reference to FIG. 4. FIG. 4 is a flowchart showing the control procedure of the image display control method according to this embodiment. In this embodiment, when information transmission and reception between the first output device 11 and the second output device 12 starts, the controller 100 starts image display control from step S1. First, in step S1, the controller 100 acquires an interior image of the vehicle interior including the posture of the second user in the vehicle interior, the operating status of the in-vehicle device 13, and information on the interior of the vehicle 2. In step S2, the controller 100 acquires an exterior image of the vehicle interior including the posture of the second user in the vehicle interior. In step S3, the controller 100 generates a vehicle-side image based on the vehicle-side information acquired in steps S1 and S2, and transmits a control signal to the first output device 11 to display the vehicle-side image to the first user via the first output device 11. In step S4, the first output device 11 generates a vehicle-side image representing the scenery inside and outside the vehicle interior based on the vehicle-side information. In step S5, the first output device 11 displays the vehicle-side image to the first user. In step S6, the first output device 11 acquires first gaze point position information. The first gaze point position information is position information of the first gaze point of the first user calculated from the posture information of the first user. In step S7, the first output device 11 transmits the first gaze point position information to the image display control device 10. In step S8, the controller 100 acquires vehicle-side information. In step S9, the controller 100 generates first gaze point display information. The first gaze point display information is, for example, a first gaze point image obtained by extracting a vehicle-side image of a peripheral area of the first gaze point position. In step S10, the controller 100 transmits a control signal for displaying the first gaze point display information to the second user via the second output device 12. The second output device 12 that has received the control signal displays the first gaze point display information to the second user.

In addition, in this embodiment, if there is a candidate gaze point close to the first gaze point, the presentation information output control unit 104 may correct the first gaze point to the candidate gaze point. For example, if the first gaze point in the vehicle side image is close to the position of a candidate gaze point such as a vehicle speed meter inside the vehicle or a specific building outside the vehicle, the first user is assumed to be gazing at the candidate gaze point, and the first gaze point is corrected to the candidate gaze point. The position information of the candidate gaze point is stored in advance in the candidate gaze point storage device 120. The presentation information output control unit 104 determines whether or not the candidate gaze point exists within a predetermined range from the first gaze point based on the first gaze point position information and the candidate gaze point information. Then, if it is determined that the candidate gaze point exists within the predetermined range from the first gaze point, the presentation information output control unit 104 corrects the first gaze point to the position of the candidate gaze point.

Using FIG. 5, the control procedure when the candidate gaze point is an outdoor candidate gaze point will be described. After receiving the first gaze point position information transmitted from the first output device 11 in step S7, the controller 100 judges the presence or absence of an outdoor candidate gaze point based on the received first gaze point position information according to the procedure of the flowchart shown in FIG. 5. First, in step S71, the controller 100 acquires information on POIs located in a gaze point range from the direction of the first user's first gaze point to a predetermined angle range and a predetermined distance. The gaze point range is a range for judging whether or not there is a candidate to be gazed upon near the gaze point, and is a range narrower than the field of view range. The POI information is acquired from map information. In step S72, the controller 100 judges whether or not there is a POI within the gaze point range. If it is determined that there is no POI within the gaze point range, the controller 100 proceeds to step S8 without correcting the position of the first gaze point transmitted from the first output device 11. If it is determined that there is a POI within the gaze range, the controller 100 determines in step S73 whether there are multiple POIs within the gaze range. If there is only one POI, the controller 100 corrects the position of the first gaze point to the position of that POI in step S77. If there are multiple POIs, the controller 100 determines in step S74 whether the multiple POIs can be compared in terms of popularity.

If the multiple POIs can be compared by popularity, the controller 100 compares the multiple POIs by popularity in step S75 and selects the POI with the highest popularity. Popularity is, for example, the number of searches in a Web search engine. The controller 100 selects the POI with the highest number of searches in a Web search engine as the POI with the highest popularity. If the multiple POIs cannot be compared by popularity, for example, if the difference in the number of searches for the multiple POIs is within a predetermined value and the popularity cannot be determined, the controller 100 compares the multiple POIs by their distance from the first gaze point in step S76. For example, the controller 100 calculates the distance between the first gaze point position and the positions of the multiple POIs and selects the POI with the shortest distance. In addition, the controller 100 may calculate the angle error of each of the multiple POIs from the first gaze point direction and select the POI with the smallest angle error. The controller 100 selects a POI based on the result of comparing the multiple POIs in step S75 or step S76, and then corrects the first gaze point position to the position of the selected POI in step S77. In addition, when generating the first gaze point image, the controller 100 may generate an image so as to highlight the POI. Examples of highlighting include surrounding the POI shown in the vehicle-side image with a rectangular figure or superimposing an image showing the name of the POI. This makes it easier to understand the target that the first user is gazing at. In addition, in this embodiment, the display mode for displaying the gaze point display information may be changed depending on whether the candidate gaze point corresponding to the corrected first gaze point is an indoor candidate gaze point or an outdoor candidate gaze point. When displaying the gaze point display information of the outdoor candidate gaze point, the controller 100 displays the gaze target that is the candidate gaze point in an enlarged mode. When displaying the gaze point display information of the indoor candidate gaze point, the controller 100 displays the gaze target in an enlarged mode. In addition, when displaying the gaze point display information of the indoor candidate gaze point, the controller 100 displays the gaze target in an overlapping mode.

As described above, in this embodiment, vehicle-side information about the inside and/or outside of the vehicle is acquired, a vehicle-side image representing the inside and/or outside of the vehicle is generated based on the acquired vehicle-side information, and a control signal for displaying the image to a first user outside the vehicle via a first display device is transmitted to the first display device. First gaze point position information of a first gaze point at which the first user is gazing in the vehicle-side image is acquired, and when the visual range when the first user is gazing is different from the visual range when a second user in a space different from the first user is gazing, first gaze point display information for displaying the first gaze point in the vehicle-side image to a second user is generated based on the first gaze point position information, and a control signal for displaying the generated first gaze point display information to the second user via a second display device is transmitted to the second display device. As a result, even when the visual range of the user of the information input/output terminal with respect to the outside world is different from the visual range of the user of another information input/output terminal with respect to the outside world, it is possible to accurately communicate between users of the information input/output terminals.

In addition, in this embodiment, first gaze point position information is acquired based on data related to the first user acquired by a sensor installed in the space in which the first user is present. This makes it possible to transmit data related to the gaze direction of a user outside the vehicle cabin with a smaller amount of data than when transmitting image data.

In addition, in this embodiment, a control signal for displaying the first gaze point display information to the second user is transmitted to the second display device via the second display device provided in the vehicle cabin. This allows the vehicle occupants to know information regarding the gaze direction of the user outside the vehicle cabin, since the display device mounted on the vehicle is used.

In addition, in this embodiment, a control signal for displaying the first gaze point display information to the second user as an augmented reality image is transmitted to the second display device via the second display device. This makes it possible to display information on the line of sight of a user outside the vehicle cabin in the field of view of a vehicle occupant.

In addition, in this embodiment, a first gaze point image is generated as first gaze point display information by extracting the surrounding area of the first gaze point from the vehicle side image based on the acquired first gaze point position information, and a control signal for displaying the generated first gaze point image to the second user via the second display device is transmitted to the second display device. This allows the vehicle occupants to see the scenery reflected in the field of view of the user outside the vehicle cabin and to communicate with the user.

In addition, in this embodiment, an indication image showing the surrounding area of the first gaze point in the vehicle-side image is generated as first gaze point display information based on the acquired first gaze point position information, and the generated indication image is displayed in the field of view of the second user via the second display device, or a control signal for displaying the generated indication image superimposed on the vehicle-side image is transmitted to the second display device. This makes it possible to display the direction of the gaze point of a user outside the vehicle cabin in the field of view of the vehicle occupants.

In addition, in this embodiment, candidate gaze points that are candidates for the first gaze point are stored in advance, and it is determined whether or not the candidate gaze point exists within a predetermined range from the first gaze point in the vehicle-side image. If it is determined that the candidate gaze point exists within the predetermined range, the first gaze point is corrected to the candidate gaze point. This allows vehicle occupants to share POI information viewed by a user outside the vehicle cabin.

In addition, in this embodiment, the candidate gaze point is stored in advance as either an indoor candidate gaze point located inside the vehicle cabin or an outdoor candidate gaze point located outside the vehicle cabin, and it is determined whether or not the candidate gaze point exists within a predetermined range from the first gaze point in the vehicle side image. If it is determined that the candidate gaze point exists within the predetermined range, the display mode of the first gaze point display information is changed depending on whether the candidate gaze point existing within the predetermined range is an indoor candidate gaze point or an outdoor candidate gaze point. This makes it easy to share information in a mode depending on whether the object that a user outside the vehicle cabin is gazing at is inside or outside the vehicle cabin.

In addition, in this embodiment, body position information indicating the position of a specific part of the first user's body in the field of view of the first user and body part information indicating the specific part of the body are acquired, and based on the acquired body position information and body part information, an image of the specific part of the body is superimposed at a position on the first gaze point image corresponding to the position of the specific part of the body in the field of view, and the first gaze point image with the image of the specific part of the body superimposed is displayed to the second user via the second display device. This makes it possible to display the gestures of a user outside the vehicle, allowing vehicle occupants to more easily understand what the user is looking at.

In addition, in this embodiment, body position information and body part information are acquired based on data about the first user acquired by a sensor installed in the space where the first user is present. This makes it possible to transmit data about a specific part of the body of a user who is outside the vehicle cabin with a smaller amount of data than when transmitting image data.

Second Embodiment
Next, the image display control system 1 according to the second embodiment will be described with reference to FIG. 6. The image display control system 1 according to the second embodiment is a modified example of the image display control system 1 according to the first embodiment. Except for the differences from the image display control system 1 according to the first embodiment in the points described below, the image display control system 1 according to the second embodiment has the same configuration as the first embodiment and operates in the same manner as the first embodiment, and the description of the first embodiment will be appropriately cited. The configuration of this embodiment that differs from the first embodiment is that it includes a remote image display control device 20. The remote image display control device 20 transmits a control signal to the first output device 11 to display position information of a second gaze point at which the second user is gazing in the vehicle space to the first user via the first output device 11. The remote image display control device 20 includes a virtual vehicle environment generation unit 201, a second gaze point display information generation unit 202, and a presentation information output control unit 203. In this embodiment, first, the image display control device 10 acquires position information of the second gaze point from the posture information of the second user acquired by the posture detection device 14. Then, the image display control device 10 transmits the vehicle-side information and the position information of the second gaze point to the remote image display control device 20. The remote image display control device 20 generates a vehicle-side image representing a virtual space corresponding to the vehicle space of the vehicle 2 based on the vehicle-side information by the virtual vehicle environment generation unit 201. The remote image display control device 20 also generates a second gaze point image by extracting a peripheral area of the second gaze point from the vehicle-side image based on the vehicle-side image and the second gaze point position information by the second gaze point display information generation unit 202. The remote image display control device 20 then generates a control signal by the presentation information output control unit 203 for displaying the vehicle-side image and the second gaze point image to the first user via the first output device. Specifically, the remote image display control device 20 generates a control signal for displaying the second gaze point image so as to be superimposed on the vehicle-side image, and transmits the control signal to the first output device. The first output device 11 displays the vehicle-side image with the second gaze point image superimposed to the first user by the image display unit 112. This allows the first user to know where the second user is gazing within the vehicle space.

As described above, this embodiment includes an image display control device and a remote image display control device that displays a vehicle-side image to a first user via a first display device, and the image display control device acquires second gaze point position information of a second gaze point at which a second user is gazing, and transmits the acquired second gaze point position information to the remote image display control device, and the remote image display control device acquires a second gaze point image in which a peripheral area of the second gaze point is extracted from the vehicle-side image based on the transmitted second gaze point position information, and displays the acquired second gaze point image to the first user via the first display device. This allows accurate communication between users of information input/output terminals even when the visual range of the user of an information input/output terminal with respect to the outside world differs from the visual range of a user of another information input/output terminal with respect to the same outside world.

The above-described embodiments are described to facilitate understanding of the present invention, and are not described to limit the present invention. Therefore, each element disclosed in the above embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention.

Reference Signs List 1: Image display control system 10: Image display control device 100: Controller 101: Vehicle-side information acquisition unit 102: Vehicle-side image display control unit 103: Remote information acquisition unit 104: Presentation information output control unit 11: First output device 111: Virtual vehicle environment generation unit 112: Image display unit 115: Line-of-sight information acquisition unit 116: Shared information acquisition unit 12: Second output device 121: Image display unit

Claims (12)

A vehicle-side information acquisition unit that acquires vehicle-side information about the interior and/or exterior of the vehicle;
a vehicle-side image display control unit that transmits a control signal to a first display device to cause a first user outside the vehicle cabin to display, via the first display device, a vehicle-side image that represents a virtual reality space corresponding to a vehicle space of the vehicle, which is generated based on the vehicle-side information acquired by the vehicle-side information acquisition unit;
a first gaze point position acquisition unit that acquires first gaze point position information of a first gaze point at which the first user is gazing in the vehicle-side image;
a gaze point information display control unit that generates first gaze point display information for displaying the first gaze point to the second user based on the first gaze point position information when a visual field range when the first user is gazing at the first gaze point differs from a visual field range when a second user in a space different from the first user is gazing at the first gaze point, and transmits a control signal to the second display device for displaying the generated first gaze point display information to the second user via the second display device ,
The second user is an image display control device that is a passenger in the vehicle or a user outside the vehicle cabin who shares the vehicle-side image via the second display device . The image display control device according to claim 1, wherein the first gaze point position acquisition unit acquires the first gaze point position information based on data related to the first user acquired by a sensor provided in a space in which the first user is present. The image display control device according to claim 1 or 2, wherein the gaze point information display control unit transmits a control signal to the second display device provided in the vehicle cabin, for displaying the first gaze point display information to the second user, via the second display device. The image display control device according to any one of claims 1 to 3, wherein the gaze point information display control unit transmits a control signal to the second display device via the second display device for displaying the first gaze point display information to the second user as an augmented reality image. The gaze point information display control unit is
generating a first gaze point image by extracting a peripheral area of the first gaze point from the vehicle-side image based on the first gaze point position information acquired by the first gaze point position acquisition unit, as the first gaze point display information;
An image display control device as described in any one of claims 1 to 4, which transmits a control signal to the second display device for displaying the generated first gaze point image to the second user via the second display device. The gaze point information display control unit is
generating, as the first gaze point display information, an indication image indicating a peripheral area of the first gaze point in the vehicle-side image based on the first gaze point position information acquired by the first gaze point position acquisition unit;
An image display control device as described in any one of claims 1 to 4, which transmits a control signal to the second display device to display the generated instruction image in the field of view of the second user via the second display device, or to display it superimposed on the vehicle-side image. A candidate gaze point storage unit that stores a candidate gaze point that is a candidate for the first gaze point in advance,
The first gaze point position acquisition unit is
determining whether the candidate gaze point is present within a predetermined range from the first gaze point in the vehicle-side image;
The image display control device according to any one of claims 1 to 6, wherein, when it is determined that the candidate gaze point exists within the predetermined range, the first gaze point is corrected to the candidate gaze point. The candidate gaze point storage unit
The candidate gaze point is stored in advance as either an indoor candidate gaze point located inside the vehicle interior or an outdoor candidate gaze point located outside the vehicle interior;
The gaze point information display control unit is
The image display control device according to claim 7, which changes the display mode of the first gaze point display information depending on whether the candidate gaze point corresponding to the corrected first gaze point is the indoor candidate gaze point or the outdoor candidate gaze point. a body information acquisition unit that acquires body position information indicating a position of a predetermined part of the body of the first user within a visual field of the first user and body part information indicating the predetermined part of the body,
The gaze point information display control unit is
based on the body position information and the body part information acquired by the body information acquisition unit, superimposing an image of the predetermined body part at a position on the first gaze point image corresponding to a position of the predetermined body part in the field of view;
The image display control device according to claim 5 , wherein the first gaze point image, onto which an image of a predetermined part of the body is superimposed, is displayed to the second user via the second display device. The image display control device according to claim 9, wherein the body information acquisition unit acquires the body position information and the body part information based on data related to the first user acquired by a sensor provided in a space in which the first user is present. The image display control device according to any one of claims 1 to 10, and a remote image display control device that displays the vehicle-side image to the first user via the first display device,
The image display control device includes:
a second gaze point position acquisition unit that acquires second gaze point position information of a second gaze point at which the second user is gazing in the vehicle space or in the vehicle-side image ;
a second gaze point position transmission unit that transmits the second gaze point position information acquired by the second gaze point position acquisition unit to the remote image display control device,
The remote image display control device includes:
acquiring a second gaze point image by extracting a peripheral area of the second gaze point from the vehicle-side image based on the second gaze point position information transmitted by the second gaze point position transmission unit;
An image display control system that displays the acquired second gaze point image to the first user via the first display device. An image display control method to be executed by a processor, comprising:
Acquire vehicle-side information about the interior and/or exterior of the vehicle;
transmits a control signal to a first display device to cause a first user outside the vehicle cabin to display, via a first display device, a vehicle-side image that represents a virtual reality space corresponding to a vehicle space of the vehicle , which is generated based on the acquired vehicle-side information;
acquiring first gaze point position information of a first gaze point at which the first user is gazing in the vehicle-side image;
when a visual field range when the first user is gazing is different from a visual field range when a second user in a space different from the first user is gazing, generating first gaze point display information for displaying the first gaze point in the vehicle-side image to the second user based on the first gaze point position information, and transmitting a control signal to the second display device for displaying the generated first gaze point display information to the second user via a second display device;
An image display control method in which the second user is a passenger in the vehicle or a user outside the vehicle cabin who shares the vehicle-side image via the second display device .

Images