JP7841466B2 - Vehicle display device - Google Patents

Description

This disclosure relates to a vehicle display device.

Patent Document 1 describes a technology for receiving an image of a remote driver operating a vehicle remotely, and displaying the received image of the remote driver on an external display unit provided on the vehicle's exterior for displaying images outside the vehicle.

Furthermore, Patent Document 2 describes transmitting images of the remote operator performing driving operations to the vehicle when remote operation of the vehicle is being performed, thereby showing the occupant the status of the remote operator.

Japanese Patent Publication No. 2021-020638 Japanese Patent Publication No. 2018-077649

When a vehicle is being remotely controlled by a remote driver, there is a problem in that the amount of information transmitted to traffic participants outside the vehicle and occupants inside the vehicle is reduced, because the remote driver is not seated in the driver's seat and driving the vehicle. Furthermore, in the technologies described in Patent Document 1 and Patent Document 2, the remote driver's image is constantly displayed outside or inside the vehicle, respectively, which may reduce the awareness of the remote driver's remote operation among occupants inside the vehicle or traffic participants outside the vehicle.

This disclosure was made in consideration of the above facts, and aims to provide a vehicle display device that can suppress the reduction in information transmitted to the surroundings when a vehicle is remotely driven by a remote driver.

A vehicle display device according to the first embodiment includes a display unit provided in a remotely driven vehicle operated remotely by a remote driver, and a display control unit that changes the method of displaying the remote driver's image on the display unit according to the driving scene of the remotely driven vehicle and the image of the remote driver to be presented based on the driving scene , wherein the display unit is a transparent display in which two displays are superimposed and arranged on the windshield of the remotely driven vehicle, and the display control unit causes the remote driver's image to be displayed on the display closer to the presentation target and the lining image to be displayed on the display further away from the presentation target .

In the first embodiment, the method of displaying the remote driver's image on the display unit is changed according to the driving scene of the remotely driven vehicle and the target of the presentation of the remote driver's image based on the driving scene of the remotely driven vehicle. This makes it possible to suppress a reduction in the amount of information transmitted to the surroundings when the vehicle is remotely driven by the remote driver.
Furthermore, in the first embodiment, the remote driver's image is displayed on the display closer to the object being presented, and the lining image is displayed on the display further away from the object being presented. This improves the visibility of the remote driver's image from the perspective of the object being presented, and makes it possible to appropriately convey information to the object being presented.

A vehicle display device according to a second embodiment includes a display unit provided in a remotely driven vehicle operated by a remote driver, a display control unit that changes the method of displaying an image of the remote driver on the display unit according to the driving scene of the remotely driven vehicle and the target of the presentation of the image of the remote driver based on the driving scene, and an angle changing unit that can change the angle of the display unit, wherein the display control unit, in the case of an abnormal driving where the driving scene is different from normal driving, sets the target of the presentation to an occupant present inside the remotely driven vehicle and changes the angle of the display unit so that the display unit faces the occupant.

In the second embodiment, the method of displaying the remote driver's image on the display unit is changed according to the driving scene of the remotely driven vehicle and the target of the presentation of the remote driver's image based on the driving scene of the remotely driven vehicle. Therefore, similar to the first embodiment, it is possible to suppress a reduction in the amount of information transmitted to the surroundings when the vehicle is remotely driven by the remote driver.
Furthermore, in the second embodiment, during non-normal operation, when an occupant inside the vehicle wants to check on the remote driver, the angle of the display unit can be changed so that it faces the occupant, thereby making it possible to appropriately convey the information the occupant needs.

A third aspect is that, in the first or second aspect, the subjects of the presentation of the remote driver's image include traffic participants located outside the remote-driven vehicle and occupants located inside the remote-driven vehicle.

In the third embodiment, the method of displaying the image of the remote driver is switched depending on whether the target of the image presentation is a traffic participant outside the remote-controlled vehicle or an occupant inside the remote-controlled vehicle, thereby suppressing a reduction in the amount of information transmitted to traffic participants outside the vehicle or occupants inside the vehicle during remote driving.

A fourth embodiment is that, in the first or second embodiment, when the driving scene is a scene of passing through an intersection or pedestrian crossing, the display control unit makes the displayed object a traffic participant located outside the remotely driven vehicle and displays an image of the upper body of the remote driver on the display unit.

In driving scenarios involving passing through intersections or pedestrian crossings, there is a high probability of communicating with traffic participants outside the vehicle using hand signals or other means. In the fourth embodiment, in driving scenarios involving passing through intersections or pedestrian crossings, an image of the remote driver's upper body is displayed on the display unit, making it possible to appropriately transmit information to traffic participants outside the vehicle in the aforementioned driving scenarios.

This disclosure has the effect of suppressing the reduction in information transmitted to the surroundings when a vehicle is remotely driven by a remote driver.

This is a block diagram showing the schematic configuration of the operation control system according to the first embodiment. This is a cross-sectional view showing the display unit according to the first embodiment. This diagram shows the display control process according to the first embodiment. This is a block diagram showing the schematic configuration of the operation control system according to the second embodiment. This is a perspective view showing the display unit according to the second embodiment. This is a flowchart showing the display control process according to the second embodiment.

An example of an embodiment of this disclosure will be described in detail below with reference to the drawings.

[First Embodiment]
As shown in Figure 1, the driving control system 10A according to the first embodiment includes an on-board system 12 mounted on a vehicle and a remote driving control device 40. The on-board system 12 and the remote driving control device are communicated together via a network 50.

Furthermore, a vehicle equipped with the in-vehicle system 12 (an example of a remotely driven vehicle in this disclosure) is provided with two driving modes: a remote driving mode and an automatic driving mode. The remote driving mode is a mode in which a remote driver, located outside the vehicle, remotely drives the vehicle by operating the remote driving control unit 44 of the remote driving control device 40 (described later). The automatic driving mode is a mode in which the automatic driving control unit 24 (described later) automatically drives (operates) the vehicle.

The in-vehicle system 12 includes a communication interface (I/F) 14, a group of sensors 16, a driver operation ACT (Actuator) 18, a driver control ECU (Electronic Control Unit) 20, a display unit 26, and a display control ECU 34, all of which are connected to each other via a bus 38.

The communication interface 14 of the in-vehicle system 12 communicates with the remote driving control device 40 via the network 50. The sensor group 16 includes multiple types of sensors that acquire information representing the surrounding environment of the vehicle. The surrounding environment information acquired by the sensor group 16 is used for automatic driving control by the automatic driving control unit 24 (described later) and for determining driving scenes (also described later). Examples of sensors included in the sensor group 16 are a GNSS (Global Navigation Satellite System) device, an in-vehicle communication device, a navigation system, a radar device, and a camera.

The GNSS device receives GNSS signals from multiple GNSS satellites to determine the vehicle's position. The on-board communication device is a communication device that performs at least one of vehicle-to-vehicle communication with other vehicles and vehicle-to-infrastructure communication with roadside units via the communication I/F 14. The navigation system includes a map information storage unit that stores map information. Based on the position information obtained from the GNSS device and the map information stored in the map information storage unit, it displays the vehicle's position on a map and provides route guidance to the destination.

The radar system detects objects such as pedestrians and other vehicles around the vehicle and acquires the relative position and relative speed of the detected objects and the vehicle. The radar system also excludes noise and roadside objects such as guardrails from its monitoring targets, tracking and monitoring pedestrians and other vehicles as the target objects, and outputting information such as the relative position and relative speed of each target object. The cameras capture images of the vehicle's surroundings using multiple cameras and output the captured images.

The driving operation ACT 18 includes a throttle ACT that changes the throttle opening of the vehicle, a brake ACT that changes the braking force generated by the vehicle's braking system, and a steering ACT that changes the amount of steering by the vehicle's steering system.

The driving control ECU 20 includes a CPU (Central Processing Unit), memory such as ROM (Read Only Memory) and RAM (Random Access Memory), non-volatile storage such as HDD (Hard Disk Drive) and SSD (Solid State Drive), and a communication interface. The driving control ECU 20 functions as a remote driving operation information acquisition unit 22 and an automatic driving control unit 24 when a predetermined program stored in the storage unit is read into memory and executed by the CPU.

The remote driving operation information acquisition unit 22 acquires remote driving operation information representing the amount of operation performed by the remote driver on the remote driving operation device 40 (described later) when the vehicle is in remote driving mode. The automatic driving control unit 24, when the vehicle is in automatic driving mode, generates and outputs automatic driving control information that controls the operation of each ACT of the driving operation ACT 18 based on information acquired by multiple types of sensors in the sensor group 16 that represents the conditions of the vehicle's surrounding environment. This enables the vehicle to automatically move.

The display unit 26 includes a transparent exterior display 28 and an interior display 30. As shown in Figure 2(A), the exterior display 28 and the interior display 30 are superimposed and positioned on the interior side of the windshield 32 of the vehicle on which the in-vehicle system 12 is mounted. The display unit 26 is an example of a transparent display in this disclosure.

The display control ECU 34 includes a CPU, memory such as ROM or RAM, a non-volatile storage unit such as an HDD or SSD, and a communication interface. The display control ECU 34 functions as a display control unit 36 when a predetermined program stored in the storage unit is read into memory and executed by the CPU. In remote driving mode, the display control unit 36 changes the method of displaying the remote driver's image on the display unit 26 according to the driving scene of the remote-controlled vehicle and the target image of the remote driver based on that driving scene. The in-vehicle system 12, including the display unit 26 and the display control ECU 34, is an example of a vehicle display device according to this disclosure.

The remote driving control device 40 includes a communication interface 42, a remote driving control unit 44, and a remote driver image capture unit 46, which are connected to each other via a bus 41. The communication interface 42 communicates with the in-vehicle system 12 via the network 50. The remote driver image capture unit 46 captures the remote driver operating the remote driving control unit 44 and transmits the image (video) of the remote driver obtained from the capture to the in-vehicle system 12.

The remote driving control unit 44 includes a throttle pedal, brake pedal, steering wheel, and display unit. The display unit shows an image representing the surrounding environment of the vehicle, captured by a camera included in the sensor group 16 of the in-vehicle system 12. The throttle pedal, brake pedal, and steering wheel of the remote driving control unit 44 are operated by the remote driver when the vehicle is remotely driven in remote driving mode. The remote driving control unit 44 also includes sensors to detect the amount of operation of each pedal and steering wheel, and transmits the amount of operation of each pedal and steering wheel by the remote driver as remote driving operation information to the in-vehicle system 12.

Next, as an example of the operation of the first embodiment, the display control process performed by the display control unit 36 of the display control ECU 34 while the vehicle's driving mode is in remote driving mode will be described with reference to Figure 3.

In step 70 of the display control processing, the display control unit 36 determines the current driving scene of the vehicle based on the detection information from the sensor group 16. In step 72, the display control unit 36 determines whether the current driving scene determined in step 70 corresponds to a first driving scene in which an image should be displayed outside the vehicle. An example of a first driving scene is a driving scene passing through an intersection or pedestrian crossing. If the determination in step 72 is negative, the process proceeds to step 74.

In step 74, the display control unit 36 determines whether the current driving scene identified in step 70 corresponds to a second driving scene in which an image should be displayed inside the vehicle. An example of a second driving scene is an abnormal driving scene that differs from normal driving, such as sudden braking, sudden steering, or sudden acceleration. A second driving scene may also include a situation where an occupant inside the vehicle speaks to the remote driver. If the determination in step 74 is negative, the process returns to step 70, and steps 70 to 74 are repeated until the determination in step 72 or step 74 is positive.

If the current driving scene corresponds to the first driving scene, the determination in step 72 is affirmed, and the system proceeds to step 76. In step 76, the display control unit 36 displays an image 60 of the remote driver's upper body on the exterior display 28 of the display unit 26, as shown in Figure 2(B) as an example. Then, in the next step 78, the display control unit 36 displays a lining image 62 on the interior display 30 of the display unit 26, and returns to step 70. In this embodiment, the lining image 62 is an opaque image with uniform density (brightness) throughout, and the color can be white, black, or gray.

In normal driving, where the driver is seated in the driver's seat, the driver may exchange hand signals with drivers of other vehicles passing at intersections, or with pedestrians crossing crosswalks. In contrast, in this embodiment, in remote driving mode, the remote driver's image is displayed on the display unit 26 facing outwards in such driving scenarios. This allows the remote driver to exchange hand signals with drivers of other vehicles passing at intersections or pedestrians crossing crosswalks, just as in normal driving. Note that in the above example, the drivers of other vehicles passing at intersections and pedestrians crossing crosswalks are examples of "traffic participants present outside the remotely driven vehicle" in this disclosure.

Furthermore, if the current driving scene corresponds to the second driving scene, the determination in step 74 is affirmed, and the system proceeds to step 80. In step 80, the display control unit 36 displays the remote driver's image 60 on the in-vehicle display 30 of the display unit 26, as shown in Figure 2(C) as an example. Then, in the next step 82, the display control unit 36 displays the lining image 62 on the out-vehicle display 28 of the display unit 26, and returns to step 70.

In remote driving mode, if abnormal driving such as sudden braking, sudden steering, or sudden acceleration occurs, the vehicle occupants cannot see the remote driver's actions, which may cause anxiety. In this embodiment, however, in remote driving mode, the remote driver's image is displayed on the display unit 26 facing the vehicle interior during the abnormal driving scene, thereby suppressing anxiety among the occupants.

Thus, in the first embodiment, the remotely driven vehicle operated by a remote driver is equipped with a display unit 26, and the display control unit 36 changes the method of displaying the remote driver's image on the display unit 26 according to the driving scene of the remotely driven vehicle and the target of the remote driver's image presentation based on the driving scene. This makes it possible to suppress a reduction in the amount of information transmitted to the surroundings when the vehicle is being remotely driven by a remote driver.

Furthermore, in the first embodiment, the images presented to the remote driver include traffic participants outside the remotely driven vehicle and occupants inside the vehicle. This helps to mitigate the reduction in information transmitted to traffic participants outside the vehicle or occupants inside the vehicle during remote driving.

Furthermore, in the first embodiment, when the driving scene involves passing through an intersection or pedestrian crossing, the display control unit 36 designates the target of the presentation as a traffic participant located outside the remotely driven vehicle and displays an image of the remote driver's upper body on the display unit 26. This makes it possible to appropriately transmit information to traffic participants located outside the vehicle during driving scenes involving passing through intersections or pedestrian crossings.

Furthermore, in the first embodiment, the display unit 26 is a transparent display consisting of two displays 28 and 30 stacked and positioned on the windshield 32 of the remotely driven vehicle. The display control unit 36 displays an image of the remote driver on the display closer to the object being presented, and displays a background image on the display further away from the object being presented. This improves the visibility of the remote driver's image from the perspective of the object being presented, enabling the appropriate transmission of information to the object being presented.

[Second Embodiment]
Next, a second embodiment of the present disclosure will be described. Parts identical to those in the first embodiment will be denoted by the same reference numerals, and their descriptions will be omitted.

As shown in Figure 4, the driving control system 10B according to the second embodiment is equipped with a display unit 54, an angle change ACT 56, and a robot hand 58 in the in-vehicle system 12, replacing the display unit 26 described in the first embodiment. As shown in Figure 5(A), the display unit 26 is located inside the vehicle, in the position where the driver's seat is normally installed, and is capable of displaying an image on one side (the image display surface). As shown in Figure 5(B), the display unit 54 is supported by a frame (not shown) so as to be rotatable around an axis tilted rearward by a predetermined angle with respect to the vehicle's vertical direction.

The angle change ACT 56 allows the angle of the display unit 54 to be changed, rotating the display unit 54 to a first position where the image display surface faces the front of the vehicle (see Figure 5(A)) or to a second position where the image display surface faces the rear of the vehicle (see Figure 5(C)). When the display unit 54 is in the first position, the image displayed on the image display surface is visible to traffic participants located in front of the vehicle. When the display unit 54 is in the second position, the image displayed on the image display surface is visible to occupants seated in the rear seats of the vehicle. Note that the angle change ACT 56 is an example of an angle change unit in this disclosure.

Furthermore, the robot hand 58 has its base attached to the aforementioned frame (not shown), and its joint degrees of freedom are set to reproduce hand signal actions typically performed by vehicle drivers, such as the hand signal to yield the right of way. In this second embodiment, the robot hand 58 functions as a telepresence robot together with the display unit 54.

Next, as an explanation of the operation of the second embodiment, referring to Figure 6, only the parts of the display control processing according to the second embodiment that differ from the first embodiment will be described. In the second embodiment, if the current driving scene of the vehicle corresponds to the first driving scene, the determination in step 72 is affirmed and the process proceeds to step 84. In step 84, the display control unit 36 rotates the display unit 54 by angle change ACT 56 so that the image display surface of the display unit 54 faces forward (to the first position shown in Figure 5(A)). After the processing in step 84, the process proceeds to step 88.

In the second embodiment, if the current driving scene of the vehicle corresponds to the second driving scene, the determination in step 74 is affirmed and the process proceeds to step 86. In step 86, the display control unit 36 rotates the display unit 54 using the angle change ACT 56 so that the image display surface of the display unit 54 faces the rear of the vehicle (to be positioned in the second position shown in Figure 5(C)). After the processing in step 86 is completed, the process proceeds to step 88.

In step 88, the display control unit 36 causes the display unit 54 to display an image of the remote driver. As a result, similar to the first embodiment, if the vehicle's current driving scene corresponds to the first driving scene, the image of the remote driver is presented to traffic participants located in front of the vehicle; if the vehicle's current driving scene corresponds to the second driving scene, the image of the remote driver is presented to the occupants seated in the rear seats of the vehicle.

In the next step 90, the display control unit 36 determines whether the remote driver is performing a hand signal action based on the image of the remote driver captured by the remote driver imaging unit 46. If the remote driver is performing a hand signal action, the robot hand 58 is driven in response to the remote driver's hand signal action, thereby simulating the hand signal action performed by the remote driver. This improves the visibility of the hand signal action performed by the remote driver.

Thus, in the second embodiment, the system further includes an angle change ACT 56 that allows the angle of the display unit 54 to be changed. The display control unit 36, in the case of abnormal driving where the driving scene differs from normal driving, sets the target of the display to the occupants present inside the remotely driven vehicle and changes the angle of the display unit 54 so that it faces the occupants. This makes it possible to appropriately convey the information desired by the occupants during abnormal driving, when they want to check on the remote driver.

In the above embodiment, a configuration was described in which the remote driver's image is not displayed on the display unit when the vehicle's current driving scene is neither the first nor the second driving scene. However, this disclosure is not limited to this configuration. For example, when the vehicle's current driving scene is neither the first nor the second driving scene, the remote driver's image may be displayed on the display unit 26 or 54 facing outwards from the vehicle.

Furthermore, when the vehicle's current driving scenario corresponds to the first driving scenario, the remote driver's image may be displayed on the display unit 26 or 54 facing outwards. The distance between the vehicle and the traffic participant viewing the image may be determined, and the image magnification may be increased as the determined distance increases. Additionally, the image density (brightness) may be changed according to weather conditions or other factors to improve image visibility.

Furthermore, although the second embodiment described a configuration in which the robot hand 58 is combined with the display unit 54, the robot hand 58 can also be combined with the display unit 26 described in the first embodiment.

The following additional information is disclosed regarding the above embodiments.

(Note 1)
A display unit installed in a remotely operated vehicle that is remotely driven by a remote operator,
A display control unit that changes the method of displaying the remote driver's image on the display unit according to the driving scene of the remote-controlled vehicle and the target of the presentation of the image of the remote driver based on the driving scene,
Vehicle display devices including...

(Note 2)
The vehicle display device described in Appendix 1 includes, for displaying images of the remote driver, traffic participants located outside the remotely driven vehicle and occupants located inside the remotely driven vehicle.

(Note 3)
The vehicle display device according to Appendix 1 or Appendix 2, wherein, when the driving scene is a scene of passing through an intersection or a pedestrian crossing, the display control unit makes the target of presentation a traffic participant located outside the remotely driven vehicle, and displays an image of the upper body of the remote driver on the display unit.

(Note 4)
The display unit is a transparent display consisting of two displays stacked on top of each other and placed on the windshield of the remotely driven vehicle.

The display control unit displays an image of the remote driver on the display closer to the object being presented, and displays an image of the lining on the display further away from the object being presented, as described in any one of the items in Appendix 1 to Appendix 3 of the vehicle display device.

(Note 5)
The display unit is further provided with an angle adjustment unit that can change the angle of the display unit,
The vehicle display device according to any one of the appendices 1 to 4, wherein the display control unit, in the case of abnormal operation where the driving scene is different from normal operation, sets the object to be displayed to the occupants present inside the remotely operated vehicle and changes the angle of the display unit so that the display unit faces the occupants.

(Note 6)
The vehicle further comprises a robotic hand installed inside the remote-controlled vehicle,
The display control unit drives the robot hand in accordance with the actions performed by the remote driver, as described in any one of Appendix 1 to Appendix 5 for a vehicle display device.

12. In-vehicle systems (vehicle display devices)
26 Display Unit 28 Exterior Display 30 Interior Display 32 Windshield 34 Display Control ECU
36 Display control unit 54 Display unit 56 Angle change ACT (angle change unit)
58 Robot Hand

Claims (4)

A display unit installed in a remotely operated vehicle that is remotely driven by a remote operator,
A display control unit that changes the method of displaying the remote driver's image on the display unit according to the driving scene of the remote-controlled vehicle and the target of the presentation of the image of the remote driver based on the driving scene,
Includes,
The display unit is a transparent display consisting of two displays stacked on top of each other and placed on the windshield of the remotely driven vehicle.
The display control unit is a vehicle display device that displays an image of the remote driver on the display closer to the object to be presented, and displays an image of the lining on the display further away from the object to be presented. A display unit installed in a remotely operated vehicle that is remotely driven by a remote operator,
A display control unit that changes the method of displaying the remote driver's image on the display unit according to the driving scene of the remote-controlled vehicle and the target of the presentation of the image of the remote driver based on the driving scene,
An angle changing unit that can change the angle of the display unit,
Includes,
The display control unit, in the case of abnormal driving where the driving scene differs from normal driving, sets the target of the display to the occupants present inside the remotely driven vehicle, and changes the angle of the display unit so that the display unit faces the occupants. The vehicle display device according to claim 1 or claim 2, wherein the subjects for which the remote driver's image is displayed include traffic participants located outside the remote-controlled vehicle and occupants located inside the remote-controlled vehicle. The vehicle display device according to claim 1 or 2, wherein the display control unit, when the driving scene is a scene of passing through an intersection or a pedestrian crossing, determines the object to be presented as a traffic participant located outside the remotely driven vehicle and displays an image of the upper body of the remote driver on the display unit.