JP7632498B2 - Remote support device, remote support method, and remote support program - Google Patents

Description

This disclosure relates to a remote assistance device, a remote assistance method, and a remote assistance program.

Patent Document 1 discloses a remote assistance device that includes a task allocation unit (40) that assigns a task of remotely operating a vehicle (10) to an operator selected from among a plurality of operators whose operating ability is confirmed by the latest evaluation result stored in a memory unit (35), an evaluation acquisition unit (42) that acquires an evaluation result of the operating ability of the selected operator when the selected operator performs the task assigned by the task allocation unit using a remote operation unit (41) that drives the vehicle by remote operation of the operator, and a storage control unit (43) that controls the storage of the latest evaluation result acquired by the evaluation acquisition unit in the storage unit.

JP 2021-170191 A

Remote operators who remotely control autonomous vehicles to assist them in their travels, as well as supervisors who oversee the operators, are required to have the ability to make accurate and rapid situational judgments.

The technology described in Patent Document 1 above presents an evaluation result of the tasks performed by the operator in order to ensure the operator's remote operation capabilities.

However, because traffic conditions are different each time even at the same support point, the evaluation may change each time. Therefore, it is difficult to determine how to improve the situation when performing remote operation. For this reason, the technology described in Patent Document 1 above has the problem that it is difficult to support the improvement of the operator's situation judgment ability.

The present disclosure aims to provide a remote assistance device, a remote assistance method, and a remote assistance program that can help improve the situational judgment capabilities of staff involved in remote operation to support the driving of autonomous vehicles.

The remote assistance device according to the first aspect of the present disclosure includes a front image transmission unit that transmits, when remote assistance at an assistance point is requested from an autonomous vehicle, a front image captured by the autonomous vehicle and including at least the area in front of the autonomous vehicle to a remote operation device that remotely operates the autonomous vehicle, a receiving unit that receives an operation signal corresponding to remote operation at the assistance point from the remote operation device, an operation signal transmission unit that transmits the received operation signal to the autonomous vehicle, and a rear image transmission unit that transmits, after the autonomous vehicle passes a switching point set to switch the display on the remote operation device based on the assistance point, a rear image that is captured to include at least the rear of the autonomous vehicle and that emphasizes the rear, to the remote operation device.

The remote support device according to the second aspect is the remote support device according to the first aspect, in which the rear image transmission unit transmits a rear image of a shooting range determined according to the content of the remote operation.

The remote support device according to the third aspect is the remote support device according to the first or second aspect, in which the receiving unit receives gaze position information regarding the gaze position of the operator of the remote control device in a forward image displayed on the remote control device, and the rear image transmitting unit transmits to the remote control device a rear image including the gaze range of the operator identified based on the gaze position information.

The remote support device according to the fourth aspect is the remote support device according to the third aspect, in which, when there are multiple gaze ranges, the rear image transmission unit transmits to the remote control device a rear image including the gaze range with the longest gaze time among the multiple gaze ranges.

The remote support device according to the fifth aspect is the remote support device according to the third aspect, in which, when there are multiple gaze ranges, the rear image transmission unit transmits a rear image including all of the multiple gaze ranges to the remote control device.

A remote assistance device according to a sixth aspect is a remote assistance device according to any one of the first to fifth aspects, in which the rear image transmission unit transmits to the remote operation device a rear image that highlights an object that is overlooked by the operator of the remote operation device.

A remote support device according to a seventh aspect is a remote support device according to any one of the third to fifth aspects, in which the rear image transmission unit transmits to the remote control device a rear image that highlights an object that is likely to be overlooked by the operator, which object is identified based on the gaze range.

The remote support device according to the eighth aspect is the remote support device according to the first aspect, in which the rear image transmission unit transmits to the remote control device a rear image in which a newly detected object in the rear image is highlighted.

The remote support device according to the ninth aspect is the remote support device according to the first aspect, in which the rear image transmission unit transmits to the remote control device a rear image displaying speed information related to the speed change of an oncoming vehicle detected by the autonomous vehicle.

The remote support device according to the tenth aspect is the remote support device according to the first aspect, in which the rear image transmission unit acquires an overhead image taken from an overhead view of the area including the switching point and transmits it to the remote control device.

The remote assistance device according to the eleventh aspect is an image in which, in the remote assistance device according to the first aspect, the rear image has a display area displaying the rear of the autonomously driven vehicle that is larger than the display area displaying the front of the autonomously driven vehicle.

In the remote assistance method according to the twelfth aspect, when remote assistance at an assistance point is requested by an autonomous vehicle, at least one processor executes processing including transmitting a forward image captured by the autonomous vehicle and including at least the front of the autonomous vehicle to a remote operation device that remotely operates the autonomous vehicle, receiving an operation signal corresponding to remote operation at the assistance point from the remote operation device, transmitting the received operation signal to the autonomous vehicle, and transmitting a rear image captured to include at least the rear of the autonomous vehicle and emphasizing the rear to the remote operation device after the autonomous vehicle passes a switching point set to switch the display on the remote operation device based on the assistance point.

The remote assistance program according to the thirteenth aspect causes at least one processor to execute processing including, when remote assistance at an assistance point is requested from an autonomous vehicle, transmitting a forward image captured by the autonomous vehicle and including at least the front of the autonomous vehicle to a remote operation device that remotely operates the autonomous vehicle, receiving an operation signal corresponding to remote operation at the assistance point from the remote operation device, transmitting the received operation signal to the autonomous vehicle, and transmitting a rear image captured to include at least the rear of the autonomous vehicle and emphasizing the rear to the remote operation device after the autonomous vehicle passes a switching point set to switch the display on the remote operation device based on the assistance point.

The present disclosure has the effect of helping to improve the situational judgment capabilities of staff involved in remote operation to assist the driving of autonomous vehicles.

FIG. 1 is a configuration diagram of an autonomous driving system. FIG. 2 is a diagram for explaining a shooting range. FIG. 2 is a functional block diagram of a remote support unit. FIG. 2 is a diagram illustrating a hardware configuration of a remote support device. 13 is a flowchart of a remote support process. FIG. 2 is a diagram showing the state of a vehicle at an intersection before remote assistance. FIG. 13 is a diagram showing the state of a vehicle after remote assistance at an intersection. FIG. 2 is a diagram showing the state of a vehicle at an intersection before remote assistance. FIG. 13 is a diagram showing the state of a vehicle after remote assistance at an intersection. FIG. 2 is a diagram showing the state of a vehicle at a bus stop before remote assistance. FIG. 13 is a diagram showing the state of a vehicle after remote assistance at a bus stop. FIG. 13 is a diagram showing the state of a vehicle before remote assistance when an obstacle is present; FIG. 13 is a diagram showing the state of the vehicle after remote assistance in the presence of an obstacle. FIG. 13 is a diagram showing the state of a vehicle at a pedestrian crossing before remote assistance. FIG. 13 is a diagram showing the state of a vehicle after remote assistance at a pedestrian crossing. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal. FIG. 11 is a functional block diagram of an operator terminal according to the second embodiment. FIG. 13 is a diagram for explaining a rear image including an operator's gaze range. FIG. 13 is a diagram for explaining an object to be overlooked. FIG. 11 is a diagram for explaining a change in speed of an oncoming vehicle. FIG. 1 is a diagram showing an intersection where an infrastructure camera is installed. FIG. 13 is a diagram showing an example of an overhead image. FIG. 13 is a diagram for explaining a shooting range according to the third embodiment. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the third embodiment. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the third embodiment. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the fourth embodiment. FIG. 13 is a diagram illustrating an example of a display screen of an operator terminal according to the fourth embodiment.

Below, we will explain in detail the form for implementing the present invention with reference to the drawings.

<First embodiment>

As shown in FIG. 1, the autonomous driving system 100 of this embodiment includes a vehicle 10, a remote assistance device 50 located at an autonomous driving center, and an operator terminal 80 as an example of a remote operation device operated by an operator. The vehicle 10 in this embodiment is an autonomous vehicle that is operated automatically. The autonomous driving system 100 is a system that allows an operator in a remote location to remotely assist the vehicle 10 when the vehicle 10 (autonomous vehicle) arrives at an assistance point.

In this embodiment, remote support refers to an operator operating the vehicle 10 from a remote location, or an operator monitoring the vehicle 10 from a remote location. A support point refers to a point where it is difficult for the vehicle 10 to drive autonomously, or a point where the operator can remotely support the vehicle. Examples of support points include, but are not limited to, intersections, construction sites, parking lots, expressways, motorways, service areas, parking areas, baggage collection points, toll booths, and interchanges.

The autonomous driving system 100 of this embodiment includes one or more vehicles 10. Each vehicle 10 includes a vehicle-side communication device 12, a camera 14, a positioning device 16, and a vehicle-side control unit 18.

The vehicle-side communication device 12 performs wireless communication with the remote assistance device 50.

As shown in FIG. 2, the camera 14 photographs the front range 30F, the right side range 30RS, the left side range 30LS, and the rear range 30R of the vehicle 10. The camera 14 may be configured with four cameras that photograph each range individually, or may be configured with a 360-degree camera that can photograph the entire range. In this embodiment, a case where four cameras are provided to photograph each range will be described. For convenience, the camera that photographs the range including at least the front range 30F will be referred to as camera 14F, the camera that photographs the range including at least the right side range 30RS will be referred to as camera 14RS, the camera that photographs the range including at least the left side range 30LS will be referred to as camera 14LS, and the camera that photographs the range including at least the rear range 30R will be referred to as camera 14R. When there is no need to distinguish between the cameras, they will be referred to as camera 14.

As shown in FIG. 2, the forward range 30F is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the forward direction (traveling direction) F of the vehicle 10, when the vehicle 10 is viewed in a plan view. The right side range 30RS is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the right direction RS of the vehicle 10, when the vehicle 10 is viewed in a plan view. The left side range 30LS is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the left direction LS of the vehicle 10, when the vehicle 10 is viewed in a plan view. The rear range 30R is a range of 45 degrees to the right and 45 degrees to the left, totaling 90 degrees, centered on the rear direction (reverse direction) R of the vehicle 10, when the vehicle 10 is viewed in a plan view.

The positioning device 16 is equipped with a GPS (Global Positioning System) receiver or a GNSS (Global Navigation Satellite System) receiver and determines the current position of the vehicle 10.

The vehicle-side control unit 18 controls the vehicle-side communication device 12, the camera 14, and the positioning device 16. The vehicle-side control unit 18 is also configured as a computer equipped with a CPU and memory, and functions as the operation control unit 20 by the CPU executing a program stored in the memory.

The operation control unit 20 controls the operation of the vehicle (acceleration, deceleration, braking, steering) based on remote operation by the operator. The operation control unit 20 also repeatedly transmits vehicle information including identification information (vehicle ID) and current location information of the vehicle 10 to the remote assistance device 50 at a predetermined cycle via the vehicle-side communication device 12. The vehicle information may include information for identifying the vehicle model and occupants of the vehicle 10, and information indicating the location of the destination. When the operator provides remote assistance for the vehicle 10, the operation control unit 20 also transmits images captured by the camera 14 to the remote assistance device 50 via the vehicle-side communication device 12. In this embodiment, the captured images are, as an example, moving images (video).

The remote support device 50 includes a communication unit 52, a control unit 60, and a storage device 70.

The communication unit 52 performs wireless communication with the vehicle 10. The communication unit 52 also performs wired or wireless communication with the operator terminal 80.

The storage device 70 stores a map database 72, an operator information database 74, and a remote assistance program 76.

The positions of the support points are registered in the map database 72. The map database 72 also records road data showing the connection status of intersections and roads, congestion information, traffic regulation information, and traffic information including traffic volume for each support point. The traffic information is obtained from another server via the communication unit 52, for example, and stored.

The operator information database 74 records, for example, the schedule of each operator who provides remote support to vehicles, the number of operators currently available, the content of the remote support, the traffic volume history at each support location, the history of the time required for remote support, etc.

The control unit 60 controls the communication unit 52 and the storage device 70. Functionally, the control unit 60 includes a route planning unit 62, a remote support unit 64, a prediction unit 66, a recording unit 68, and an allocation unit 69.

The route planning unit 62 specifies the route along which each vehicle 10 will travel. More specifically, the route planning unit 62 specifies the route along which each vehicle 10 will travel from the departure point to the destination while passing through the support points, based on the road data, traffic information, and the positions of the support points recorded in the map database 72. The departure point, destination, and support points to be passed through are set, for example, by the operator or the administrator, or in response to a request from the vehicle 10. The route planning unit 62 distributes the specified route to each vehicle 10 via the communication unit 52. Note that the route may be specified in the vehicle-side control unit 18 of each vehicle 10. The route may also be specified in response to a request from each vehicle 10.

The remote support unit 64 provides remote support for the vehicle 10 at a support point based on remote operation by the operator. As shown in FIG. 3, the remote support unit 64 includes a front image transmission unit 40, a receiving unit 42, an operation signal transmission unit 44, and a rear image transmission unit 46.

When remote assistance at an assistance point is requested from the vehicle 10, the forward image transmission unit 40 transmits a forward image including at least the area in front of the vehicle 10 captured by the vehicle 10, for example a forward image including the forward range 30F, to the operator terminal 80 that remotely controls the vehicle 10.

The receiver 42 receives an operation signal corresponding to remote operation at the support point from the operator terminal 80.

The operation signal transmission unit 44 transmits the operation signal received from the operator terminal 80 to the vehicle 10.

After the vehicle 10 passes a switching point that is set to switch the display on the operator terminal 80 based on the support point, the rear image transmission unit 46 transmits to the operator terminal 80 a rear image that is captured by the vehicle 10 so as to include at least the rear of the vehicle 10 and emphasizes the rear, for example, a rear image including the rear range 30R.

The prediction unit 66 predicts the timing when remote support of the vehicle 10 will be started by the operator based on the estimated time when the vehicle 10 will arrive at a support point included in the route specified by the route planning unit 62, and predicts the time for which the remote support will be performed (hereinafter referred to as "required time"). The prediction unit 66 can, for example, learn the relationship between the past required time and traffic volume for each support point, and predict the required time from the learning results and the current traffic volume. The prediction unit 66 in this embodiment has a function of predicting the number of operators required to simultaneously support multiple vehicles based on the estimated time of arrival at a support point included in the route specified for each vehicle 10.

The recording unit 68 records in the operator information database 74 the history of the number of operators who provided remote support for each time period. The recording unit 68 also records in the operator information database 74 the history of the time required for each operator to remotely support the vehicle 10 at a support point for each support point. The recording unit 68 also records in the operator information database 74 the traffic volume history for each support point. The recording unit 68 also records the support content in the operator information database 74. The support content includes, for example, right turn support and left turn support at intersections, door opening/closing support and starting support at bus stops, obstacle avoidance, and handling of pedestrians crossing the road.

The allocation unit 69 allocates operators to the support points included in the route of each vehicle 10 at the start timing of remote support predicted by the prediction unit 66 or at the timing of receiving a remote support request from the vehicle 10, and determines a schedule for each operator. The allocation unit 69 records the determined schedule in the operator information database 74. In the schedule, for example, the time when remote support is to be performed, the support point, and the vehicle are registered for each operator. Note that in this embodiment, a case where an operator is allocated when a remote support request is received from a vehicle 10 will be described.

Figure 4 is a diagram showing the hardware configuration of the control unit 60. The control unit 60 is configured as a device including a general computer.

As shown in FIG. 4, the control unit 60 includes a CPU (Central Processing Unit) 60A, a ROM (Read Only Memory) 60B, a RAM (Random Access Memory) 60C, a non-volatile memory 60D, and an input/output interface (I/O) 60E. The CPU 60A, ROM 60B, RAM 60C, non-volatile memory 60D, and I/O 60E are connected to each other via a bus 60F. The communication unit 52 and the storage device 70 are connected to the I/O 60E.

The CPU 60A is an example of a computer. The term computer here refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

The storage device 70 is composed of a non-volatile storage device such as a hard disk. The CPU 60A reads and executes the remote assistance program 76 stored in the storage device 70. As a result, the CPU 60A functions as each functional unit constituting the control unit 60 described above. The remote assistance program 76 may be realized by storing it in a non-volatile non-transitive recording medium or distributing it via a network and installing it appropriately in the remote assistance device 50.

Examples of non-volatile non-transient recording media include CD-ROMs (Compact Disc Read Only Memory), optical magnetic disks, HDDs (hard disk drives), DVD-ROMs (Digital Versatile Disc Read Only Memory), flash memory, and memory cards.

The operator terminal 80 is configured as a computer or mobile terminal equipped with a CPU and memory. The autonomous driving system 100 in this embodiment includes one or more operator terminals 80. Each operator terminal 80 includes an operation unit 82 and a display unit 84. The operation unit 82 accepts operations for remotely supporting the vehicle 10 from the operator. The accepted operations are transmitted as operation signals to the vehicle 10 through the remote support device 50. The display unit 84 displays a user interface that is used for operation by the operator. The display unit 84 also displays images from the camera 14 transmitted from the vehicle 10.

The remote assistance process executed by the CPU 60A of the control unit 60 of the autonomous driving system 100 will be described with reference to FIG. 5. The remote assistance process shown in FIG. 5 is a process that is repeatedly executed at a predetermined interval by the remote assistance device 50 while the autonomous driving system 100 is operating. In the following description, it is assumed that the route for each vehicle 10 has been specified by the route planning unit 62.

When the vehicle 10 is driving autonomously along a set route and reaches an assistance point and determines that remote assistance from an operator is required, the vehicle 10 sends a remote assistance request to the remote assistance device 50.

Therefore, in step S100, the CPU 60A determines whether or not a remote assistance request has been received from the vehicle 10. If a remote assistance request has been received, the process proceeds to step S102. On the other hand, if a remote assistance request has not been received from the vehicle 10, the process waits until a remote assistance request is received.

In step S102, the CPU 60A executes an operator assignment process. Specifically, an operator is assigned to the vehicle 10 that has received the remote assistance request, for example, based on a schedule recorded in the operator information database 74.

In step S104, the CPU 60A receives the vehicle information and captured images transmitted from the vehicle 10. The captured images are four captured images (moving images) of each range captured by the cameras 14F, 14RS, 14LS, and 14R. The vehicle information and captured images are transmitted periodically from the vehicle 10, and are received successively from step S104 onwards until the routine ends.

In step S106, the CPU 60A calculates the position of a switching point at which the type of captured image to be transmitted to the operator terminal 80 is switched after the remote assistance is completed. Specifically, the position of the switching point is calculated based on the current position of the vehicle 10 included in the vehicle information received in step S104, i.e., the position of the assistance point where remote assistance was requested, and the route of the vehicle 10. Specifically, the position of the switching point is the position of a point where the operation corresponding to remote operation can be considered to have ended when the vehicle 10 passes the switching point.

For example, as shown in FIG. 6A, assume that the vehicle 10 is making a right turn at an intersection Cr according to a route Rt, and requests remote assistance at the intersection Cr, and is instructed to turn right as a remote operation. In this case, the point where the vehicle 10 turns right is set as the switching point Cg.

As another example, as shown in FIG. 7A, a situation is assumed in which the vehicle 10 travels along the non-priority lane S1 according to the route Rt, turns left at an intersection Cr, and travels along a priority road S2, and remote assistance is requested just before the intersection Cr, and a left turn is instructed as a remote operation. In this case, the point where the vehicle 10 turns left is set as the switching point Cg.

As another example, as shown in FIG. 8A, a situation is assumed in which vehicle 10 stops at bus stop BS according to route Rt, then departs and travels along lane S1, and remote assistance is requested at bus stop BS, and an instruction to merge onto road S1 is given as a remote operation. In this case, the point where vehicle 10 merges into lane S1 is set as switching point Cg.

As another example, as shown in FIG. 9A, a situation is assumed in which the vehicle 10 encounters an obstacle OB while traveling in the left lane S1 of two lanes along a route Rt, and the vehicle requests remote assistance just before the obstacle OB, and is instructed by remote control to travel in the adjacent lane S2 to the right. In this case, the point where the vehicle 10 changes lanes to lane S2 is set as the switching point Cg.

As another example, as shown in FIG. 10A, a case is assumed in which the vehicle 10 travels along the lane S1 following the route Rt, stops just before the crosswalk PC and requests remote assistance, and is instructed to start moving by remote control after the pedestrian PD crosses the crosswalk PC. In this case, the point where the vehicle 10 passes through the center of the crosswalk is set as the switching point Cg.

In step S108, the CPU 60A transmits to the operator terminal 80, as forward images, images including, for example, image 32F of the forward range 30F, image 32RS of the right side range 30RS, and image 32LS of the left side range 30LS, as shown in FIG. 6A, of the captured images of each range received in step S104. As a result, on the display unit 84 of the operator terminal 80, as shown in FIG. 11, captured image 32F is displayed in the center, captured image 32RS is displayed on the right, and captured image 32LS is displayed on the left.

In the examples of Figures 6A, 7A, and 10A, the forward image is a captured image of range 30 including forward range 30F, right side range 30RS, and left side range 30LS, but in a scene in which vehicle 10 travels diagonally forward to the right as shown in Figures 8A and 9A, for example, a captured image of range 30 including forward range 30F, right side range 30RS, and rear range 30R may be transmitted to operator terminal 80 as a forward image. Note that the process of step S108 is repeatedly executed each time a captured image is received from vehicle 10 until this routine is terminated.

The operator of the operator terminal 80 checks the captured image displayed on the display unit 84 and operates the operation unit 82 to remotely control the vehicle 10. For example, as shown in FIG. 6A, if the vehicle 10 is in a situation where it is about to turn right at an intersection Cr, the operator checks for oncoming vehicles and pedestrians, etc., and issues an instruction to turn right when it is determined that it is OK to turn right. When the operator operates the operation unit 82 to remotely control the vehicle 10, an operation signal corresponding to the remote operation is transmitted to the remote assistance device 50.

In step S110, the CPU 60A determines whether an operation signal has been received from the operator terminal 80. If an operation signal has been received, the process proceeds to step S112. On the other hand, if an operation signal has not been received, the process waits until an operation signal is received.

In step S112, the CPU 60A transmits the operation signal received in step S110 to the vehicle 10. As a result, the vehicle 10 performs an operation according to the operation signal. For example, as in the example of FIG. 6A, if the operation signal is an operation signal indicating an instruction to turn right, the vehicle 10 performs an operation to turn right. As a result, the vehicle 10 turns right at the intersection Cr, as shown in FIG. 6B.

In step S114, it is determined whether the vehicle 10 has passed the switching point Cg set in step S106. Specifically, it is determined whether the vehicle 10 has passed the switching point Cg based on the history of the current position of the vehicle 10 and the route contained in the vehicle information successively received from the vehicle 10.

If the vehicle 10 has passed the switching point Cg, the process proceeds to step S116. On the other hand, if the vehicle 10 has not passed the switching point Cg, the process waits until the vehicle 10 passes the switching point Cg.

In step S116, the CPU 60A transmits to the operator terminal 80, as rear images, images including the captured image 32R of the rear range 30R, the captured image 32RS of the right range 30RS, and the captured image 32LS of the left range 30LS, among the captured images of each range received from the vehicle 10. As a result, on the display unit 84 of the operator terminal 80, as shown in FIG. 12, the captured image 32R is displayed in the center, the captured image 32RS is displayed on the right, and the captured image 32LS is displayed on the left. That is, when the vehicle 10 passes the switching point Cg, the captured image 32F that was displayed in the center of the display unit 84 until then is switched to the captured image 32R. This allows the operator to check the rear of the vehicle 10 after remote assistance. The process of step S116 is repeated every time a captured image is received from the vehicle 10 until this routine is terminated.

In addition to the rear image, the captured image 32F of the forward range 30F may be transmitted to the operator terminal 80. As a result, as shown in FIG. 13, the captured image 32F is displayed at the top of the display unit 84, and the operator can check not only the rear but also the front. Furthermore, a single captured image obtained by combining the captured images 32R, 32RS, and 32LS may be generated as a rear image and transmitted to the operator terminal 80. As a result, as shown in FIG. 14, the rear image is displayed on the display unit 84 of the operator terminal 80 as a captured image 32P like a single panoramic image.

In the example of FIG. 6A, until the vehicle 10 passes the switching point Cg, which is the point where the vehicle 10 turns right, a forward image captured of the range 30 including the forward range 30F, the right side range 30RS, and the left side range 30LS is transmitted to the operator terminal 80 and displayed on the display unit 84.

Then, when the vehicle 10 passes the switching point Cg, as shown in FIG. 6B, a rear image of the area 30 including the rear area 30R, the right side area 30RS, and the left side area 30LS is transmitted to the operator terminal 80 and displayed on the display unit 84. The same applies to the cases of FIGS. 7B, 8B, 9B, and 10B.

In this way, when the switching point Cg is passed, the image displayed on the display unit 84 of the operator terminal 80 switches from a forward image to a rearward image. This allows the operator to easily evaluate whether the remote operation he or she instructed the vehicle 10 to perform was appropriate. For example, in the example of Figures 6A and 6B, when the vehicle 10 turns right at the intersection Cr and passes the switching point Cg, the operator checks the rearward image. If the operator can see a pedestrian crossing the crosswalk in the rearward image, the operator can recognize that the pedestrian may have been overlooked even though he or she is trying to cross the crosswalk PC. Therefore, the operator is prompted to pay special attention to whether there are pedestrians trying to cross the crosswalk from the next time, which helps improve the operator's situation judgment ability.

In the example of FIG. 6B, the rear image is an image of the range 30 including the rear range 30R, the right side range 30RS, and the left side range 30LS, but a rear image of a capture range determined according to the content of the remote operation may be transmitted to the operator terminal 80. For example, in the example of FIG. 6B, if the important range that the operator should check after the vehicle 10 turns right is the rear range 30R and the left side range 30LS, a capture image of the range 30 including the rear range 30R and the left side range 30LS may be transmitted to the operator terminal 80. In this way, the capture range of the rear image to be transmitted to the operator terminal 80 may be determined in advance according to the content of the remote operation.

In step S118, the CPU 60A determines whether the vehicle 10 has resumed autonomous driving. For example, after the vehicle 10 performs an operation corresponding to the operation signal, the vehicle 10 transmits a withdrawal request to the remote assistance device 50 to withdraw the remote assistance request, and resumes autonomous driving. Therefore, whether autonomous driving has resumed can be determined, for example, by whether a withdrawal request to withdraw the remote assistance request has been received from the vehicle 10.

If the vehicle 10 has resumed autonomous driving, the process proceeds to step S120. On the other hand, if the vehicle 10 has not resumed autonomous driving, the process waits until autonomous driving is resumed.

In step S120, the CPU 60A stops sending captured images to the operator terminal 80 and releases the operator from remote support of the vehicle 10.

In this manner, in this embodiment, until the vehicle 10 passes the switching point Cg, a forward image that includes at least the area in front of the vehicle 10, for example a forward image that includes the forward range 30F, is displayed on the display unit 84 of the operator terminal 80, and after the vehicle 10 passes the switching point Cg, a rear image that includes at least the area behind the vehicle 10, for example a rear image that includes the rear range 30R, is displayed on the display unit 84 of the operator terminal 80.

This allows the operator to easily and quickly evaluate whether the remote operation of the vehicle 10 that he or she performed was appropriate, thereby helping to improve the operator's ability to judge the situation.

<Second embodiment>

A second embodiment will be described. In the second embodiment, a gaze range is detected based on the operator's line of sight, and a rear image including the detected gaze range is displayed on the display unit 84 of the operator terminal 80.

As shown in FIG. 15, the operator terminal 80 according to this embodiment includes an operation unit 82, a display unit 84, a camera 86, and a gaze detection unit 88.

The camera 86 is positioned so that it can photograph the operator's eyes.

The gaze detection unit 88 detects the gaze position, which indicates at what point in the forward image displayed on the display unit 84 the operator is looking, based on the captured image including the operator's eyes captured by the camera 86, and sequentially transmits gaze position information indicating the detected gaze position to the remote support device 50.

The receiver 42 of the remote assistance device 50 receives the gaze position information transmitted from the operator terminal 80, and when the vehicle 10 passes the switching point Cg and transmits a rear image to the operator terminal 80, the rear image transmitter 46 identifies the gaze range based on the gaze position information received from the operator terminal 80 and transmits a rear image including the identified gaze range to the operator terminal 80. Here, the gaze range refers to the range in which the gaze position remains for a predetermined time (e.g., 2 seconds) or more.

When multiple gaze ranges exist, for example, when three gaze ranges M1, M2, and M3 exist as shown in FIG. 16, the remote assistance device 50 transmits a rear image including all of the gaze ranges M1, M2, and M3 to the operator terminal 80.

In addition, a rear image including the gaze range with the longest gaze time among the multiple gaze ranges may be transmitted to the operator terminal 80. For example, if the gaze range with the longest gaze time among gaze ranges M1, M2, and M3 is gaze range M1, a rear image including at least gaze range M1 may be transmitted to the operator terminal 80.

The rear image transmission unit 46 may also transmit to the operator terminal 80 a rear image in which an object that is identified based on the gaze range and that is likely to be overlooked by the operator is highlighted. For example, a known object extraction process is performed on the rear image to extract objects. Here, objects are moving objects such as people and vehicles. Of the objects extracted from the rear image, objects outside the gaze range are regarded as objects that are likely to be overlooked by the operator. Then, the rear image in which the object that is likely to be overlooked is highlighted is transmitted to the operator terminal 80. For example, as shown in FIG. 17, the object that is likely to be overlooked is highlighted by surrounding it with a frame W in the captured image 32R of the rear range 30R, and transmitted to the operator terminal 80. This allows the operator to easily recognize objects that he or she may have overlooked.

The rear image transmission unit 46 may also be configured to transmit to the operator terminal 80 a rear image in which an object that was not detected in the forward image but was newly detected in the rear image is highlighted. This allows the operator to easily check, for example, vehicles and pedestrians in the blind spot in the forward image that the operator was unable to recognize. In this case as well, as shown in FIG. 17, the newly detected object in the rear image is highlighted, for example by surrounding it with a frame W, and transmitted to the operator terminal 80. This allows the operator to easily recognize an object that was in the blind spot and went unnoticed.

The rear image transmission unit 46 may also transmit to the operator terminal 80 a rear image displaying speed information related to the speed change of the oncoming vehicle detected by the vehicle 10. In this case, the vehicle 10 is configured to be equipped with a radar device. The vehicle 10 detects speed information related to the speed change of the oncoming vehicle and transmits it to the remote support device 50. The rear image transmission unit 46 transmits to the operator terminal 80 a rear image displaying the speed information of the oncoming vehicle. For example, as shown in FIG. 18, if the speed information received from the vehicle 10 indicates that the oncoming vehicle has decelerated by 10 km/h, a rear image is transmitted to the operator terminal 80 in which the oncoming vehicle is surrounded by a frame W and the words "10 km/h deceleration" are displayed. This allows the operator to recognize that the oncoming vehicle has decelerated by 10 km/h due to his/her remote operation, and to easily recognize that the remote operation may not have been appropriate.

Also, as shown in FIG. 19, if an infrastructure camera 34 is installed that captures an overhead view of the area including the switching point Cg, the rear image transmission unit 46 may acquire an overhead image from the infrastructure camera 34 and transmit it to the operator terminal 80. As a result, an overhead image 36 as shown in FIG. 20 is displayed on the display unit 84 of the operator terminal 80, making it easy to check the situation around the vehicle 10.

<Third embodiment>

The third embodiment will be described. Note that the same parts as those in the above embodiments are given the same reference numerals and detailed descriptions will be omitted.

In the third embodiment, a modified example of the forward image transmitted to the operator terminal 80 in step S108 of FIG. 5 and a modified example of the rearward image transmitted to the operator terminal 80 in step S116 of FIG. 5 are described.

In the third embodiment, as shown in FIG. 21, the right side range 30RS shown in FIG. 2 is divided into a right front range 30FRS and a right rear range 30RRS. Also, the left side range 30LS shown in FIG. 2 is divided into two ranges, a left front range 30FLS and a left rear range 30RLS.

In the third embodiment, in step S108 of FIG. 5, the CPU 60A transmits an image including the captured image 32F of the forward range 30F, the captured image 32FRS of the right forward range 30FRS, the captured image 32FLS of the left forward range 30FLS, the captured image 32R of the rear range 30R, the captured image 32RRS of the right rear range 30RRS, and the captured image 32RLS of the left rear range 30RLS to the operator terminal 80 as a forward image.

As a result, as shown in FIG. 22, the display unit 84 of the operator terminal 80 displays the captured images 32F and 32R in the center, the captured images 32FRS and 32RRS on the right, and the captured images 32FLS and 32RLS on the left. Note that in the example of FIG. 21, the display area of the captured image 32F is larger than that of the captured image 32R, the display area of the captured image 32FRS is larger than that of the captured image 32RRS, and the display area of the captured image 32FLS is larger than that of the captured image 32RLS. In other words, the display unit 84 displays the front of the vehicle 10 more strongly than the rear of the vehicle 10.

In addition, in step S116 of FIG. 5, the CPU 60A transmits an image with the rear emphasized as a rear image to the operator terminal 80 as shown in FIG. 23. In the rear image shown in FIG. 23, compared to the front image shown in FIG. 22, the display area of the captured image 32R is larger than that of the captured image 32F, the display area of the captured image 32RRS is larger than that of the captured image 32FRS, and the display area of the captured image 32RLS is larger than that of the captured image 32FLS. In other words, the rear of the vehicle 10 is displayed on the display unit 84 with emphasis on the front of the vehicle 10.

<Fourth embodiment>

The fourth embodiment will be described. Note that the same parts as those in the above embodiments are given the same reference numerals and detailed descriptions will be omitted.

In the fourth embodiment, a modified example is described in which a rear image that highlights an object that the operator may have overlooked is sent to the operator terminal 80.

24 shows an example of a forward image that the CPU 60A transmits to the operator terminal 80 in step S108 of FIG. 5. In the example of FIG. 24, the forward image includes a captured image 32F of the forward range 30F, a captured image 32FRS of the right forward range 30FRS, a captured image 32FLS of the left forward range 30FLS, a captured image 32R of the rear range 30R, a captured image 32RRS of the right rear range 30RRS, and a captured image 32RLS of the left rear range 30RLS. In the example of FIG. 24, the display areas of the captured images 32F and 32R are the same in size, the display areas of the captured images 32FRS and 32RRS are the same in size, and the display areas of the captured images 32FLS and RLS are the same in size.

FIG. 25 also shows an example of a rear image that the CPU 60A transmits to the operator terminal 80 in step S116 in FIG. 5. In the example of FIG. 25, the rear image is an image in which the forward image shown in FIG. 24 is enhanced by surrounding objects that the operator may overlook with frames W1 and W2.

The method for extracting objects that may have been overlooked by the operator is the same as that described in the second embodiment, and for example, a known object extraction process is performed on the front image and rear image to extract objects. Then, of the objects extracted from the rear image, those that were not included in the front image are set as objects that may have been overlooked by the operator. The objects that may have been overlooked are then highlighted by surrounding them with frames W1 and W2, and transmitted to the operator terminal 80. This allows the operator to easily recognize objects that he or she may have overlooked.

Note that this disclosure is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

For example, in the above embodiment, a case has been described in which the operator remotely controls the vehicle 10 by operating the operator terminal 80, but the disclosed technology can also be applied to a case in which the vehicle 10 is remotely controlled by operating a remote control device equipped with a steering wheel, an accelerator pedal, and a brake pedal.

Although the above embodiment describes a case where an operator is supported in improving his/her situation judgment ability, the disclosed technology can also be applied to a case where a supervisor who supervises an operator is supported in improving his/her situation judgment ability. The disclosed technology can also be applied to a case where a supervisor evaluates the remote support provided to an operator and provides feedback to the operator on areas for improvement. The disclosed technology can also be applied to a case where another operator is supported in improving their situation judgment ability by presenting the results of remote support to the other operator. For example, the disclosed technology can also be applied to a case where it is desired to train new employees or to level-match operators.

In addition, the configuration of the autonomous driving system 100 described in the above embodiment (see FIG. 1) is merely an example, and it goes without saying that unnecessary parts may be deleted or new parts may be added without departing from the spirit of the present invention.

Furthermore, the processing flow of the remote assistance program 76 described in the above embodiment (see FIG. 5) is also one example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the present invention.

The control unit and the method described in the present disclosure may be realized by a special-purpose computer having a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the device and the method described in the present disclosure may be realized by a special-purpose computer having a processor configured by a dedicated hardware logic circuit. Alternatively, the device and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

The following notes are disclosed regarding the technology of the present disclosure.
<Additional Notes>
(Appendix 1)
a forward image transmission unit (40) that transmits, when a remote assistance at an assistance point is requested from an autonomous vehicle (10), a forward image including at least a front of the autonomous vehicle, captured by the autonomous vehicle, to a remote operation device (80) that remotely operates the autonomous vehicle;
a receiving unit (42) for receiving an operation signal corresponding to a remote operation at the support point from the remote operation device;
an operation signal transmission unit (44) that transmits the received operation signal to the autonomous driving vehicle;
a rear image transmission unit (46) that transmits to the remote control device a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear after the autonomous vehicle passes a switching point that is set to switch the display on the remote control device based on the assistance point; and
A remote support device (50) comprising:
(Appendix 2)
The remote assistance device according to claim 1, wherein the rear image transmission unit transmits a rear image of a shooting range determined according to the content of the remote operation.
(Appendix 3)
the receiving unit receives gaze position information regarding a gaze position of an operator of the remote control device in a forward image displayed on the remote control device,
The remote assistance device according to claim 1 or 2, wherein the rear image transmission unit transmits a rear image including the gaze range of the operator identified based on the gaze position information to the remote operation device.
(Appendix 4)
The remote assistance device according to claim 3, wherein, when there are a plurality of the gaze ranges, the rear image transmission unit transmits to the remote operation device a rear image including the gaze range having the longest gaze time among the plurality of the gaze ranges.
(Appendix 5)
The remote assistance device according to claim 3, wherein, when there are a plurality of gaze ranges, the rear image transmission unit transmits a rear image including all of the plurality of gaze ranges to the remote operation device.
(Appendix 6)
The remote assistance device according to any one of claims 1 to 5, wherein the rear image transmission unit transmits to the remote operation device a rear image in which an object that is likely to be overlooked by an operator of the remote operation device is highlighted.
(Appendix 7)
The remote assistance device according to any one of appendices 3 to 5, wherein the rear image transmission unit transmits to the remote operation device a rear image in which an object that is to be overlooked by the operator and that is identified based on the gaze range is highlighted.
(Appendix 8)
The remote assistance device according to any one of claims 1 to 7, wherein the rear image transmission unit transmits to the remote operation device a rear image in which a newly detected object in the rear image is highlighted.
(Appendix 9)
The remote assistance device according to any one of appendices 1 to 8, wherein the rear image transmission unit transmits to the remote operation device a rear image displaying speed information relating to a speed change of an oncoming vehicle detected by the autonomous driving vehicle.
(Appendix 10)
The remote assistance device according to any one of claims 1 to 9, wherein the rear image transmission unit obtains an overhead image captured from an overhead angle of an area including the switching point, and transmits the overhead image to the remote operation device.
(Appendix 11)
The remote assistance device according to claim 1 , wherein the rear image is an image in which a display area displaying the rear of the autonomously driven vehicle is larger than a display area displaying the front of the autonomously driven vehicle.
11. A remote assistance device according to any one of claims 1 to 10.
(Appendix 12)
At least one processor (60A)
When remote assistance at an assistance point is requested from an autonomous vehicle, a forward image including at least a front of the autonomous vehicle, which is captured by the autonomous vehicle, is transmitted to a remote control device that remotely controls the autonomous vehicle;
receiving an operation signal corresponding to a remote operation at the support point from the remote operation device;
Transmitting the received operation signal to the autonomous vehicle;
A remote assistance method that performs processing including transmitting, to the remote operation device, a rear image that is taken to include at least the rear of the autonomous vehicle and that emphasizes the rear, after the autonomous vehicle passes a switching point that is set to switch the display on the remote operation device based on the assistance point.
(Appendix 13)
At least one processor,
When remote assistance at an assistance point is requested from an autonomous vehicle, a forward image including at least a front of the autonomous vehicle, which is captured by the autonomous vehicle, is transmitted to a remote control device that remotely controls the autonomous vehicle;
receiving an operation signal corresponding to a remote operation at the support point from the remote operation device;
Transmitting the received operation signal to the autonomous vehicle;
A remote assistance program (76) for executing a process including transmitting, to the remote operation device, a rear image that is taken to include at least the rear of the autonomous vehicle and that emphasizes the rear, after the autonomous vehicle passes a switching point that is set to switch the display on the remote operation device based on the assistance point.

10 Vehicle, 40 Front image transmission unit, 42 Receiving unit, 44 Operation signal transmission unit, 46 Rear image transmission unit, 50 Remote support device, 52 Communication unit, 60 Control unit, 62 Route planning unit, 64 Remote support unit, 66 Prediction unit, 68 Recording unit, 69 Allocation unit, 70 Storage device, 72 Map database, 74 Operator information database, 76 Remote support program, 80 Operator terminal, 100 Autonomous driving system

Claims (13)

a forward image transmission unit (40) that transmits, when a remote assistance at an assistance point is requested from an autonomous vehicle (10), a forward image including at least a front of the autonomous vehicle, captured by the autonomous vehicle, to a remote operation device (80) that remotely operates the autonomous vehicle;
a receiving unit (42) for receiving an operation signal corresponding to a remote operation at the support point from the remote operation device;
an operation signal transmission unit (44) that transmits the received operation signal to the autonomous driving vehicle;
a rear image transmission unit (46) that transmits to the remote control device a rear image that is captured so as to include at least the rear of the autonomous vehicle and emphasizes the rear after the autonomous vehicle passes a switching point that is set to switch the display on the remote control device based on the assistance point; and
A remote support device (50) comprising: The remote assistance device according to claim 1 , wherein the rear image transmission unit transmits a rear image of a shooting range determined according to the content of the remote operation. the receiving unit receives gaze position information regarding a gaze position of an operator of the remote control device in a forward image displayed on the remote control device,
The remote assistance device according to claim 1 , wherein the rear image transmission unit transmits a rear image including the gaze range of the operator specified based on the gaze position information to the remote operation device. The remote assistance device according to claim 3 , wherein, when there are a plurality of the gaze ranges, the rear image transmission unit transmits to the remote operation device a rear image including the gaze range having the longest gaze time among the plurality of the gaze ranges. The remote assistance device according to claim 3 , wherein, when there are a plurality of the gaze ranges, the rear image transmission unit transmits a rear image including all of the plurality of the gaze ranges to the remote operation device. The remote assistance device according to claim 1 , wherein the rear image transmission unit transmits to the remote operation device a rear image in which an object that is likely to be overlooked by an operator of the remote operation device is highlighted. The remote assistance device according to any one of claims 3 to 5, wherein the rear image transmission unit transmits to the remote operation device a rear image in which an object that is identified based on the gaze range and that is likely to be overlooked by the operator is highlighted. The remote assistance device according to claim 1 , wherein the rear image transmission unit transmits to the remote operation device a rear image in which a newly detected object in the rear image is highlighted. The remote assistance device according to claim 1 , wherein the rear image transmission unit transmits to the remote operation device a rear image displaying speed information relating to a speed change of an oncoming vehicle detected by the autonomous driving vehicle. The remote assistance device according to claim 1 , wherein the rear image transmission unit obtains an overhead image obtained by capturing an area including the switching point from an overhead perspective and transmits the image to the remote operation device. The remote assistance device according to claim 1 , wherein the rear image is an image in which a display area displaying the rear of the autonomously driven vehicle is larger than a display area displaying the front of the autonomously driven vehicle. At least one processor (60A)
When remote assistance at an assistance point is requested from an autonomous vehicle, a forward image including at least a front of the autonomous vehicle, which is captured by the autonomous vehicle, is transmitted to a remote control device that remotely controls the autonomous vehicle;
receiving an operation signal corresponding to a remote operation at the support point from the remote operation device;
Transmitting the received operation signal to the autonomous vehicle;
A remote assistance method that performs processing including transmitting, to the remote operation device, a rear image that is taken to include at least the rear of the autonomous vehicle and that emphasizes the rear, after the autonomous vehicle passes a switching point that is set to switch the display on the remote operation device based on the assistance point. At least one processor,
When remote assistance at an assistance point is requested from an autonomous vehicle, a forward image including at least a front of the autonomous vehicle, which is captured by the autonomous vehicle, is transmitted to a remote control device that remotely controls the autonomous vehicle;
receiving an operation signal corresponding to a remote operation at the support point from the remote operation device;
Transmitting the received operation signal to the autonomous vehicle;
A remote assistance program (76) for executing a process including transmitting, to the remote operation device, a rear image that is taken to include at least the rear of the autonomous vehicle and that emphasizes the rear, after the autonomous vehicle passes a switching point that is set to switch the display on the remote operation device based on the assistance point.