JP7679731B2 - Remote Operation System - Google Patents

Description

This disclosure relates to a remote driving system for a vehicle.

Patent Document 1 describes a camera image display device used in a vehicle remote control system. In this remote control system, the remotely controlled vehicle captures images of the surroundings of the vehicle using television cameras mounted on the front, rear, left and right sides, and transmits the images to the remote control system. The camera image display device of the remote control system displays the received images of the surroundings of the vehicle on a monitor. At this time, the camera image display device switches the display state of the monitor image so that the image in the direction of travel or operation of the vehicle is enlarged.

Japanese Patent Application Publication No. 10-145777

Imagine an operator watching an image displayed on a monitor and operating a remote control member of a remote driving system to remotely control a vehicle to drive it in reverse. If an image of the rear of the vehicle taken from the front by the vehicle's rear camera is displayed directly on the monitor, the direction in which the wheels are turned when driving in reverse (i.e., the steering direction of the vehicle's steering device) will be opposite to the left-right direction on the displayed screen. In this state, when driving the vehicle in reverse, the operator must operate the remote control member in the opposite direction to the direction shown in the image displayed on the monitor, which makes remote control feel strange.

To solve the problems described above, the present disclosure aims to provide a technology that can match the left and right directions on the image displayed on the monitor with the direction of operation of the remote control members of the remote driving device, even when the vehicle is traveling in reverse.

The present disclosure provides, as an example, a remote driving device. The remote driving device includes a receiving unit, a display unit, an image switching unit, a remote control member, and an operation amount setting unit. The receiving unit receives image data of the surroundings of the vehicle. The image data of the surroundings of the vehicle is acquired by a front camera mounted on the vehicle to acquire image data in front of the vehicle, and a rear camera mounted on the vehicle to acquire image data behind the vehicle. The display unit displays an image of the surroundings of the vehicle based on the image data. When the traveling direction of the vehicle is a forward direction, the image switching unit displays an image of the front of the vehicle based on the image data acquired by the front camera, and when the traveling direction of the vehicle is a reverse direction, the image displayed on the display unit displays an image of the rear of the vehicle based on the image data acquired by the rear camera. The remote control member is a member for remotely operating a steering device of the vehicle. The operation amount setting unit sets an operation amount of the steering device of the vehicle according to an operation amount of the remote control member. In addition, the operation amount setting unit sets the operation amount of the steering device so that the rotation direction of the operation amount of the steering device and the rotation direction of the operation amount of the remote control member are opposite when the vehicle is traveling backwards.

According to the remote driving device disclosed herein, when the vehicle is traveling in reverse, the left-right direction of the image on the display unit and the left-right direction of the vehicle relative to the vehicle's forward direction are reversed, but the amount of operation of the remote control member is converted so that the rotation direction is reversed and transmitted to the vehicle. Therefore, an operator operating the remote control member can operate the vehicle's steering device in the desired direction by operating the remote control member according to the left-right direction on the image displayed on the display unit. Therefore, the operator can remotely operate the vehicle without feeling any discomfort even when traveling in reverse.

1 is a conceptual diagram showing an overview of a remote operation system according to a first embodiment of the present invention; FIG. 2 is a conceptual diagram for explaining an overview of remote control of a vehicle according to the first embodiment. 2 is a diagram for explaining a schematic configuration of a monitor of the remote driving device according to the first embodiment. FIG. 10A and 10B are diagrams for explaining the relationship between the display on the monitor and the steering direction of the steering wheel of the remote control member when driving backward in the first embodiment. 13 is a diagram for explaining the relationship between the display on the monitor and the steering direction of the steering wheel of the remote control member when driving backward in another example of the first embodiment. FIG.

1. First embodiment
1-1. Configuration of the remote driving system FIG. 1 is a conceptual diagram showing an overview of a remote driving system 1 according to the present embodiment. The remote driving system 1 includes a remote driving device 100 and a vehicle 200 that is the subject of remote operation (remote driving). The remote driving device 100 is a device for remotely operating the vehicle 200, and is used by an operator D. The remote driving device 100 and the vehicle 200 are connected to each other so as to be able to communicate with each other via a wireless communication network 10. The wireless communication network 10 is, for example, a 5G network.

Vehicle 200 is, for example, an automobile (e.g., a passenger car, a truck, and a bus) of various sizes and weights. Vehicle 200 is equipped with a steering device, a drive device, a braking device, an on-board electronic control unit (on-board ECU), a communication device, etc. The steering device steers the wheels of vehicle 200. The drive device generates the driving force of vehicle 200, and the braking device generates the braking force of vehicle 200.

The vehicle 200 is also equipped with a front camera 201 and a rear camera 202. The front camera 201 and the rear camera 202 are installed facing the front and rear of the vehicle 200, respectively, and capture the surroundings of the vehicle on the front and rear sides of the vehicle 200. The image data obtained by the front camera 201 and the rear camera 202 is transmitted to the remote driving device 100 via a communication device. In this embodiment, there is no limit to the number of cameras mounted on the vehicle 200 as long as the surrounding environment of the vehicle 200 can be captured. For example, the vehicle 200 may be equipped with multiple front cameras 201 and multiple rear cameras 202. The vehicle 200 may also be equipped with a left camera and a right camera. In this case, the left camera captures the surroundings of the vehicle on the left side of the vehicle 200, and the right camera captures the surroundings of the vehicle on the right side of the vehicle 200, and the obtained image data is transmitted to the remote driving device 100.

The remote driving device 100 includes a monitor 110, a remote control member 120, a control device 130, and a receiving unit 140. The receiving unit 140 receives various information transmitted from the vehicle 200. The received information includes video data captured by the front camera 201 and the rear camera 202.

The remote control member 120 is a member that is operated by the operator D when remotely operating the vehicle 200. The remote control member 120 includes a steering wheel 121. Although not shown, the remote control member 120 also includes, in addition to the steering wheel 121, an accelerator pedal and a brake pedal, for example. Note that the member for remotely operating the steering device is not limited to the steering wheel 121. For example, the remote driving device 100 may include a joystick or the like as the remote control member 120, which allows the driving of the vehicle 200 to be remotely controlled.

Figure 2 is a conceptual diagram for explaining an overview of remote operation of the vehicle 200. The operator D can remotely operate the steering device of the vehicle 200 by operating the steering wheel 121. The amount of operation of the steering wheel 121 by the operator D is the steering angle. For convenience, the amount of operation of the steering wheel 121 will be referred to below as the "first amount of operation." The first amount of operation of the steering wheel 121 represents the intention of the operator D.

The control device 130 has a function as an operation amount setting unit. When the control device 130 functions as an operation amount setting unit, it calculates a second operation amount by reflecting a first operation amount of the steering 121, and transmits the second operation amount to the vehicle 200 via the wireless communication network 10. The second operation amount calculated here may be the first operation amount itself, or may be a corrected operation amount obtained by correcting the operation amount.

The control device 220 of the vehicle 200 receives the second operation amount from the remote driving device 100. The control device 220 of the vehicle 200 controls the steering device 210 of the vehicle 200 according to the second operation amount. The steering device 210 of the vehicle 200 may include, for example, an electric power steering such as a rack-parallel EPS.

The vehicle 200 also has an operating section including a steering wheel, an accelerator pedal, and a brake pedal. The driver of the vehicle 200 can operate (drive) the vehicle 200 by operating the operating section of the vehicle 200. In the case of manual driving, the control device of the vehicle 200 operates the vehicle 200 according to the amount of operation of the operating section by the driver. On the other hand, in the case of remote operation (remote driving), the control device 220 of the vehicle 200 operates the vehicle 200 according to operation instruction information from the remote driving device 100. The vehicle 200 may be capable of switching between manual driving and automatic driving. The remote operation of the vehicle 200 may be performed to assist the automatic driving of the vehicle 200.

Figure 3 is a diagram for explaining the schematic configuration of the monitor 110 of the remote driving device 100. In the example shown in Figure 3, the monitor 110 is composed of a left monitor 111, a center monitor 112, and a right monitor 113. However, the number of monitors constituting the monitor 110 is not limited to three. In Figure 3, the monitor 110 displays an image of the surroundings of the vehicle based on image data acquired by the front camera 201 and the rear camera 202 mounted on the vehicle 200. The center monitor 112 is the main display unit, and displays an image of the traveling direction of the vehicle 200. The left monitor 111 displays an image of the left or right side of the vehicle 200. The right monitor 113 displays an image of the left or right side of the vehicle 200. The operator D can operate the steering wheel 121 while watching the image displayed on the monitor 110.

The control device 130 of the remote driving device 100 also functions as an image switching unit that switches the image displayed on the monitor 110. When the vehicle 200 is traveling forward, the control device 130 causes the monitor 110 to display the surrounding image of the vehicle 200 acquired by the forward camera 201. The example in FIG. 3 is an example of a display during forward traveling, in which the central monitor 112 displays an image in front of the vehicle 200, the left monitor 111 displays an image of the surroundings on the left side of the vehicle 200 following the left end of the image on the central monitor 112, and the right monitor 113 displays an image of the surroundings on the right side of the vehicle 200 following the right end of the image on the central monitor 112.

1-2. Control when vehicle is traveling in reverse The control device 130 of the remote driving device 100 can determine whether the vehicle 200 is traveling forward or backward. Specifically, the control device 130 determines that the vehicle 200 is traveling in reverse based on information from the remote cockpit, and changes the mode of the vehicle 200 to reverse. Alternatively, the control device 130 determines that the vehicle 200 is traveling in reverse based on a shift operation by the operator D, and changes the mode of the vehicle 200 to reverse.

Figure 4 is a diagram for explaining the relationship between the display on the monitor 110 and the steering direction of the steering wheel 121 when driving backward. When the vehicle 200 is switched to reverse mode, the control device 130 switches the image on the center monitor 112 to the image from the rear camera 202. The display on the left monitor 111 and the right monitor 113 is not switched, and the left monitor 111 and the right monitor 113 display the surrounding images on the left and right sides of the vehicle 200 captured by the front camera 201, the same as when driving forward.

The rear camera 202 captures an image of the rear side of the vehicle 200 from the front side (i.e., the vehicle 200 side). Therefore, as shown in FIG. 4, the image from the rear camera 202 is displayed directly on the central monitor 112, and the left and right directions confirmed on the screen of the central monitor 112 are opposite to the steering direction of the wheels of the vehicle 200. Therefore, for example, as shown in FIG. 4, when the vehicle 200 is traveling backwards and the driver tries to turn the vehicle 200 to the right of the screen while watching the monitor 110, the steering direction of the wheels of the vehicle must actually be to the left.

In this embodiment, when the vehicle 200 is traveling backwards, the control device 130 calculates an operation amount in which the rotation direction of the first operation amount, which is the operation amount of the steering wheel 121, is reversed from left to right, and transmits this to the vehicle 200 as the second operation amount. As a result, when the vehicle 200 is traveling backwards, the turning direction of the vehicle 200 that can be confirmed on the central monitor 112 and the operation direction of the steering wheel 121 are made to match. In other words, the operator D can turn the vehicle 200 in the desired direction even when traveling backwards by rotating the steering wheel 121 in the direction displayed on the screen while checking the central monitor 112, just as when the vehicle 200 is traveling forward.

Note that the first operation amount is reversed only when the steering wheel 121 is in the neutral position. In order to ensure that the first operation amount is reversed when driving backward, the steering wheel 121 and the tire angle of the vehicle 200 are automatically transitioned to the neutral position when the vehicle is switched to driving backward.

When the first operation amount is reversed, a guard value is set for the amount of change in the steering device on the vehicle 200 side to suppress a sudden change in the tire angle. After the instruction to reverse the first operation amount is given, the vehicle is kept stopped until the change in the tire angle is complete.

As described above, according to this embodiment, even when driving in reverse, the vehicle 200 can be remotely controlled by operating the steering wheel 121 in the same direction as the screen while checking the screen displayed on the central monitor 112. This allows the vehicle 200 to be driven forward or backward without distinction, particularly when the vehicle 200 to be remotely controlled is configured symmetrically on the front and rear and left and right, and does not have a concept of forward or backward movement.

2. Other Embodiments Instead of controlling the first operation amount to be inverted left and right and used as an instruction value for the steering device of the vehicle 200 when traveling backward, the screen of the central monitor 112 may be inverted left and right.

Figure 5 is a diagram for explaining the relationship between the display on the monitor 110 and the steering direction of the steering wheel 121 when driving backward. The screen displayed on the central monitor 112 in Figure 5 is in the same location as the screen displayed on the central monitor 112 in Figure 4, but the screen in Figure 5 is a left-right inverted version of the image in Figure 4. In this way, by displaying the image from the rear camera 202 in a left-right inverted manner, the direction in which the wheels are steered when driving backward can be made to match the steering direction of the steering wheel 121. This allows operator D to operate the steering wheel 121 while watching the central monitor 112 in the same way as when the vehicle 200 is driving forward.

As shown in FIG. 5, when the screen of the center monitor 112 is switched to an inverted image of the rear of the vehicle 200 during reverse driving, the images of the left monitor 111 and the right monitor 113 may also be switched. In this case, for example, the screen displayed on the left monitor 111 and the screen displayed on the right monitor 113 may be swapped. At this time, the screens displayed on the left monitor 111 and the right monitor 113 may also be displayed in a horizontally inverted manner.

In addition, if the remote driving device 100 is configured to be able to display a navigation screen that displays a car navigation map, and if the screen of the central monitor 112 is displayed in a left-right inverted manner when driving in reverse, the map displayed on the navigation screen may also be displayed in a left-right inverted manner. Similarly, if the remote driving device 100 is configured to have a voice guidance device that provides voice guidance for remote operation, and if the screen of the central monitor 112 is displayed in a left-right inverted manner when driving in reverse, the left-right direction of the voice instructions may be reversed to provide guidance.

The remote driving device 100 may be configured to be capable of executing both a first mode in which the first operation amount is reversed left and right and transmitted to the vehicle 200 when driving in reverse, and a second mode in which the display on the monitor 110 is reversed left and right. In this case, the remote driving device 100 may be configured to be able to select the mode when driving in reverse between the first mode and the second mode.

For example, in the first mode, the first operation amount is reversed, so the delay is likely to be greater than in the second mode. Therefore, the second mode is more convenient for operations that require frequent switching between forward and reverse, such as when parking the vehicle 200.

On the other hand, in the second mode, the screen of the central monitor 112 is inverted left to right when driving in reverse, so the text and signs are also inverted left to right. This may reduce the readability of the text and signs displayed on the monitor 110. Therefore, the first mode is more convenient when driving a relatively long distance while driving in reverse, for example. Also, the first mode is more convenient for vehicles that do not have the concept of forward and reverse, such as the above-mentioned vehicles that do not switch between forward and reverse for a while after the direction of travel is determined when the IG is turned on.

REFERENCE SIGNS LIST 1 Remote driving system 10 Wireless communication network 100 Remote driving device 110 Monitor 111 Left monitor 112 Center monitor 113 Right monitor 120 Remote control member 121 Steering 130 Control device 200 Vehicle 201 Front camera 202 Rear camera 210 Steering device 220 Control device

Claims (2)

A receiving unit that receives image data of the surroundings of the vehicle acquired by a front camera mounted on the vehicle and acquiring image data of the front of the vehicle and a rear camera mounted on the vehicle and acquiring image data of the rear of the vehicle;
a display unit that displays an image of the surroundings of the vehicle based on the image data;
A remote control member for remotely controlling a steering device that steers the wheels of the vehicle;
A control device;
Equipped with
The control device includes:
an image switching unit that switches the image to be displayed on the display unit to an image of the front of the vehicle based on image data acquired by the front camera when the traveling direction of the vehicle is a forward direction, and switches the image to an image of the rear of the vehicle based on image data acquired by the rear camera when the traveling direction of the vehicle is a backward direction;
an operation amount setting unit that sets an operation amount of the steering device in accordance with an operation amount of the remote control member;
Equipped with
The operation amount setting unit is
a first setting state in which an operation amount of the steering device is set so that a steering direction of the wheels by the steering device and a steering direction of the remote control member are opposite to each other, and a second setting state in which an operation amount of the steering device is set so that a steering direction of the wheels by the steering device and a steering direction of the remote control member are the same,
The video switching unit is
When displaying an image behind the vehicle on the display unit, if the operation amount setting unit sets the steering amount of the steering device in the first setting state, the image can be displayed on the display unit in a first state that is not inverted left and right, and if the operation amount setting unit sets the steering amount of the steering device in the second setting state, the image can be displayed on the display unit in a second state that is inverted left and right,
The control device includes:
When the vehicle is moving backwards,
a first mode in which an image of the rear of the vehicle is displayed on the display unit in the first state and an operation amount of the steering device is set in the first setting state;
a second mode in which an image of the rear of the vehicle is displayed on the display unit in the second state and an operation amount of the steering device is set in the second setting state;
A remote driving device characterized by: Further comprising a voice guidance device for providing voice guidance on the remote operation,
The voice guidance device includes:
When the second mode is selected during reverse driving of the vehicle and an image of the rear of the vehicle is displayed on the display unit in the second state, a left-right direction of the voice instruction is reversed.
The remote driving device according to claim 1.

Images