JP7639612B2 - Remote Control System - Google Patents

Description

This disclosure relates to a remote control system.

In a remote control system in which a vehicle is driven by an operator operating a remote control device installed outside the vehicle, there is a risk that vehicle occupants may feel uneasy about driving the vehicle because the operator is outside the vehicle.

Several proposals have been made to alleviate such concerns among crew members.

For example, Patent Document 1 proposes displaying an image of an operator operating a remote control device on a vehicle monitor. This aims to alleviate the above concerns by enabling vehicle occupants to visually confirm the operator operating the device.

Patent Document 2 also proposes a configuration in which the operator's operation information is superimposed on an image that is substantially the same as the image of the vehicle's surroundings that is displayed for the operator on the remote control device and displayed on the vehicle monitor. This allows the vehicle occupants to understand the details of the operator's operations, eliminating the above-mentioned concerns.

JP 2017-158149 A JP 2020-166605 A

In the technology described in Patent Document 1, although the vehicle occupants can see the operator, it is difficult for them to predict the operation content of the operator, i.e., the behavior of the vehicle, and there is room for improvement in alleviating the anxiety of the occupants.

In addition, with the technology described in Patent Document 2, the vehicle occupant can understand the operation content (vehicle behavior) of the operator, but it is difficult to understand the operator's intention regarding the operation content. In other words, there is room for improvement in alleviating the anxiety of the occupants.

This disclosure has been made in consideration of the above-mentioned facts, and aims to provide a remote control system that allows a vehicle occupant to understand the intentions of the remote operator of the remote control device.

The remote control system according to the present disclosure is a remote control system in which a remote operator drives a vehicle by operating a remote control device provided outside the vehicle, and includes an imaging unit provided on the vehicle for capturing an image of the surroundings of the vehicle, a first display unit provided on the remote control device for displaying the surroundings image transmitted from the vehicle, a viewpoint detection unit for detecting the viewpoint of the remote operator with respect to the surroundings image displayed on the first display unit, and a second display unit provided on the vehicle for displaying a viewpoint symbol indicating the viewpoint superimposed on the surroundings image.

In the present disclosure, in a remote control system, an image of the surroundings of a vehicle captured by an imaging unit of the vehicle is displayed on a first display unit of a remote control device, and a viewpoint detection unit detects where in the surrounding image the viewpoint of the remote operator of the remote control device is located (information regarding the viewpoint), and a viewpoint symbol indicating the viewpoint of the remote operator is superimposed on the surrounding image and displayed on a second display unit of the vehicle.

Therefore, by looking at the second display unit of the vehicle, the vehicle occupants can confirm where the remote operator's viewpoint (viewpoint symbol) is in the vehicle's surroundings (image) and can confirm the remote operator's intentions for operation. This can reduce the anxiety of the vehicle occupants due to the absence of a remote operator (driver).

The present disclosure has the effect of allowing vehicle occupants to understand the operation intentions of the remote operator of the remote control device.

1 is a conceptual diagram showing a schematic configuration of a vehicle control system configured to include a vehicle according to the present disclosure. FIG. 2 is a block diagram showing a hardware configuration of a vehicle. 2 is a block diagram showing an example of a functional configuration of a vehicle control device provided in a vehicle. FIG. FIG. 2 is a block diagram showing a hardware configuration of the remote control device. 2 is a block diagram showing an example of a functional configuration of a remote control device included in the remote operation device. FIG. 4 is a flowchart showing a flow of processing executed by a vehicle control device. 4 is a flowchart showing a flow of processing executed by a vehicle control device. 5 is a flowchart showing a flow of processing executed by the remote control device. 5 is a flowchart showing a flow of processing executed by the remote control device. 4A and 4B are diagrams illustrating operation images displayed on the remote control device when the vehicle is traveling straight. 1 is a diagram showing a schematic diagram of a guide image displayed on a vehicle when the vehicle is traveling straight ahead; 1 is a diagram showing a schematic diagram of a guide image displayed on a vehicle when the vehicle is traveling straight and increasing in speed; 10 is a diagram showing a schematic diagram of an operation image displayed on the remote control device when turning left; FIG. FIG. 2 is a diagram showing a schematic diagram of a guide image displayed on a vehicle when making a left turn;

The following describes a remote control system 10 according to an embodiment of the present invention, using Figures 1 to 14.

As shown in FIG. 1, the remote control system 10 according to this embodiment includes a vehicle 12 and a remote control device 14.

The vehicle 12 is equipped with a vehicle control device 20. The remote operation device 14 is configured to include a remote control device 40. In the above remote operation system 10, the vehicle control device 20 of the vehicle 12 and the remote control device 40 of the remote operation device 14 are connected to each other so that they can communicate with each other via a network N1. This network N1 is, for example, a wired and wireless communication network using a public line such as the Internet.

(Vehicle configuration)

The vehicle 12 is configured to be capable of performing manual driving based on the driving operation of a driver who is an occupant of the vehicle 12, and remote driving based on the driving operation of a remote operator (remote driver) using the remote control device 14. The vehicle 12 is normally set to a manual driving mode.

Figure 2 is a block diagram showing the hardware configuration of the devices installed in the vehicle 12. In addition to the vehicle control device 20 described above, the vehicle 12 includes an external sensor 24, an internal sensor 26, an input device 28, an actuator 30, and a user I/F (Inter Face) 32.

As shown in FIG. 2, the vehicle control device 20 includes a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a storage 20D, a communication I/F (Inter Face) 20E, and an input/output I/F 20F. The CPU 20A, ROM 20B, RAM 20C, storage 20D, communication I/F 20E, and input/output I/F 20F are connected to each other so as to be able to communicate with each other via a bus 20G.

The CPU 20A is a central processing unit that executes various programs and controls each part. That is, the CPU 20A reads the programs from the ROM 20B and executes the programs using the RAM 20C as a working area. In this embodiment, the programs are stored in the ROM 20B. When the CPU 20A executes the programs, the vehicle control device 20 functions as the surrounding information acquisition unit 200, the behavior information acquisition unit 210, the operation reception unit 220, the driving control unit 230, the authorization unit 240, the transmission unit 250, the reception unit 260, and the guidance image generation unit 270 shown in FIG. 3.

As shown in FIG. 2, ROM 20B stores various programs and various data. RAM 20C temporarily stores programs or data as a working area. Storage 20D is configured with a HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including an operating system and various data. Communication I/F 20E includes an interface for connecting to network N1 to communicate with remote control device 40. For example, a communication standard such as LTE or Wi-Fi (registered trademark) is used as the interface.

The communication I/F 20E in this embodiment transmits an image captured by the camera 24A (peripheral image 70, described later) to a remote control device 14 outside the vehicle 12 via the network N1, and receives an operation image 80 and viewpoint information, described later, from the remote control device 14. The communication I/F 20E also receives remote operation information, which is operation information for operating the vehicle 12, from the remote control device 14 via the network N1. The camera 24A is equipped with a microphone (not shown), and sound data recorded together with the captured image is transmitted and received together with the captured image data.

The input/output I/F 20F is an interface for communicating with each device mounted on the vehicle 12. In this embodiment, the vehicle control device 20 is connected to the external sensor 24, the internal sensor 26, the input device 28, the actuator 30, and the user I/F 32 via the input/output I/F 20F. Note that the external sensor 24, the internal sensor 26, the input device 28, the actuator 30, and the user I/F 32 may be directly connected to the bus 20G.

The external sensor 24 is a group of sensors that detects peripheral information around the vehicle 12. The external sensor 24 includes a camera 24A that captures an image of a predetermined range. The camera 24A corresponds to the "imaging unit."

The internal sensors 26 are a group of sensors that detect the driving state of the vehicle 12. The internal sensors 26 include a steering angle sensor 26A that detects the amount of steering wheel operation, and a vehicle speed sensor 26B that detects the speed (vehicle speed) of the vehicle 12.

The input device 28 is a group of switches operated by a passenger in the vehicle 12. The input device 28 includes a steering wheel 28A as a switch for steering the wheels of the vehicle 12, an accelerator pedal 28B as a switch for accelerating the vehicle 12, and a brake pedal 28C as a switch for decelerating the vehicle 12.

The actuators 30 include a steering wheel actuator that drives the steered wheels of the vehicle 12, an accelerator actuator that controls the acceleration of the vehicle 12, and a brake actuator that controls the deceleration of the vehicle 12.

The user I/F 32 includes a display 32A and a speaker (not shown) which is an audio output unit. The display 32A is disposed, for example, on the instrument panel of the vehicle 12. The speakers are disposed, for example, on each of the left and right side doors of the vehicle 12. The display 32A corresponds to the "second display unit."

Figure 3 is a block diagram showing an example of the functional configuration of the vehicle control device 20. As shown in Figure 3, the vehicle control device 20 has a surrounding information acquisition unit 200, a behavior information acquisition unit 210, an operation reception unit 220, a driving control unit 230, an authorization unit 240, a transmission unit 250, a reception unit 260, and a guidance image generation unit 270. Each functional configuration is realized by the CPU 20A reading and executing a program stored in the ROM 20B.

The surrounding information acquisition unit 200 has a function of acquiring surrounding information of the surroundings of the vehicle 12. The surrounding information acquisition unit 200 acquires surrounding information of the vehicle 12 from the external sensor 24 via the input/output I/F 20F. This surrounding information includes an image of the surroundings of the vehicle 12 captured by the camera 24A (hereinafter referred to as the "surrounding image").

The behavior information acquisition unit 210 has a function of acquiring behavior information related to the behavior of the vehicle 12, such as the steering angle and vehicle speed of the vehicle 12. The behavior information acquisition unit 210 has a function of acquiring the behavior information of the vehicle 12 from the internal sensor 26 via the input/output I/F 20F.

The operation reception unit 220 has a function of receiving signals output from each input device 28 when manual driving is performed based on the operation of the occupant of the vehicle 12. The operation reception unit 220 generates vehicle operation information, which is operation information for controlling the actuator 30, based on the signals received from each input device 28.

The driving control unit 230 has the function of controlling remote driving based on remote operation information received (acquired) from the remote control device 14, and manual driving based on vehicle operation information received from the operation reception unit 220.

The authority granting unit 240 has a function of granting operation authority, which is the authority to operate the vehicle 12 equipped with the vehicle control device 20, to a remote operator who operates the remote operation device 14. When operation authority is granted to a remote operator, the vehicle control device 20 transmits an authority granting signal to the remote operation device 14 operated by that remote operator. By transferring the operation authority from the vehicle 12 to the remote operator, the vehicle 12 runs based on the remote operation information received from the remote operation device 14. In other words, remote driving is performed by the remote operator.

When the remote driving by the remote control device 14 is to be ended, the authority granting unit 240 transmits an authority transfer signal to the remote control device 14 to notify the remote control device 14 that the operating authority will be transferred to the vehicle 12. When the remote control device 14 receives the authority transfer signal, it executes a termination process to end the remote driving.

During remote driving, the transmission unit 250 has the function of acquiring specification information (dimensions, etc.) of the vehicle 12 from the storage 20D, adding the specification information to the surrounding images and behavior information of the vehicle 12 to generate vehicle information, and transmitting this to the remote control device 40.

The receiving unit 260 has the function of receiving the operation image and viewpoint information transmitted from the remote control device 40 during remote operation.

The guide image generating unit 270 has a function of generating a guide image 100 (see Figures 11, 12, and 14) in which a viewpoint symbol 90 (black circle portion) indicating the operator's viewpoint is superimposed on the operation image 80 based on the viewpoint information, and displaying the guide image 100 on the display 32A.

(Remote control device)

Figure 4 is a block diagram showing the hardware configuration of the equipment mounted on the remote control device 14 of this embodiment. In addition to the remote control device 40 described above, the remote control device 14 includes a display 42, a speaker 44, and an input device 48. The display 42 corresponds to the "first display unit."

The remote control device 40 includes a CPU 40A, a ROM 40B, a RAM 40C, a storage 40D, a communication I/F 40E, and an input/output I/F 40F. The CPU 40A, the ROM 40B, the RAM 40C, the storage 40D, the communication I/F 40E, and the input/output I/F 40F are connected to each other via a bus 40G so that they can communicate with each other. The functions of the CPU 40A, the ROM 40B, the RAM 40C, the storage 40D, the communication I/F 40E, and the input/output I/F 40F are similar to those of the CPU 20A, the ROM 20B, the RAM 20C, the storage 20D, the communication I/F 20E, and the input/output I/F 20F of the vehicle control device 20 described above.

The CPU 40A reads the program from the ROM 40B and executes the program using the RAM 40C as a working area. In this embodiment, the program is stored in the ROM 40B. When the CPU 40A executes the program, the remote control device 40 functions as the vehicle information acquisition unit 400, the operation image generation unit 410, the viewpoint detection unit 420, the operation information generation unit 430, and the operation switching unit 440 shown in FIG. 5.

In this embodiment, the remote control device 40 is connected to a display 42, a speaker 44, a camera 46, and an input device 48 via an input/output I/F 40F. The display 42, the speaker 44, the camera 46, and the input device 48 may also be directly connected to the bus 40G.

The display 42 is a liquid crystal display for displaying images of the surroundings of the vehicle 12 captured by the camera 24A of the vehicle 12, as well as various information related to the vehicle 12. The speaker 44 outputs sounds recorded together with the captured images by a microphone attached to the camera 24A of the vehicle 12.

The camera 46 constitutes a gaze detection device that detects the gaze of the remote operator, captures an image of the face of the remote operator seated in front of the display 42 from the front, and outputs the captured image to the remote control device 40.

The input device 48 is a controller operated by a remote operator using the remote control device 14. The input device 48 includes a steering wheel 48A as a switch for steering the wheels of the vehicle 12, an accelerator pedal 48B as a switch for accelerating the vehicle 12, and a brake pedal 48C as a switch for decelerating the vehicle 12. Note that the form of each input device 48 is not limited to this. For example, a lever switch may be provided instead of the steering wheel 48A. Also, for example, a push button switch or a lever switch may be provided instead of the pedal switches of the accelerator pedal 48B and the brake pedal 48C.

FIG. 5 is a block diagram showing an example of the functional configuration of the remote control device 40. As shown in FIG. 5, the remote control device 40 has a vehicle information acquisition unit 400, an operation image generation unit 410, a viewpoint detection unit 420, an operation information generation unit 430, and an operation switching unit 440.

The vehicle information acquisition unit 400 has the function of acquiring images and recorded sounds from the camera 24A transmitted from the vehicle control device 20, as well as vehicle information such as steering angle and vehicle speed.

As shown in FIG. 10, the operation image generating unit 410 has a function of creating an auxiliary image 60 that assists the operator's operation based on the acquired vehicle information, i.e., the vehicle 12's specification information, steering angle, vehicle speed, and other behavior information, and generating an operation image 80 by superimposing the auxiliary image 60 on the acquired captured image (surrounding image 70 of the vehicle 12). The operation image generating unit 410 also has a function of displaying the operation image 80 on the display 42 and transmitting the operation image 80 to the vehicle control device 20.

The viewpoint detection unit 420 has the function of detecting the line of sight of the remote operator based on the image captured by the camera 46, and detecting the viewpoint (point of gaze) of the remote operator on the peripheral image displayed on the display 42.

The operation information generating unit 430 has a function of receiving signals output from each input device 48 when remote driving is performed based on the operation of a remote operator. The operation information generating unit 430 also has a function of generating remote operation information based on the signals received from each input device 48 and transmitting the information to the vehicle control device 20.

The operation switching unit 440 has a function of controlling switching regarding remote driving of the vehicle 12. Specifically, when the operation switching unit 440 receives an authority granting signal from the vehicle 12, it executes processing for remote driving of the vehicle 12. In addition, when the operation switching unit 440 receives an authority transfer signal from the vehicle 12, it executes termination processing for terminating remote driving of the vehicle 12.

(Control flow)

Next, the control flow in the vehicle 12 and the remote control device 14 that constitute the remote control system 10 will be explained using Figures 6 to 9.

(Processing in the vehicle control device)

Next, the flow of processing executed in the vehicle control device 20 of the vehicle 12 will be described using the flowcharts of Figures 6 and 7. Note that in this description of processing, general remote operation, i.e., by operating the input device 48 of the remote control device 14, remote operation information is transmitted from the remote control device 40 to the vehicle control device 20, and the actuator 30 of the vehicle 12 is controlled based on this, is not described. In other words, the control of generating an operation image 80 in the remote control device 14 and displaying it on the display 42, and the control of generating a guidance image 100 in the vehicle 12 and displaying it on the display 32A will be described.

In step S10, the CPU 20A determines whether or not an occupant of the vehicle 12 has selected remote driving. Specifically, it determines whether or not a remote switching signal has been received. If this determination is positive, the process proceeds to the next step S12, and if this determination is negative, the process in step S10 is repeated.

If the process proceeds to step S12, the CPU 20A transmits an authorization signal to the remote control device 40.

In step S14, the CPU 20A determines whether or not an authority acceptance signal has been received from the remote control device 40. If the determination is positive, the process proceeds to the next step S16, and if the determination is negative, the process in step S14 is repeated.

In step S16, the CPU 20A generates vehicle information consisting of the surrounding image captured by the camera 24A, behavior information including the steering angle and vehicle speed detected by the steering angle sensor 26A and the vehicle speed sensor 26B, respectively, at the time of capture, and specification information (dimensions, etc.) of the vehicle 12 stored in the storage 20D. Note that the specification information of the vehicle 12 is generated only the first time and is omitted from the next time onwards.

Next, in step S18, the CPU 20A transmits the generated vehicle information to the remote control device 40.

In step S20, the CPU 20A determines whether or not an operation image has been received from the remote control device 40. If this determination is positive, the process proceeds to the next step S22, and if this determination is negative, the process in step S20 is repeated.

In step S22, the CPU 20A determines whether viewpoint information has been received from the remote control device 40. If this determination is positive, the process proceeds to the next step S24, and if this determination is negative, the process in step S22 is repeated.

In step S24, the CPU 20A generates a guide image 100 in which a viewpoint symbol 90 is superimposed on the operation image 80 based on the viewpoint information, as shown in FIG. 11.

In step S26, CPU 20A controls the display to display the guidance image on display 32A.

Furthermore, in step S28, the CPU 20A determines whether or not the occupant of the vehicle 12 has selected manual driving. Specifically, it determines whether or not a manual switching signal has been received. If this determination is positive, the process proceeds to step S30, and if this determination is negative, the process from step S16 onwards is repeated.

If the process proceeds to step S30, the CPU 20A sends an authority transfer signal to the remote control device 14 and ends the series of processes.

(Processing by remote control device)

Next, the flow of processing executed by the remote control device 14 will be explained using the flowcharts in Figures 8 and 9.

In step S40 of FIG. 8, the CPU 40A determines whether or not an authorization signal has been received. If the determination is positive, the process proceeds to the next step S42, and if the determination is negative, the process in step S40 is repeated.

In step S42, when the selected remote operator completes preparation for remote driving, the CPU 40A sends an authority acceptance signal to the vehicle control device 20.

In step S44, the CPU 40A acquires vehicle information (surrounding images, specification information of the vehicle 12, and behavior information of the vehicle 12) from the vehicle control device 20.

Next, in step S46, the CPU 40A creates an auxiliary image 60 (see the thick lines in FIG. 10) based on the specifications information of the vehicle 12 and the behavior information of the vehicle 12. That is, a pair of vehicle width lines 62A, 62B are created (at positions corresponding to both ends) based on the overall width of the vehicle 12, and a vehicle center line 64 is created (at a position corresponding to the center of the vehicle width). In addition, a tip guide line 66 is created that indicates the tip position of the vehicle 12 and extends in the vehicle width direction. Furthermore, a meter section 68 that displays the vehicle speed (numerical value) is created in the center of the tip guide line 66 in the vehicle width direction.

The vehicle speed is displayed in the meter section 68 of the auxiliary image 60 based on the behavior information of the vehicle 12.

Also, as shown in FIG. 13, the vehicle center line 64 and vehicle width lines 62A, 62B are tilted left and right from the front in accordance with the steering angle of the behavior information.

The lengths of the vehicle width lines 62A, 62B and the vehicle center line 64 may also be changed depending on the vehicle speed. For example, as shown in FIG. 12, the vehicle width lines 62A, 62B and the vehicle center line 64 may be controlled to become longer as the vehicle speed increases.

Next, in step S48, the CPU 40A creates an operation image 80 by superimposing the auxiliary image 60 on the peripheral image 70, as shown in FIG. 10.

Furthermore, in step S50, the CPU 40A causes the operation image 80 to be displayed on the display 42.

Furthermore, in step S52, the CPU 40A detects the line of sight of the remote operator from the image captured by the camera 46, and acquires viewpoint (gazing point) information on the operation image 80 displayed on the display 42 from the line of sight.

Next, in step S54, the CPU 40A transmits the operation image and viewpoint information to the vehicle control device 20.

In step S56, the CPU 40A determines whether the remote operation has ended, i.e., whether an authority transfer signal has been received. If the determination is positive, the series of controls ends, and if the determination is negative, the processing from step S44 onwards is repeated.

(Summary of this embodiment)

In the remote operation system 10 according to this embodiment, during remote driving, an operation image 80 is generated by the remote operation device 14 (remote control device 40) in which an auxiliary image 60 is superimposed on a surrounding image 70 of the vehicle 12 to be used by the remote operator for remote driving, and a guide image 100 is generated by the vehicle 12 (vehicle control device 20) in which a viewpoint symbol 90 indicating the viewpoint of the remote operator is superimposed on this operation image 80, and is displayed on the display 32A of the vehicle 12.

That is, the guide image 100 displays a viewpoint symbol 90 indicating the viewpoint of the remote operator on the operation image 80 (peripheral image 70), so that the occupant of the vehicle 12 can visually confirm where the remote operator is looking on the peripheral image 70, and can predict the movement (behavior) of the vehicle 12. For example, as shown in FIG. 11, when the viewpoint symbol 90 indicating the viewpoint of the remote operator in the guide image 100 is on a straight road, the vehicle 12 is expected to go straight through the intersection, and when the viewpoint symbol 90 indicating the viewpoint of the remote operator in the guide image 100 is on the left road, as shown in FIG. 14, the vehicle 12 is expected to turn left at the intersection. That is, by visually confirming the viewpoint symbol 90 of the guide image 100 displayed on the display 32A, the occupant of the vehicle 12 can confirm the operation (driving) intention of the remote operator and predict the behavior of the vehicle 12.

In addition, the auxiliary image 60 included in the guide image 100 displays the vehicle center line 64 and vehicle width lines 62A, 62B, the directions of which change according to the steering angle, so that the occupants of the vehicle 12 can predict which direction the vehicle 12 will head simply by visually checking the directions of the vehicle center line 64 and vehicle width lines 62A, 62B.

Furthermore, as shown in FIG. 12, the length of the vehicle center line 64 and vehicle width lines 62A, 62B change (extend) depending on the vehicle speed, and the vehicle speed is displayed on the meter section 68, so the driver can feel the change in vehicle speed.

(Additional explanation of the embodiment)

In the above embodiment, the vehicle 12 is configured to be capable of performing manual driving and remote driving, but the present invention is not limited to this. The vehicle 12 may be configured to be capable of performing manual driving, automatic driving, and remote driving. Alternatively, the vehicle 12 may be configured to be capable of performing only remote driving.

In addition, in the above embodiment, the auxiliary image 60 is superimposed on the peripheral image 70 to generate the operation image 80, and the viewpoint symbol 90 is superimposed on the operation image 80 to generate the guide image 100, but the guide image 100 may be generated simply by superimposing the viewpoint symbol 90 on the peripheral image 70.

Furthermore, in the above embodiment, the lengths of the vehicle center line 64 and vehicle width lines 62A, 62B in the auxiliary image 60 are configured to change depending on the vehicle speed, but this is not essential. Similarly, the meter section 68 in the auxiliary image 60 is provided to display the vehicle speed, but this is also not essential.

In addition, in the above embodiment, the orientation of the vehicle center line 64 and vehicle width lines 62A, 62B in the auxiliary image 60 is changed according to the steering angle, but this is not essential.

Furthermore, when the vehicle 12 is turning (when there is a steering angle), the vehicle center line 64 and vehicle width lines 62A, 62B in the guide image 100 (operation image 80) are displayed as straight lines inclined left and right from the front, but this is not limited to this. The vehicle center line 64 and vehicle width lines 62A, 62B may also be displayed as curved lines curving left and right from the front.

In addition, in the above embodiment, the remote operator's line of sight (point of view) is detected from an image captured by the camera 46 (of the remote operator's face), but this is not limited to this. For example, the remote operator may wear a glasses-type viewpoint detection device to detect the remote operator's line of sight (point of view).

Furthermore, although the guide image 100 is displayed on the display 32A provided in the passenger compartment of the vehicle 12, the present invention is not limited to this. For example, the guide image 100 may be displayed on the windshield of the vehicle 12 using a head-up display device.

In addition, in the above embodiment, the remote control device 40 transmits the operation image 80 and viewpoint information to the vehicle control device 20, and the vehicle control device 20 generates the guide image 100 by superimposing the viewpoint symbol 90 on the operation image 80, but the remote control device 40 may generate the guide image 100 and transmit the guide image 100 to the vehicle control device 20.

Furthermore, in the above embodiment, the processes executed by the CPU 20A, 40A by reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) that are processors with circuit configurations designed specifically to execute specific processes. Each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

In the above embodiment, the programs are described as being pre-stored (installed) in a non-transitory computer-readable recording medium. For example, in the remote control system 10, the programs are pre-stored in storages 20D and 40D. However, the present invention is not limited to this, and the programs may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. Each program may also be downloaded from an external device via a network.

The process flow described in the above embodiment is also one example, and unnecessary steps may be deleted, new steps may be added, or the process order may be rearranged, without departing from the spirit of the invention.

The present invention can be modified in various ways without departing from the spirit of the invention. It goes without saying that the scope of the invention is not limited to the above embodiment.

10 Remote control system 12 Vehicle 14 Remote control device 24A Camera (imaging unit)
32A Display (second display unit)
42 Display (first display unit)
70 Peripheral image 90 Viewpoint symbol 420 Viewpoint detection unit

Claims (1)

A remote operation system in which a vehicle is driven by a remote operator operating a remote operation device provided outside the vehicle,
An imaging unit provided in the vehicle and configured to capture an image of the surroundings of the vehicle;
A detection unit provided in the vehicle for detecting a steering angle and a vehicle speed of the vehicle;
a first display unit that is provided in the remote control device and displays the surrounding image transmitted from the vehicle;
a viewpoint detection unit that detects a viewpoint of the remote operator with respect to the peripheral image displayed on the first display unit;
a second display unit that is provided on the vehicle and displays a viewpoint symbol indicating the viewpoint on the peripheral image in a superimposed manner, and that increases the length of a line displayed at a position corresponding to a vehicle width center or both ends of the vehicle in the peripheral image as the vehicle speed increases, and changes the direction of the line in accordance with the steering angle or curves the line in accordance with the steering angle;
A remote control system comprising: