JP7658199B2 - Remote Control Device - Google Patents

Description

The present invention relates to the technical field of a remote driving device that remotely drives a target vehicle.

One such device that has been proposed is one that transmits images from multiple cameras mounted on vehicles to a control center and displays them on a monitor at the control center (see Patent Document 1).

Japanese Patent Application Publication No. 10-145777

The user who remotely drives the target vehicle from outside the vehicle is not in the target vehicle. This poses a technical problem in that there may be a discrepancy between the speed that the user perceives from, for example, an image (video) displayed on a monitor and the actual speed of the target vehicle.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a remote driving device that can bring the speed perceived by a user remotely driving a target vehicle closer to the actual speed of the target vehicle.

A remote driving device according to one embodiment of the present invention is a remote driving device that remotely drives a target vehicle, and is equipped with a generation means for generating synthetic sound information that includes external environmental sounds, which are sounds around the target vehicle collected by a microphone, and virtual driving sounds that simulate the driving sounds of the target vehicle, and when the speed of the target vehicle is high, the generation means increases the proportion of the external environmental sounds included in the synthetic sound information compared to when the speed of the target vehicle is low.

1 is a block diagram showing a configuration of a remote operation system according to a first embodiment. 3 is a diagram showing an example of an imaging range of a camera of a vehicle according to the first embodiment; FIG. FIG. 2 is a diagram illustrating an example of a display according to the first embodiment. FIG. 4 is a diagram illustrating an example of a distributor according to the first embodiment. 13 is an example of a map that defines a ratio between external environmental sounds and virtual driving sounds. 4 is a flowchart showing the operation of the remote driving device according to the first embodiment. 13 is another example of a map that defines the ratio between external environmental sounds and virtual driving sounds. 10 is a flowchart showing the operation of the remote driving device according to the second embodiment. FIG. 1 is a block diagram showing a configuration of a computer according to an embodiment.

First Embodiment
A first embodiment of a remote driving device will be described with reference to Fig. 1 to Fig. 6. A remote driving device 10 according to the first embodiment constitutes a part of a remote driving system 1. In Fig. 1, the remote driving system 1 includes the remote driving device 10 and a vehicle 20 that is remotely driven by the remote driving device 10.

The remote driving device 10 is designed to resemble, for example, a vehicle cockpit. The remote driving device 10 is provided with a steering wheel, an accelerator pedal, and a brake pedal (none of which are shown) that are operated by a user who remotely drives a target vehicle (here, vehicle 20) using the remote driving device 10.

The remote driving device 10 is configured with a control device 11, a communication unit 12, a steering sensor 13, an accelerator pedal sensor 14, a brake pedal sensor 15, a display 16, and a speaker 17. The steering sensor 13 detects the amount of operation of the steering wheel. The accelerator pedal sensor 14 detects the amount of operation of the accelerator pedal. The brake pedal sensor 15 detects the amount of operation of the brake pedal.

The vehicle 20 is configured with cameras 21L, 21C, and 21R, a microphone 22 (hereinafter referred to as "mic 22"), a speed sensor 23, an acceleration sensor 24, a yaw rate sensor 25, and a communication unit 26.

The microphone 22 detects sounds around the vehicle 20 (hereinafter referred to as "external environmental sounds"). The speed sensor 23 detects the speed of the vehicle 20. The acceleration sensor 24 detects the acceleration of the vehicle 20. The yaw rate sensor 25 detects the yaw rate of the vehicle 20. The external environmental sounds detected by the microphone 22, the speed detected by the speed sensor 23, the acceleration detected by the acceleration sensor 24, and the yaw rate detected by the yaw rate sensor 25 are transmitted to the remote driving device 10 via the communication unit 26.

Camera 21C is attached to vehicle 20 so as to be able to capture an image of the area in front of vehicle 20. Camera 21L is attached to vehicle 20 so as to be able to capture an image of the left side area of vehicle 20. Camera 21R is attached to vehicle 20 so as to be able to capture an image of the right side area of vehicle 20. Images captured by cameras 21L, 21C, and 21R are transmitted to remote driving device 10 via communication unit 26.

2, range IRL corresponds to the imaging range of camera 21L, range IRC corresponds to the imaging range of camera 21C, and range IRR corresponds to the imaging range of camera 21R. In addition to cameras 21L, 21C, and 21R, vehicle 20 may be equipped with other cameras, such as a camera capable of imaging the rear area of vehicle 20.

The control device 11 of the remote driving device 10 has a vehicle control unit 111, a display control unit 112, and a sound control unit 113, which are implemented either as a logical block logically implemented therein or as a processing circuit physically implemented therein.

The vehicle control unit 111 acquires the speed, acceleration, and yaw rate transmitted from the vehicle 20 via the communication unit 12. The vehicle control unit 111 calculates a command value for controlling the vehicle 20 based on the acquired speed, acceleration, and yaw rate, as well as the operation amount detected by the steering sensor 13, the operation amount detected by the accelerator pedal sensor 14, and the operation amount detected by the brake pedal sensor 15. The vehicle control unit 111 transmits the calculated command value to the vehicle 20 via the communication unit 12.

The display control unit 112 acquires images captured by each of the cameras 21L, 21C, and 21R via the communication unit 12. Here, the display 16 has display areas DRL, DRC, and DRR, for example, as shown in FIG. 3. The display control unit 112 displays the image captured by the camera 21L in the display area DRL, displays the image captured by the camera 21C in the display area DRC, and displays the image captured by the camera 21R in the display area DRR.

Each of the display areas DRL, DRC, and DRR may be realized as a partial area within one display. The display areas DRL, DRC, and DRR may be realized by multiple (e.g., three) displays. Furthermore, instead of individually displaying the images captured by each of the cameras 21L, 21C, and 21R, the display control unit 112 may synthesize the images captured by each of the cameras 21L, 21C, and 21R, and then display the synthesized image on the display 16.

The sound control unit 113 acquires the external environmental sound transmitted from the vehicle 20 via the communication unit 12. The sound control unit 113 generates a virtual running sound that simulates the running sound of the vehicle 20. The sound control unit 113 generates synthetic sound information that includes at least one of the external environmental sound and the virtual running sound. The sound control unit 113 controls the speaker 17 so that a sound indicated by the generated synthetic sound information is emitted.

Here, the sound control unit 113 may have a distributor 1131 (see FIG. 4). The synthetic sound information may be generated, for example, by inputting the external environmental sound and the virtual driving sound to the distributor 1131 and synthesizing the two by the distributor 1131. Note that various existing methods can be applied to the method of generating the virtual driving sound, so detailed explanations are omitted.

In particular, the sound control unit 113 changes the ratio of external environmental sounds and virtual running sounds included in the synthetic sound information according to the speed of the vehicle 20. The sound control unit 113 may change the ratio of external environmental sounds and virtual running sounds included in the synthetic sound information, for example, according to a map such as that shown in FIG. 5. When the speed of the vehicle 20 is relatively low, the synthetic sound information may include only virtual running sounds (i.e., the ratio of virtual running sounds may be 100%). When the speed of the vehicle 20 is relatively high, the synthetic sound information may include only external environmental sounds (i.e., the ratio of external environmental sounds may be 100%).

The operation of the remote driving device 10 configured as described above will now be described with reference to the flowchart of FIG. 6. In FIG. 6, the control device 11 of the remote driving device 10 acquires the speed of the vehicle 20 as the target vehicle via the communication unit 12 (step S101). The sound control unit 113 of the control device 11 determines the ratio of external environmental sounds and virtual driving sounds included in the synthetic sound information according to the speed acquired in the processing of step S101 (step S102).

In the processing of step S102, the sound control unit 113 may determine the proportion of external environmental sounds and virtual driving sounds, for example, according to a map such as that shown in FIG. 5. Alternatively, when the speed of the target vehicle is lower than a threshold, the sound control unit 113 may increase the proportion of virtual driving sounds included in the synthetic sound information greater than the proportion of external environmental sounds. When the speed of the target vehicle is higher than a threshold, the sound control unit 113 may increase the proportion of external environmental sounds included in the synthetic sound information greater than the proportion of virtual driving sounds.

Then, the sound control unit 113 generates synthetic sound information from the external environmental sound and the virtual driving sound so that the ratio is determined in the processing of step S102. Then, the sound control unit 113 controls the speaker 17 so that the sound indicated by the generated synthetic sound information is emitted (step S103).

(Technical effect)
A user who remotely drives a target vehicle (here, the vehicle 20) using the remote driving device 10 cannot experience, for example, vibrations and accelerations caused by the behavior of the vehicle 20. In order to enhance the sense of realism of the user who is remotely driving, external environmental sounds are emitted, for example, from the speaker 17. However, when the speed of the vehicle 20 is relatively low, for example, vibration sounds and friction sounds of the tires of the vehicle 20 are relatively small. For this reason, even if external environmental sounds are emitted from the speaker 17, the effect of improving the sense of realism is not as great as expected.

Therefore, in the remote driving device 10, when the speed of the vehicle 20 is relatively low, synthetic sound information is generated that has a higher proportion of virtual running sounds compared to when the speed of the vehicle 20 is relatively high. Specifically, when the speed of the vehicle 20 is relatively low, synthetic sound information is generated that has a higher proportion of virtual running sounds than external environmental sounds. On the other hand, when the speed of the vehicle 20 is relatively high, synthetic sound information is generated that has a higher proportion of external environmental sounds than virtual running sounds. This is because, for example, the vibration sounds and friction sounds of the tires of the vehicle 20 are sufficiently loud when the speed of the vehicle 20 is relatively high.

The remote driving device 10 can make the speed perceived by the user remotely driving the target vehicle closer to the actual speed of the target vehicle.

In addition, among the functions of the sound control unit 113, the function of generating synthetic sound information from external environmental sounds and virtual driving sounds may be provided in the target vehicle (e.g., vehicle 20) to be remotely driven.

Second Embodiment
A second embodiment of the remote driving device will be described with reference to Fig. 7 and Fig. 8. The second embodiment is similar to the first embodiment described above, except for a part of the operation of the sound control unit 113. Therefore, the second embodiment will be described mainly with respect to the differences from the first embodiment, and descriptions that overlap with the first embodiment will be omitted as appropriate.

The sound control unit 113 (see FIG. 1) acquires external environmental sounds transmitted from the vehicle 20 (i.e., the target vehicle for remote driving) via the communication unit 12. The sound control unit 113 determines whether the acquired external environmental sounds include a siren sound emitted from an emergency vehicle. The sound control unit 113 may, for example, perform a predetermined frequency analysis on the external environmental sounds. Then, when the result of the frequency analysis includes a frequency specific to a siren sound emitted from an emergency vehicle, the sound control unit 113 may determine that the external environmental sounds include a siren sound emitted from an emergency vehicle.

When the external environmental sound includes a siren sound emitted by an emergency vehicle, the sound control unit 113 increases the proportion of the external environmental sound included in the synthetic sound information to be greater than the proportion of the virtual driving sound, regardless of the speed of the vehicle 20. Specifically, the sound control unit 113 may determine the proportion of the external environmental sound and the virtual driving sound according to a map such as that shown in FIG. 7.

As a result, a sound represented by the synthesized sound information containing a relatively large amount of external environmental sounds is emitted from the speaker 17. With this configuration, it is possible to make a user who is remotely driving the target vehicle, vehicle 20, using the remote driving device 10 aware of the presence of an emergency vehicle around the vehicle 20. If the user notices the presence of the emergency vehicle, he or she can, for example, move the vehicle 20 to the shoulder of the road.

If the external environmental sounds do not include siren sounds emitted by emergency vehicles, the sound control unit 113 may perform the same operation as in the first embodiment described above. In other words, the sound control unit 113 may determine the ratio of external environmental sounds and virtual driving sounds according to a map such as that shown in FIG. 5, for example.

The operation of the sound control unit 113 according to this embodiment will be described with reference to the flowchart in FIG. 8. Note that steps S101, S102, and S103 in FIG. 8 correspond to steps S101, S102, and S103 in FIG. 6.

In FIG. 8, after the processing of step S101, the sound control unit 113 determines whether or not an emergency vehicle is present around the vehicle 20 as the target vehicle for remote driving (step S201). Specifically, the sound control unit 113 determines whether or not the external environmental sound transmitted from the vehicle 20 includes a siren sound emitted from an emergency vehicle.

If it is determined in the processing of step S201 that an emergency vehicle is present around the vehicle 20 (step S201: Yes), the sound control unit 113 determines the proportion of external environmental sounds and virtual driving sounds included in the synthetic sound information according to the speed obtained in the processing of step S101 so that the proportion of external environmental sounds included in the synthetic sound information is greater than the proportion of virtual driving sounds (step S202).

Then, the sound control unit 113 generates synthetic sound information from the external environmental sound and the virtual driving sound so that the ratio is determined in the processing of step S202. Then, the sound control unit 113 controls the speaker 17 so that the sound indicated by the generated synthetic sound information is emitted (step S103).

If it is determined in the processing of step S201 that there is no emergency vehicle around the vehicle 20 (step S201: No), the sound control unit 113 performs the processing of step S102.

(Technical effect)
According to the remote driving device 10, the speed perceived by the user remotely driving the target vehicle can be made closer to the actual speed of the target vehicle, while allowing the user to perceive the presence of an emergency vehicle.

<Computer Program>
An embodiment of a computer program will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the configuration of a computer according to the embodiment.

In FIG. 9, computer 50 constitutes part of the remote operation device. Computer 50 is configured with a CPU (Central Processing Unit) 51, RAM 52, HDD (Hard Disk Drive) 53, and I/O 54. CPU 51, RAM 52, HDD 53, and I/O 54 are interconnected by bus 55. A computer program 531 according to this embodiment is pre-stored in HDD 53. The I/O 54 may constitute the communication unit 12 described above.

The processing of the CPU 51 by the computer program 531 will be described. The CPU 51 acquires the speed, acceleration, and yaw rate transmitted from the vehicle 20 via the I/O 54. The CPU 51 acquires images captured by the cameras 21L, 21C, and 21R via the I/O 54. The CPU 51 acquires external environmental sounds transmitted from the vehicle 20 via the I/O 54.

The CPU 51 generates a virtual running sound that simulates the running sound of the vehicle 20. The CPU 51 generates synthetic sound information that includes at least one of an external environmental sound and a virtual running sound. The CPU 51 controls a speaker (not shown) via the I/O 54 so that a sound indicated by the generated synthetic sound information is emitted.

Here, the CPU 51 changes the ratio of the external environmental sound and the virtual driving sound included in the synthetic sound information according to the speed of the vehicle 20. The CPU 51 may change the ratio of the external environmental sound and the virtual driving sound included in the synthetic sound information according to a map such as that shown in FIG. 5. The map such as that shown in FIG. 5 may be stored in the HDD 53.

The CPU 51 may determine whether the external environmental sound includes a siren sound emitted by an emergency vehicle (i.e., whether an emergency vehicle is present around the vehicle 20). If the external environmental sound includes a siren sound emitted by an emergency vehicle, the CPU 51 may increase the proportion of the external environmental sound included in the synthetic sound information to be greater than the proportion of the virtual driving sound, regardless of the speed of the vehicle 20. In this case, the CPU 51 may determine the proportion of the external environmental sound and the virtual driving sound according to a map such as that shown in FIG. 7. The map such as that shown in FIG. 7 may be stored in the HDD 53.

The computer program 531 may be stored in the HDD 53 by the computer 50 reading the computer program 531 from a recording medium, such as an optical disk, such as a CD-ROM (Compact Disc Read Only Memory), or a USB (Universal Serial Bus) memory, that stores the computer program 531. Alternatively, the computer 50 may download the computer program 531 via a network, such as the Internet, and thereby the computer program 531 may be stored in the HDD 53.

According to the computer program 531, similar to the remote driving device 10 in the first embodiment described above, the speed perceived by the user remotely driving the target vehicle can be made to approach the actual speed of the target vehicle. In addition, according to the computer program 531, similar to the remote driving device 10 in the second embodiment described above, the presence of an emergency vehicle can be made to be perceived by the user remotely driving the target vehicle. According to the computer program 531, the remote driving device 10 in the first and second embodiments described above can be realized relatively easily.

Various aspects of the invention derived from the embodiments described above are described below.

A remote driving device according to one aspect of the invention is a remote driving device that remotely drives a target vehicle, and includes a generating means for generating synthetic sound information including at least one of external environmental sound, which is sound around the target vehicle collected by a microphone, and a virtual running sound that simulates the running sound of the target vehicle, and the generating means increases the proportion of the external environmental sound included in the synthetic sound information when the speed of the target vehicle is high, compared to when the speed of the target vehicle is low. In the above-mentioned embodiment, the "sound control unit 113" corresponds to an example of the "generating means".

The remote driving device may include a detection means for detecting an emergency vehicle present around the target vehicle, and when an emergency vehicle is detected by the detection means, the generation means may increase the proportion of the external environmental sound contained in the synthetic sound information to be greater than the proportion of the virtual driving sound contained in the synthetic sound information. In the above-described embodiment, the "sound control unit 113" corresponds to an example of a "detection means".

A remote driving method according to one aspect of the invention is a remote driving method in a remote driving device that remotely drives a target vehicle, and includes a generation step of generating synthetic sound information including at least one of external environmental sound, which is sound around the target vehicle collected by a microphone, and a virtual driving sound that simulates the driving sound of the target vehicle, and in the generation step, when the speed of the target vehicle is high, the proportion of the external environmental sound included in the synthetic sound information is increased compared to when the speed of the target vehicle is low.

The remote driving method may include a detection step of detecting an emergency vehicle present around the target vehicle prior to the generation step, and in the generation step, if an emergency vehicle is detected by the detection step, the proportion of the external environmental sound included in the synthetic sound information may be made larger than the proportion of the virtual driving sound included in the synthetic sound information.

A computer program according to one aspect of the invention causes a computer of a remote driving device that remotely drives a target vehicle to function as a generating means that generates synthetic sound information including at least one of external environmental sound, which is sound around the target vehicle collected by a microphone, and virtual driving sound, which simulates the driving sound of the target vehicle, and that increases the proportion of the external environmental sound included in the synthetic sound information when the speed of the target vehicle is high compared to when the speed of the target vehicle is low.

The computer program may further cause the computer to function as a detection means for detecting an emergency vehicle present around the target vehicle, and when an emergency vehicle is detected by the detection means, the generation means may increase the proportion of the external environmental sound contained in the synthetic sound information to be greater than the proportion of the virtual driving sound contained in the synthetic sound information.

The present invention is not limited to the above-described embodiment, but may be modified as appropriate within the scope of the claims and the entire specification without violating the spirit or concept of the invention, and remote control devices with such modifications are also included in the technical scope of the present invention.

1...Remote driving system, 10...Remote driving device, 11...Control device, 12, 26...Communication unit, 13...Steering sensor, 14...Accelerator pedal sensor, 15...Brake pedal sensor, 16...Display, 17...Speaker, 20...Vehicle, 21L, 21C, 21R...Camera, 22...Microphone, 23...Speed sensor, 24...Acceleration sensor, 25...Yaw rate sensor, 111...Vehicle control unit, 112...Display control unit, 113...Sound control unit

Claims (2)

A remote driving device for remotely driving a target vehicle,
A generating unit generates synthetic sound information including an external environmental sound, which is a sound around the target vehicle collected by a microphone, and a virtual running sound, which simulates a running sound of the target vehicle,
The generating means increases a proportion of the external environmental sound included in the synthetic sound information when the speed of the target vehicle is high compared to when the speed of the target vehicle is low. a detection means for detecting an emergency vehicle existing around the target vehicle;
The remote driving device according to claim 1, characterized in that, when an emergency vehicle is detected by the detection means, the generation means increases the proportion of the external environmental sound contained in the synthetic sound information to be greater than the proportion of the virtual driving sound contained in the synthetic sound information.

Images