JP7615960B2 - Remote Control Device - Google Patents

Description

The present invention relates to a device for remotely driving a vehicle.

JP 2004-295360 A discloses a device for remotely driving a vehicle. This conventional device is installed in a control center that communicates with the vehicle. This conventional device also includes a control device, a display, a driver's seat, various sensors, etc. Images from a camera in the vehicle are output to the display. The driver's seat has a configuration similar to that of a driver's seat in an actual car. A remote operator sitting in the driver's seat operates input devices such as the steering wheel, accelerator pedal, and brake pedal while looking at the image from the vehicle-mounted camera output to the display. Various sensors detect the amount of operation of the input devices by the remote operator. The control device calculates the control amount of the vehicle's running devices such as the steering device, drive device, and braking device based on this amount of operation.

JP 2004-295360 A

During remote driving, the remote operator may need to wait for the vehicle to start. For example, while the nearest traffic light is showing a red light, the remote operator stops the vehicle and waits for it to start. Another example would be while waiting for a train to pass at a railroad crossing. Another example would be when the vehicle is stuck in traffic. Therefore, it is desirable to develop technology that can provide the remote operator with useful information during such waiting times.

One objective of the present invention is to provide a technology that provides advertising information to a remote operator while the vehicle is waiting to start while being remotely driven.

The present invention is a remote driving device for remotely driving a vehicle, and has the following features.
The remote operation device is
A data processing device including a memory in which various data are stored and a processor for processing the various data;
A communication device for communicating with the vehicle;
a display device that provides a remote operator with display image data generated based on the various data;
Equipped with.
The various data include surrounding image data of the vehicle, driving environment data of the vehicle, and advertisement image data.
The processor determines whether a condition for temporarily stopping the vehicle is met based on the driving environment data during remote driving of the vehicle by the remote operator.
When it is determined that the temporary stop condition is not satisfied, the processor outputs the display image data generated using the surrounding image data to the display device.
When the processor determines that the pause condition is satisfied, it outputs an advertising start notification to the display device, and after outputting the advertising start notification, outputs the display image data generated using the surrounding image data and the advertising image data to the display device.
The processor determines whether or not an advertisement end condition is satisfied when the display image data generated using the surrounding image data and the advertisement image data is output to the display device.
When the processor determines that the advertisement end condition is satisfied, it outputs an advertisement end notification to the display device, and after outputting the advertisement end notification, outputs the display image data generated using the surrounding image data to the display device.
The various data further includes interior image data of the vehicle.
The advertising image data is created in accordance with the attributes of customers targeted by the advertising provider.
The subjects of transportation by remote driving of the vehicle include the family and relatives of the remote operator.
The processor estimates attributes of the object to be transported by analyzing the indoor image data, and narrows down the advertising image data to be output to the display device based on the attributes of the object to be transported.

According to the present invention, when it is determined that the vehicle is in a temporary stop state, display image data generated using surrounding image data and advertising image data is output to the display device. Therefore, it is possible to provide advertising information to the remote operator while the vehicle is waiting to start. Advertising information that matches the attributes, tastes, etc. of the remote operator can be useful information for the remote operator. When transporting a specific object by remote driving, advertising information that matches the attributes of the object to be transported, the destination of the transport, etc. can be useful information for the remote operator.

1 is a diagram illustrating an application example of remote driving by a remote driving device according to an embodiment of the present invention. FIG. FIG. 1 is a diagram illustrating features of an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a vehicle. FIG. 2 is a diagram illustrating an example of the configuration of a remote driving device. 5 is a flowchart showing an example of remote driving processing by a data processing device of the remote driving device.

Below, a remote operation device according to an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same or corresponding parts are given the same reference numerals, and their description will be simplified or omitted.

1. Application Example FIG. 1 is a diagram for explaining an application example of remote driving by a remote driving device according to an embodiment. In the example shown in FIG. 1, the remote driving device 1 is installed at a first point P1 (specifically, the home HM of a remote operator OP). However, the remote driving device 1 does not have to be installed at the first point P1. That is, the remote driving device 1 may be installed at a point different from the first point P1. The remote driving device 1 communicates with a vehicle 2. The communication between the remote driving device 1 and the vehicle 2 is performed via a base station 3.

In the example shown in FIG. 1, a vehicle 2 is remotely driven by a remote driving device 1. In this remote driving, the vehicle 2 travels back and forth between a first point P1 (specifically, a parking lot at the home HM) and a second point P2 (specifically, a parking lot at a facility FC such as a school, cram school, hospital, etc.). The vehicle 2 travels back and forth to pick up or drop off a transport target OB at the second point P2. Examples of the transport target OB include the family or relatives of the remote operator OP.

To remotely drive the vehicle 2, the remote operator OP starts the remote driving device 1. At the time of this start-up, the remote driving device 1 authenticates the remote operator OP. If the authentication conditions are met, the remote operator OP is able to remotely drive the vehicle 2. During remote driving, the remote operator OP operates input devices such as the steering wheel, accelerator pedal, and brake pedal while looking at the display device 11 of the remote driving device 1.

During remote driving, surrounding image data of the vehicle 2 is transmitted from the vehicle 2 to the remote driving device 1. The remote driving device 1 uses this surrounding image data to generate display image data, which is output to the display device 11. Examples of surrounding images include a front center image, a left front image, a right front image, a left side image, a right side image, and a rear image of the vehicle 2. The display image is a composite of these images, etc. The display image is typically a moving image, but may also be a still image.

2. Features of the embodiment Fig. 2 is a diagram for explaining features of the embodiment. Fig. 2 shows three display devices 11a to 11c as an example of the display device 11. Display image data DIMa is displayed on the display device 11a. Display image data DIMb is displayed on the display device 11b, and display image data DIMc is displayed on the display device 11c. In the example shown in Fig. 2(I), the display image data DIMa is data of a center front image ICF, the display image data DIMb is data of a left front image ILF, and the display image data DIMc is data of a right front image IRF.

The left and right side image data and the rear image data are output to a display device other than the display devices 11a to 11c. However, these data may be combined with any image data output to the display devices 11a to 11c and output to any of the display devices 11a to 11c. For example, the rear image data may be combined with the front center image ICF data and output to the display device 11a. The left side image data may be combined with the left front image ILF data and output to the display device 11b. The right side image data may be combined with the right front image IRF data and output to the display device 11c.

During remote driving, driving environment data is transmitted from vehicle 2 to remote driving device 1 in addition to surrounding image data. The driving environment data is data related to the driving environment of vehicle 2. The driving environment data includes data related to the driving conditions of vehicle 2, such as vehicle speed data, acceleration data, and steering angle data. The driving environment data also includes data related to objects around vehicle 2 that are recognized by vehicle 2. The driving environment data further includes position data of vehicle 2. The driving environment data further includes VICS (Vehicle Information and Communication System, registered trademark) data such as traffic congestion information and accident information.

The remote driving device 1 judges whether the vehicle 2 is in a temporary stop state based on the driving environment data. For example, if the vehicle speed is below a predetermined value and the amount of accelerator pedal operation by the remote operator OP is below a predetermined value, it is judged to be in a temporary stop state. In another example, if the vehicle speed is below a predetermined value and a red light is recognized in front of the vehicle 2, it is judged to be in a temporary stop state. In yet another example, if the vehicle speed is below a predetermined value and an operating railroad crossing is recognized in front of the vehicle 2, it is judged to be in a temporary stop state. In yet another example, if the vehicle speed is below a predetermined value and it is estimated from the position data and traffic information of the vehicle 2 that the vehicle 2 is caught in a traffic jam, it is judged to be in a temporary stop state.

When it is determined that the vehicle 2 is in a temporarily stopped state, the remote driving device 1 generates display image data by combining the surrounding image data with the advertising image data. Figure 2 (II) shows examples of display image data DIMa to DIMc that are generated after it is determined that the vehicle 2 is in a temporarily stopped state. In the example shown in Figure 2 (II), the display image data DIMa is data of the front center image ICF, and the display image data DIMc is data of the right front image IRF. Up to this point, it is the same as the example shown in Figure 2 (I). On the other hand, the display image data DIMb is data that combines the left front image ILF and the advertising image IAD.

In FIG. 2 (II), the advertising image IAD is superimposed on a partial area of the left front image ILF. However, the advertising image IAD may be superimposed on the entire area of the left front image ILF. The advertising image IAD may be combined with the front center image ICF and displayed on the display device 11a, or may be combined with the right front image IRF and displayed on the display device 11b. The advertising image IAD may be combined with the left front image ILF and at least one of the front center image ICF and the right front image IRF. In this case, the display image data DIM may be displayed simultaneously on the display device 11c and at least one of the display devices 11a and 11b.

While the remote driving device 1 is outputting display image data generated using advertising image data from the display device 11, the remote driving device 1 determines whether the state of the vehicle 2 is ready to start based on the driving environment data. For example, if a green light is recognized ahead of the vehicle 2, it is determined that the vehicle is ready to start. In another example, if the departure of a vehicle preceding the vehicle 2 is recognized, it is determined that the vehicle 2 is ready to start. In yet another example, if the end of operation of the railroad crossing ahead of the vehicle 2 is recognized, it is determined that the vehicle is ready to start.

When it is determined that the state of the vehicle 2 is ready to start, the remote driving device 1 ends the generation of display image data using the advertising image data, and resumes the generation of display image data using the surrounding image data. Figure 2 (III) shows an example of display image data DIMa to DIMc that was generated after it was determined that the state of the vehicle 2 is ready to start. In the example shown in Figure 2 (III), data similar to the display image data DIMa to DIMc shown in Figure 2 (I) is output to the display devices 11a to 11c, respectively.

Thus, according to the features of the embodiment, when it is determined that the state of the vehicle 2 is in a temporary stop state, the display image data DIM generated using the surrounding image data and the advertising image data is output to the display device 11. Therefore, it is possible to provide advertising information to the remote operator OP during the waiting time for the vehicle 2 to depart.

Below, we will explain an example of the configuration of the remote driving device 1 to realize the features of the above-mentioned embodiment.

3. Remote Driving Device 3-1. Example of the Configuration of a Remotely Driven Vehicle Fig. 3 is a block diagram showing an example of the configuration of the vehicle 2 shown in Fig. 1. The example of the configuration shown in Fig. 3 includes a camera 21, a microphone 22, a group of sensors 23, a map database (DB) 24, a communication device 25, a driving device 26, and a data processing device 27.

The cameras 21 are provided in multiple units inside and outside the vehicle 2. The cameras 21 include an exterior camera that captures images of the surroundings of the vehicle 2. Examples of surrounding images captured by the exterior cameras include a front center image, a right front image, a left front image, a right side image, a left side image, and a rear image. The exterior cameras transmit these image data to the data processing device 27 as "surrounding image data IMG". The cameras 21 also include an interior camera that captures images of the interior of the vehicle 2. The interior camera transmits data of interior images to the data processing device 27.

Multiple microphones 22 are provided on the exterior sides of the vehicle 2. Each microphone 22 has directionality and records sounds surrounding the vehicle 2. Examples of ambient sounds include right front sound, left front sound, right rear sound, and left rear sound. Each microphone 22 transmits the recorded sound data to the data processing device 27 as "ambient sound data SUD."

The sensor group 23 includes a status sensor that detects the status of the vehicle 2. Examples of the status sensor include a vehicle speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. The sensor group 23 also includes a position sensor that acquires position data of the vehicle 2. Examples of the position sensor include a Global Navigation Satellite System (GNSS) sensor. The status sensor transmits the status data of the vehicle 2 to the data processing device 27, and the position sensor transmits the position data to the data processing device 27. The sensor group 23 further includes a recognition sensor other than the camera 21. The recognition sensor recognizes (detects) targets around the vehicle 2 using radio waves or light. Examples of the recognition sensor include a millimeter wave radar and a LIDAR (Laser Imaging Detection and Ranging). The recognition sensor transmits recognition data of the target to the data processing device 27.

The status data of vehicle 2, the position data of vehicle 2, and the recognition data of surrounding objects recognized by camera 21 or a recognition sensor other than camera 21 are included in the "driving environment data ENV".

Map data is stored in the map DB 24. Examples of map data include data on road positions, data on road shapes (e.g., types of curves and straight lines), and data on the positions of intersections and structures. The map DB 24 is formed in an on-board storage device (e.g., a non-volatile recording medium such as a hard disk or flash memory). The map DB 24 may also be formed in a computer (e.g., an external server) capable of communicating with the vehicle 2.

The communication device 25 performs wireless communication with the base station 3. Examples of communication standards for this wireless communication include mobile communication standards such as 4G, LTE, and 5G. The connection destinations of the communication device 25 include at least the remote driving device 1. In communication with the remote driving device 1, the communication device 25 transmits various data received from the data processing device 27 to the remote driving device 1.

The traveling device 26 accelerates, decelerates, and steers the vehicle 2. The traveling device 26 includes, for example, a drive device, a steering device, and a brake device. The drive device drives the tires of the vehicle 2. The steering device steers the tires of the vehicle 2. The brake device applies a braking force to the vehicle 2. The acceleration of the vehicle 2 is performed by controlling the drive device. The deceleration of the vehicle 2 is performed by controlling the brake device. If the drive device is a motor, the braking of the vehicle 2 may be performed using regenerative braking controlled by the motor. The steering of the vehicle 2 is performed by controlling the steering device.

The data processing device 27 is a computer for processing various data acquired by the vehicle 2. The data processing device 27 includes at least one processor 27a and at least one memory 27b. The processor 27a includes a CPU (Central Processing Unit). The memory 27b is a volatile memory such as a DDR memory, and expands various programs used by the processor 27a and temporarily stores various data. Various data acquired by the vehicle 2 is stored in the memory 27b. This various data includes surrounding image data IMG, surrounding sound data SUD, and driving environment data ENV. When the vehicle 2 is remotely driven, this various data also includes control data for the driving device 26 received from the remote driving device 1.

The processor 27a executes programs stored in the memory 27b to perform various controls related to the driving of the vehicle 2. When the vehicle 2 is driven remotely, the processor 27a also encodes various data stored in the memory 27b and outputs the encoded data to the communication device 25. The various data include ambient image data IMG, ambient sound data SUD, and driving environment data ENV. These data may be compressed during the encoding process. The processor 27a further controls the driving device 26 based on the control data stored in the memory 27b.

3-2. Example of the configuration of the remote driving device Fig. 4 is a block diagram showing an example of the configuration of the remote driving device 1 shown in Fig. 1. The example of the configuration shown in Fig. 4 includes a display device 11, a sound reproduction device 12, an input device 13, a map database (DB) 14, an advertisement database (DB) 15, a communication device 16, and a data processing device 17.

The display device 11 outputs display image data DIM generated based on the surrounding image data IMG. Examples of the display device 11 include a liquid crystal display (LCD) and an organic light emitting diode (OLED) display. The display device 11 includes, for example, three display devices 11 that respectively display data of a front center image, a left front image, and a right front image of the vehicle 2. The display device 11 may be wearable glasses. The wearable glasses are glasses-type or goggle-type terminals worn by the remote operator OP. When the display device 11 is wearable glasses, the front center image, the left front image, and the right front image are synthesized and displayed on the wearable glasses as a single image.

The sound reproducing device 12 reproduces the reproduced sound data PSU generated based on the ambient sound data SUD. The ambient sound data SUD is the recorded sound data in each directional direction of the microphone 22. The reproduced sound data PSU may be data after processing such as wave field synthesis. The reproduced sound data PSU may be data in which sounds other than the vehicle sound and the environmental sound recognized by the analysis of the ambient sound data SUD are cut. The vehicle sound to be recognized is a sound generated with steering, acceleration, or deceleration. Examples of sounds generated with steering include the sound of the tires turning and the friction sound between the tires and the road surface. Examples of sounds generated with acceleration include the rotation sound of the drive motor. Examples of sounds generated with deceleration include the rotation sound of the drive motor and the friction sound between the tires and the road surface. Examples of environmental sounds to be recognized include warning sounds issued by railroad crossings and traffic signals, warning sounds issued by emergency vehicles, and warning sounds issued by vehicles around the vehicle 2.

The input device 13 is a device operated by the remote operator OP. The input device 13 includes driving input devices for controlling the driving of the vehicle 2, such as an accelerator pedal, a brake pedal, and a steering wheel. The input device 13 also includes auxiliary input devices for assisting the driving of the vehicle 2, such as a turn signal switch, a light switch, and a wiper switch. The input device 13 detects the amount of operation of the driving input devices by the remote operator OP, or the operation status of the auxiliary input devices, and transmits it to the data processing device 27.

Map data is stored in map DB 14. Examples of map data include the same map data as the example of map data stored in map DB 24 described in FIG. 3.

Advertising DB15 stores advertising image IAD data and advertising sound SAD data. The advertising image IAD and advertising sound SAD are stored for each piece of advertising information. The advertising information is created according to the attributes (age group, gender, etc.) of customers targeted by the advertising provider. The advertising image IAD and advertising sound SAD are also read out for each piece of advertising information. Data constituting any piece of advertising information is read out from advertising DB15.

Here, when the remote operator OP performs remote driving, authentication of the remote operator OP may be necessary. In this case, identification data of the remote operator OP is separately input to the data processing device 17. In the case of the application example described in FIG. 1, current position data of the remote operator OP and the vehicle 2 and destination data of the vehicle 2 are identified. Furthermore, when data of the interior image of the vehicle 2 is analyzed, attributes of the object OB to be transported by the vehicle 2 can also be estimated. When reading data from the advertising DB 15, the read data may be narrowed down based on these linking data.

The communication device 16 performs wireless communication with the base station 3. Examples of communication standards for this wireless communication include mobile communication standards such as 4G, LTE, and 5G. The connection destinations of the communication device 16 include at least the vehicle 2. In communication with the vehicle 2, the communication device 16 transmits various data (e.g., data for controlling the driving device 26) received from the data processing device 17 to the vehicle 2. The connection destinations of the communication device 16 may include a VICS (registered trademark) data server. In this case, the communication device 16 obtains VICS (registered trademark) data around the current location of the vehicle 2 from this server.

The data processing device 17 is a computer that processes various data related to the remote driving of the vehicle 2. The data processing device 17 includes at least one processor 17a and at least one memory 17b. The basic configuration of the data processing device 17 is the same as that of the data processing device 27 described in FIG. 3. The memory 17b stores various programs and various data used by the processor 17a. This various data includes ambient image data IMG, ambient sound data SUD, and driving environment data ENV.

When remote driving the vehicle 2, the processor 17a decodes various data received from the vehicle 2, generates display image data DIS based on the ambient image data IMG, and outputs it to the display device 11. The processor 17a also generates playback sound data PSU based on the ambient sound data SUD, and outputs it to the sound playback device 12. If the various data received from the vehicle 2 is compressed, this information is expanded in the decoding process.

The processor 17a further generates control data for the vehicle 2 based on the amount of operation of the driving input device received from the input device 13, and outputs the control data to the communication device 16. The processor 17a further generates operation data for various driving assistance devices of the vehicle 2 based on the operation status of the auxiliary input device received from the input device 13, and outputs the operation data to the communication device 16.

3-3. Example of processing by the data processing device Fig. 5 is a flowchart showing an example of remote driving processing by the data processing device 17 (processor 17a). The routine shown in Fig. 5 is repeatedly executed at a predetermined period after the remote operator OP starts remote driving the vehicle 2.

5, first, it is determined whether or not the temporary stop condition is satisfied (step S11). In the process of step S11, for example, it is determined whether or not any one of the temporary stop conditions C1 to C4 is satisfied based on the traveling environment data ENV.
(C1) The vehicle speed is below a predetermined value, and the amount of accelerator pedal operation by the remote operator OP is below a predetermined value. (C2) The vehicle speed is below a predetermined value, and a red light is recognized in front of the vehicle 2. (C3) The vehicle speed is below a predetermined value, and an operating railroad crossing is recognized in front of the vehicle 2. (C4) The vehicle speed is below a predetermined value, and it is estimated that the vehicle 2 is caught in a traffic jam based on the position data and traffic information of the vehicle 2.

If the determination result of step S11 is negative, the process of step S16 (described below) is performed. If the determination result of step S11 is positive, an advertisement start notification is performed (step S12). In the process of step S12, for example, icon image data of "Advertisement starting" is combined with the surrounding image data IMG and output to the display device 11. In another example, audio data of "Advertisement starting" is output to the sound playback device 12.

Following the processing of step S12, an advertising image IAD and an advertising sound SAD are provided (step S13). In the processing of step S13, for example, data is narrowed down based on attributes of the remote operator OP or the target OB. Next, the advertising image IAD data read from the advertising DB 15 is synthesized with predetermined surrounding image data IMG. Then, the display image data DIS, which is the image data after the synthesis, is output to the display device 11, which outputs the predetermined surrounding image data IMG. In the processing of step S13, the advertising sound SAD data read from the advertising DB 15 is also output to the sound playback device 12.

Following the process of step S13, it is determined whether or not the advertisement termination condition is satisfied (step S14). In the process of step S14, for example, it is determined whether or not any one of the advertisement termination conditions C5 to C8 is satisfied based on the driving environment data ENV (recognition data).
(C5) A green light is recognized ahead of vehicle 2. (C6) The departure of a vehicle ahead of vehicle 2 is recognized. (C7) The end of operation of the railroad crossing ahead of vehicle 2 is recognized. (C8) An operation for starting vehicle 2 by remote operator OP (e.g., depressing the accelerator pedal) is detected.

The process of step S14 is repeated until a positive determination result is obtained. If the determination result of step S14 is positive, an advertisement end notification is issued (step S15). The advertisement end notification is, for example, output to the display device 11 by combining icon image data of "Advertisement will end soon. Please resume driving" with the surrounding image data IMG. In another example, audio data of "Advertisement will end soon. Please resume driving" is output to the sound playback device 12. In yet another example, a signal that vibrates the steering wheel, the driver's seat, etc. is output.

In the processing of step S16, display image data DIM is generated using the surrounding image data IMG, and is output to the display device 11 that outputs this surrounding image data IMG. In the processing of step S16, playback sound data PSU is also generated using the surrounding sound data SUD, and is output to the sound playback device 12. When generating the playback sound data PSU, processing such as wave field synthesis of the surrounding sound data SUD may be performed, and sounds other than vehicle sounds and environmental sounds recognized by analysis of the surrounding sound data SUD may be cut.

REFERENCE SIGNS LIST 1 Remote driving device 2 Vehicle 11, 11a, 11b, 11c Display device 15 Advertising database 16, 25 Communication device 17, 27 Data processing device 17a, 27a Processor 17b, 27b Memory FC Facility HM Home OB Transport object OP Remote operator DIM, DIMa, DIMb, DIMc Display image data ENV Driving environment data IAD Advertising image ICF Front center image ILF Left front image IRF Right front image IMG Surrounding image data PSU Playback sound SUD Surrounding sound data

Claims (1)

A remote driving device for remotely driving a vehicle,
A data processing device including a memory in which various data are stored and a processor for processing the various data;
A communication device for communicating with the vehicle;
a display device that provides a remote operator with display image data generated based on the various data;
Equipped with
The various data include surrounding image data of the vehicle, driving environment data of the vehicle, and advertisement image data,
The processor,
determining whether a temporary stop condition for the vehicle is satisfied based on the driving environment data during remote driving of the vehicle by the remote operator;
When it is determined that the temporary stop condition is not satisfied, the display image data generated using the surrounding image data is output to the display device;
When it is determined that the pause condition is satisfied, an advertisement start notification is output to the display device;
After outputting the advertisement start notification, outputting the display image data generated by using the surrounding image data and the advertisement image data to the display device;
The processor further comprises:
determining whether or not an advertisement end condition is satisfied when the display image data generated using the surrounding image data and the advertisement image data is output to the display device;
When it is determined that the advertisement termination condition is satisfied, an advertisement termination notice is output to the display device;
After outputting the advertisement end notification, outputting the display image data generated using the surrounding image data to the display device ;
The various data further includes interior image data of the vehicle,

The advertising image data is created in accordance with the attributes of customers targeted by the advertising provider,
The transportation targets by remote driving of the vehicle include the family and relatives of the remote operator,
The processor further comprises:
estimating attributes of the transportation object by analyzing the indoor image data;
The advertisement image data to be output to the display device is narrowed down based on the attributes of the transportation object.
A remote driving device characterized by:

Images