JP7794119B2 - Communication device and remote instruction system - Google Patents

Description

The present invention relates to a communication device and a remote instruction system.

A remote video output device is known that, when the communication delay from the vehicle via the network is a second delay time that is longer than a first delay time, extracts a second range from the video that is narrower than the first range and outputs it (see, for example, Patent Document 1).

International Publication No. 2018/155159

In the technical field of remote vehicle support, efforts are being made to reduce the amount of video data transmitted from vehicles over communication networks. However, the video data referenced by the remote commander must have a quality that allows the remote commander to visually recognize the information needed to issue remote instructions, so there is room for improvement in reducing the amount of data transmitted.

One aspect of the present invention is a communications device mounted on a vehicle capable of remote support in response to remote instructions from a remote commander, which transmits video data from the vehicle via a communications network. The communications device includes a transmission condition setting unit that sets transmission conditions, including a target value for the transmission frame rate of the video data and a target value for the transmission resolution of the video data, based on the vehicle speed, and a transmission unit that transmits the video data based on the transmission conditions so that the transmission resolution and transmission frame rate reach their respective target values. The transmission condition setting unit determines whether to suppress the transmission image quality of the video data, including the transmission frame rate and transmission resolution, based on the vehicle speed and communication quality information of the communications network. If it is determined that the transmission image quality should be suppressed, the transmission image quality is suppressed by maintaining the target value for the transmission frame rate and reducing the target value for the transmission resolution.

In a communication device according to one aspect of the present invention, a determination is made as to whether to suppress the transmission image quality of video data, including the transmission frame rate and transmission resolution, based on the vehicle speed and communication quality information of the communication network. If it is determined that the transmission image quality should be suppressed, the target value of the transmission frame rate is maintained and the target value of the transmission resolution is reduced. This reduces the amount of transmission data without lowering the transmission frame rate of the video data, while taking into account both the vehicle speed and the communication quality information of the communication network. This prevents, for example, the transmission frame rate from being insufficient compared to the vehicle speed. This makes it possible to achieve both good visibility of video data by a remote commander and a reduction in the amount of transmission data.

In one embodiment, the transmission condition setting unit may reduce or increase the target value of the transmission resolution so that the absolute value of the time rate of change of the target value of the transmission resolution does not exceed a predetermined upper limit rate of change. In this case, it is possible to reduce the impact on the visibility of video data by the remote commander caused by, for example, a sudden change in the transmission resolution.

In one embodiment, when the transmission condition setting unit determines that the transmission image quality should be reduced when the target value of the transmission resolution has reached a predetermined minimum transmission resolution, it may maintain the target value of the transmission resolution while reducing the target value of the transmission frame rate. In this case, the amount of transmission data can be reduced so as not to cause the transmission resolution to fall below the minimum transmission resolution.

Another aspect of the present invention is a remote instruction system including the above-mentioned communication device and a remote instruction device that receives video data transmitted from the above-mentioned communication device via a communication network. The remote instruction device includes an information receiving unit that acquires the reception resolution of the video data and an information transmitting unit that transmits the reception resolution to the communication device via the communication network. When the reception resolution is lower than the target transmission resolution of the video data, the transmission condition setting unit maintains the target transmission frame rate and sets the target transmission resolution based on the reception resolution.

In a remote instruction system according to another aspect of the present invention, when the receiving resolution is lower than the target value of the transmission resolution of the video data, the target value of the transmission frame rate is maintained, and the target value of the transmission resolution is set based on the receiving resolution. This makes it possible to efficiently reduce the amount of data transmitted by utilizing information about the receiving resolution.

This invention makes it possible to achieve both high visibility of video data by a remote commander and a reduction in the amount of data transmitted.

1 is a diagram illustrating an example of an overall image of a remote instruction system according to an embodiment; 1 is a block diagram illustrating an example of a configuration of a vehicle including a communication device according to an embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of a remote instruction server. 2 is a block diagram showing an example of a configuration of a remote instruction device and a functional configuration of a remote instruction server. FIG. 10A is a diagram illustrating an example of the relationship between a vehicle speed and a target value of transmission resolution, and FIG. 10B is a diagram illustrating an example of the relationship between a vehicle speed and a target value of transmission frame rate. FIG. 10 is a sequence diagram illustrating an example of processing by the remote instruction system. 10 is a flowchart illustrating an example of a transmission condition setting process.

Embodiments of the present invention will be described below with reference to the drawings. In the following description, identical or corresponding elements will be designated by the same reference numerals, and duplicate explanations will be omitted.

[Remote instruction system overview]
FIG. 1 is a diagram illustrating an example of an overall view of a remote instruction system according to an embodiment. As shown in FIG. 1, the remote instruction system 100 includes a remote instruction device 1 to which a remote commander R inputs remote instructions, a vehicle 2, and a communication device 4. A remote instruction server 10 of the remote instruction device 1 is communicably connected to a plurality of vehicles 2 via a communication network N, for example. The communication network N is a wireless communication network. For example, the communication network N includes a network (communication network) operated by one or more communication carriers. Various types of information are sent to the remote instruction device 1 from the communication device 4 of the vehicle 2.

The remote instruction system 100 performs remote support for the vehicle 2 in response to remote instructions from the remote commander R. A remote instruction is an instruction from the remote commander R regarding remote support for the vehicle 2. The remote commander R is an operator who provides remote support for the vehicle 2. The remote commander R is deployed, for example, in a remote cockpit located in a facility or the like away from the vehicle 2. The number of remote commanders R may be one or two or more.

Vehicle 2 is a vehicle configured to be able to perform remote support. Vehicle 2 may also be configured to be able to perform autonomous driving control. There is no particular limit to the number of vehicles 2 that can communicate with remote instruction system 100.

Remote support includes remote monitoring, in which the remote commander R monitors the situation around the vehicle and the driver's condition; remote assistance, in which the remote commander R provides information and instructions to the vehicle driver; and remote driving, in which the remote commander R gives instructions regarding the autonomous driving of the vehicle. For example, instructions regarding autonomous driving include instructions for vehicle 2 to proceed and instructions for vehicle 2 to stop. Instructions regarding autonomous driving may also include instructions for vehicle 2 to change lanes. Instructions regarding autonomous driving may also include instructions to offset and avoid obstacles ahead, instructions to overtake a leading vehicle, instructions to make an emergency evacuation, etc.

In the remote instruction system 100, video data of a surrounding environment image is obtained from detection data of the surrounding environment detected, for example, by an external sensor 22 of the vehicle 2. The surrounding environment image is a captured image of the surrounding environment of the vehicle 2 detected by the external sensor 22. The video data of the surrounding environment image is transmitted from the vehicle 2 via the communication network N to the remote instruction server 10 of the remote instruction device 1 by the communication device 4 of the vehicle 2. The remote instruction server 10 displays an image representing the surrounding environment image to the remote commander R. The remote commander R inputs remote instructions to the commander interface 3 of the remote instruction device 1 while referring to the surrounding environment image. The remote instruction device 1 transmits the remote instructions to the vehicle 2 via the communication network N. The vehicle 2 performs remote support in accordance with the remote instructions.

[Vehicle configuration]
First, an example of the configuration of the vehicle 2 will be described. FIG. 2 is a block diagram showing an example of the configuration of a vehicle including a communication device according to the embodiment. As shown in FIG. 2, the vehicle 2 includes, as an example, a remote driving ECU 30. The remote driving ECU 30 is an electronic control unit including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The remote driving ECU 30 realizes various functions, for example, by loading a program recorded in the ROM into the RAM and executing the program loaded into the RAM by the CPU. The remote driving ECU 30 may be composed of multiple electronic units. The remote driving ECU 30 is also referred to as, for example, an RDK (Remote Driving Kit).

The remote driving ECU 30 is connected to a GPS (Global Positioning System) receiver 21, an external sensor 22, an internal sensor 23, a map database 24, an actuator 25, and a communication unit 26.

The GPS receiver unit 21 measures the position of the vehicle 2 (e.g., the latitude and longitude of the vehicle 2) by receiving signals from three or more GPS satellites. The GPS receiver unit 21 transmits the measured position information of the vehicle 2 to the remote driving ECU 30.

The external sensor 22 is an on-board sensor that detects the surrounding environment of the vehicle 2. The external sensor 22 transmits the detected data to the remote driving ECU 30.

The external sensor 22 includes at least a camera. The camera is an imaging device that captures images of the environment surrounding the vehicle 2. The camera is installed, for example, behind the windshield of the vehicle 2 and captures images in front of the vehicle. The camera may also capture images of the sides and rear of the vehicle 2.

The external sensor 22 may include a radar sensor. A radar sensor is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle 2. Radar sensors include, for example, radar (millimeter wave radar) or LiDAR (Light Detection and Ranging).

The internal sensor 23 is an on-board sensor that detects the driving state of the vehicle 2. The internal sensor 23 may include a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. Known sensors can be used as the vehicle speed sensor, acceleration sensor, and yaw rate sensor.

The map database 24 is a database that records map information. The map database 24 is formed, for example, in a recording device such as a hard disk drive (HDD) mounted on the vehicle 2. The map information includes road position information, road shape information (e.g., curvature information), intersection and branch point position information, etc.

The actuators 25 are devices used to control the driving of the vehicle 2 and operate in response to control signals from the remote driving ECU 30. The actuators 25 include at least a drive actuator, a brake actuator, and a steering actuator. The drive actuator is provided, for example, in an engine or motor serving as a power source, and controls the driving force of the vehicle 2. The brake actuator is provided, for example, in a hydraulic brake system, and controls the braking force applied to the wheels of the vehicle 2. The steering actuator is, for example, an assist motor in an electric power steering system, and controls the steering torque of the vehicle 2.

The communication unit 26 is a communication device that controls wireless communication with the outside of the vehicle 2. The communication unit 26 transmits and receives various information to and from the remote instruction device 1, for example, via a communication carrier's communication network N.

Next, an example of the functional configuration of the remote driving ECU 30 will be described. The remote driving ECU 30 has a vehicle position acquisition unit 31, a surrounding environment recognition unit 32, a driving condition recognition unit 33, a route generation unit 34, a remote driving control unit 35, a display information acquisition unit 36, a transmission condition setting unit 37, and a display information transmission unit (transmission unit) 38.

The vehicle position acquisition unit 31 acquires the position information (position on the map) of the vehicle 2 based on the position information from the GPS receiver unit 21 and the map information from the map database 24, or by using SLAM (Simultaneous Localization and Mapping) technology.

The surrounding environment recognition unit 32 recognizes the surrounding environment of the vehicle 2 based on detection data from the external sensor 22. The surrounding environment may include information used for autonomous driving, such as the relative position, relative speed, and movement direction of surrounding objects relative to the vehicle 2.

The driving condition recognition unit 33 recognizes the driving condition of the vehicle 2 based on the detection results of the internal sensor 23. The driving condition includes the vehicle speed, acceleration, and yaw rate (orientation) of the vehicle 2.

The trajectory generation unit 34 generates a trajectory to be used for autonomous driving of the vehicle 2. The trajectory generation unit 34 generates a trajectory for autonomous driving based on a preset target route, map information, the position information of the vehicle 2, the surrounding environment of the vehicle 2, and the driving state of the vehicle 2. The trajectory corresponds to a driving plan for autonomous driving. Well-known methods related to autonomous driving can be used to generate the trajectory in the trajectory generation unit 34. The same applies to the content of the trajectory.

The remote driving control unit 35 performs remote assistance, remote driving, or automatic driving of the vehicle 2. The remote driving control unit 35 may provide information and instructions to the driver of the vehicle 2 in response to remote instructions from the remote commander R. The remote driving control unit 35 may perform automatic driving of the vehicle 2 based on the surrounding environment of the vehicle 2, the driving state of the vehicle 2, and the route generated by the route generation unit 34. The remote driving control unit 35 may issue instructions related to the automatic driving of the vehicle 2 in response to remote instructions from the remote commander R. The remote driving control unit 35 can perform remote driving and automatic driving of the vehicle 2 by sending control signals to the actuator 25.

The display information acquisition unit 36 acquires display information to be displayed on the remote commander R based on detection data from the external sensors 22 or calculation results from the remote driving ECU 30. The display information may include information on the position of the vehicle 2, the destination, the route of the vehicle 2 generated by the route generation unit 34, and driving conditions (e.g., vehicle speed). The display information may be information outside the operation design domain (ODD) of the autonomous driving system. The ODD is the range within which the autonomous driving system operates as designed.

The display information includes video data of the surrounding environment image that is transmitted to the remote commander R to display the surrounding environment image. The video data of the surrounding environment image may include, for example, an image of the scenery in front of the vehicle 2, an overhead image of the vehicle 2, etc. The video data of the surrounding environment image may also include detection data of an image of the side or rear of the vehicle 2 captured by a camera on the vehicle 2. The display information acquisition unit 36 acquires the video data of the surrounding environment image based on the image captured by the camera of the external sensor 22.

The transmission condition setting unit 37 sets the transmission conditions based on the speed of the vehicle 2. The transmission conditions are conditions (settings) related to the transmission image quality of the surrounding environment image. The transmission image quality is the image quality when transmitting video data of the surrounding environment image from the communication device 4 to the communication network N. The transmission image quality includes a target value for the transmission frame rate of the surrounding environment image (video data) and a target value for the transmission resolution of the surrounding environment image (video data).

The transmission condition setting unit 37 acquires communication quality information of the communication network N. The communication quality information is information relating to the communication quality of the communication network N, and includes, for example, the transmission frame rate and transmission resolution of the surrounding environment image (video data).

The transmission frame rate is the actual frame rate value for the video data of the surrounding environment image transmitted so as to match the target transmission frame rate value. The transmission frame rate is, for example, an actual value provided by the communications carrier of communication network N. The transmission frame rate may also be an estimated value obtained based on information provided by the communications carrier of communication network N.

The transmission resolution is the actual resolution value of the video data of the surrounding environment image transmitted so as to match the target transmission resolution value. The transmission resolution is, for example, an actual value provided by the communications carrier of the communications network N. The transmission resolution may also be an estimated value obtained based on information provided by the communications carrier of the communications network N.

As other communication quality information, the transmission condition setting unit 37 may calculate packet loss of video data of the surrounding environment image based on measured values of the amount of data sent and received during communication.

The transmission condition setting unit 37 determines whether to suppress the transmission image quality of the video data based on the vehicle speed of the vehicle 2 and communication quality information of the communication network N. "Suppressing the transmission image quality of the video data" includes reducing at least one of the transmission frame rate and the transmission resolution. The transmission condition setting unit 37 sets the transmission conditions for the video data of the surrounding environment image according to the vehicle speed of the vehicle 2 so that the remote commander R can obtain the minimum transmission image quality required to remotely support the vehicle 2.

The transmission condition setting unit 37 can adjust the transmission image quality of the surrounding environment image between a predetermined rated transmission image quality (e.g., 1080 bps and 30 FPS) and a transmission image quality with a reduced amount of transmission data compared to the rated transmission image quality. The transmission image quality of the surrounding environment image can be reduced in amount of transmission data compared to the rated transmission image quality, taking into account the visibility of the video data by the remote commander R. "Taking into account the visibility of the video data by the remote commander R" means that the remote commander R can recognize necessary information through the surrounding environment image. One example of "taking into account the visibility of the video data by the remote commander R" is when the remote commander R recognizes a "Stop" sign in the surrounding environment image while remotely driving vehicle 2 and is able to stop vehicle 2 at the stop line.

Specifically, for example, in a situation where a "Stop" sign is present ahead of vehicle 2, remote commander R issues a remote command to brake vehicle 2 while referring to the surrounding environment image received by remote instruction device 1, providing remote support to stop vehicle 2 at the stop line. The time lag (the time required for recognition, judgment, and operation) between remote commander R recognizing the stop sign in the surrounding environment image and issuing the remote command can be, for example, 500 ms to 1000 ms. The time required for communication of the remote command from remote instruction device 1 to vehicle 2 (G2G delay) can be, for example, 100 ms to 300 ms. The time required for actuator 25 in vehicle 2 to operate in response to the remote command (operation delay) can be, for example, 80 ms to 200 ms. The deceleration from actuator 25 operation to vehicle 2 stopping can be, for example, 0.2 G to 0.3 G. By performing calculations that satisfy the above requirements, the distance from the stop sign to the front of vehicle 2 in the direction of travel can be determined according to vehicle speed. This distance corresponds to the timing at which the remote commander R should recognize the stop sign in the surrounding environment image, for example, under deceleration that will prevent a vehicle following vehicle 2 from colliding with vehicle 2 from the rear. The transmission condition setting unit 37 sets the transmission frame rate and transmission resolution of the transmission image quality of the surrounding environment image so as to satisfy the above requirements.

Figure 5(a) is a diagram illustrating the relationship between vehicle speed and the target value of transmission resolution. The horizontal axis of Figure 5(a) is vehicle speed, and the vertical axis is the target value of transmission resolution. The dashed line L1 in Figure 5(a) is the minimum target value of the transmission resolution of the surrounding environment image determined to satisfy the above-mentioned requirements. The dashed line L1 in Figure 5(a) is the preset minimum transmission resolution. The solid line L2 in Figure 5(a) is the target value of the transmission resolution (e.g., 1080 bps) corresponding to a predetermined rated transmission image quality. The solid line L3 in Figure 5(a) is the target value of the transmission resolution corresponding to a transmission image quality with a reduced amount of transmission data compared to the predetermined rated transmission image quality. The target value of the solid line L3 partially overlaps with the minimum transmission resolution. All of the target values of the solid line L3 may be greater than the minimum transmission resolution.

As shown in FIG. 5(a), the rated transmission image quality (solid line L2 and part of solid line L3) can be set to a constant value regardless of vehicle speed, for example. The transmission resolution (dashed line L1), which corresponds to a transmission image quality with a reduced amount of transmission data compared to the rated transmission image quality, can be set to a higher resolution value, for example, as the vehicle speed increases. In other words, the minimum transmission resolution required to satisfy the above requirements is calculated to be a higher resolution value as the vehicle speed increases. This tendency corresponds to the fact that the higher the vehicle speed of the vehicle 2, the higher the resolution required for the video data referenced by the remote commander R in order to properly view the information used by the remote commander R to issue remote instructions. The transmission condition setting unit 37 can adjust the target value of the transmission resolution, for example, between the transmission image quality of solid line L2, the transmission image quality of solid line L3, and the transmission image quality of dashed line L1 in FIG. 5(a).

Figure 5(b) is a diagram illustrating the relationship between vehicle speed and the target value of the transmission frame rate. The horizontal axis of Figure 5(b) is vehicle speed, and the vertical axis is the target value of the transmission frame rate. The dashed line L4 in Figure 5(b) is the minimum target value of the transmission frame rate of the surrounding environment image, which is preset in accordance with an increase or decrease in the vehicle speed of the vehicle 2, as a frame rate required for the remote commander R to recognize various objects in the surrounding environment image while remotely driving the vehicle 2. The dashed line L4 in Figure 5(b) is the preset minimum transmission frame rate. The solid line L5 in Figure 5(b) is the target value of the transmission frame rate corresponding to a predetermined rated transmission image quality (e.g., 30 FPS). The solid line L6 in Figure 5(b) is the target value of the transmission frame rate corresponding to a transmission image quality with a reduced amount of transmission data compared to the predetermined rated transmission image quality. The target value of the solid line L6 partially overlaps with the minimum transmission frame rate. All of the target values of the solid line L6 may be greater than the minimum transmission frame rate.

As shown in Figure 5(b), the rated transmission image quality (solid lines L5 and L6) can be constant relative to vehicle speed, for example. The transmission frame rate (dashed line L4), which corresponds to a transmission image quality with a reduced amount of transmission data compared to the rated transmission image quality, can be set to a higher frame rate value, for example, as the vehicle speed increases. In other words, the minimum transmission frame rate required to satisfy the above requirements is calculated as a higher frame rate value as the vehicle speed increases. This tendency corresponds to the fact that the higher the vehicle speed of the vehicle 2, the higher the transmission frame rate required for the video data referenced by the remote commander R in order to properly view the information used by the remote commander R to issue remote instructions. The transmission condition setting unit 37 can adjust the target value of the transmission frame rate, for example, between the transmission image quality of solid line L5, solid line L6, and dashed line L4 in Figure 5(b).

The transmission condition setting unit 37 determines whether to suppress the transmission image quality, including the transmission frame rate and transmission resolution, based on, for example, the vehicle speed of the vehicle 2 and communication quality information of the communication network N. Here, if the transmission condition setting unit 37 determines that the transmission image quality should be suppressed, it prioritizes reducing the target value of the transmission resolution over the target value of the transmission frame rate.

More specifically, the transmission condition setting unit 37 compares the transmission resolution and transmission frame rate of the video data of the surrounding environment image transmitted so as to achieve the set target transmission resolution and transmission frame rate values based on the communication quality information of the communication network N with the target values. The transmission condition setting unit 37 may determine that the transmission image quality should be suppressed, for example, if the actual transmission resolution value is smaller than the target transmission resolution value by a certain value or more. An estimated transmission resolution value may be used to determine the transmission resolution. Alternatively, the transmission condition setting unit 37 may determine that the transmission image quality should be suppressed if the calculated packet loss of the video data of the surrounding environment image exceeds a certain value. The "certain value" used for the determination may be, for example, a predetermined positive constant value.

When the transmission condition setting unit 37 determines that the transmission image quality should be suppressed, it suppresses the transmission image quality by maintaining the target value of the transmission frame rate and reducing the target value of the transmission resolution. In the example of Figure 5(a), when the target value of the transmission resolution is set to solid line L2, which corresponds to the rated transmission image quality, and the actual value of the transmission resolution becomes smaller than the target value by a certain value or more, or when packet loss of the video data of the surrounding environment image becomes equal to or greater than a certain value, the transmission condition setting unit 37 determines that the transmission image quality should be suppressed. In this case, the transmission condition setting unit 37 switches the target value of the transmission resolution from solid line L2, which corresponds to the rated transmission image quality, to solid line L3. At this time, the transmission condition setting unit 37 maintains the target value of the transmission frame rate at solid line L5 in Figure 5(b), which corresponds to the predetermined rated transmission image quality.

Furthermore, in the example of Figure 5(a), when the target value of transmission resolution is solid line L3, if the actual value of transmission resolution becomes smaller by a certain value or more than the target value of transmission resolution corresponding to solid line L3, or if packet loss of video data of the surrounding environment image becomes a certain value or more, the transmission condition setting unit 37 determines to suppress the transmission image quality. In this case, the transmission condition setting unit 37 switches the target value of transmission resolution from solid line L3 to dashed line L1. At this time, the transmission condition setting unit 37 maintains the target value of the transmission frame rate as solid line L5 in Figure 5(b), which corresponds to the specified rated transmission image quality. Here, when the target value of transmission resolution becomes dashed line L1, it means that the preset minimum transmission resolution has been reached.

When the transmission condition setting unit 37 determines to suppress the transmission image quality because the target transmission resolution has reached a predetermined minimum transmission resolution, it may maintain the target transmission resolution and reduce the target transmission frame rate. For example, in a situation where the target transmission resolution is the dashed line L1 in FIG. 5(a) and the target transmission frame rate is the solid line L5 corresponding to the rated transmission image quality in FIG. 5(b), the transmission condition setting unit 37 may determine to suppress the transmission image quality when the actual transmission frame rate is lower than the target value by a certain value or when packet loss of the video data of the surrounding environment image exceeds a certain value. In this case, the transmission condition setting unit 37 switches the target transmission frame rate from the solid line L5 corresponding to the rated transmission image quality to the solid line L6. At this time, the transmission condition setting unit 37 maintains the target transmission resolution at the dashed line L1 in FIG. 5(a). Furthermore, in the example of FIG. 5(b), when the target value of the transmission frame rate is set to solid line L6, the transmission condition setting unit 37 may determine to suppress the transmission image quality if the actual transmission frame rate becomes smaller than the target value corresponding to solid line L6 by a certain value or more, or if packet loss of the video data of the surrounding environment image exceeds a certain value. In this case, the transmission condition setting unit 37 switches the target value of the transmission frame rate from solid line L6 to dashed line L4. At this time, the transmission condition setting unit 37 maintains the target value of the transmission resolution as dashed line L1 in FIG. 5(a).

If the transmission condition setting unit 37 determines not to suppress the transmission image quality, it may increase the target value of the transmission resolution and also increase the target value of the transmission frame rate. Alternatively, if the transmission condition setting unit 37 determines not to suppress the transmission image quality, it may increase the target value of the transmission frame rate, giving priority over the target value of the transmission resolution. In this case, the transmission condition setting unit 37 may increase the target value of the transmission resolution when the target value of the transmission frame rate reaches the solid line L5, which corresponds to the rated transmission image quality. If the transmission condition setting unit 37 determines not to suppress the transmission image quality, it may maintain the target value of the transmission resolution and also maintain the target value of the transmission frame rate.

Incidentally, the transmission condition setting unit 37 may reduce or increase the target value of the transmission resolution so that the absolute value of the rate of change of the target value of the transmission resolution over time does not exceed a predetermined upper limit of change rate. The upper limit of change rate of the transmission resolution is the upper limit of the absolute value of the rate of change of the target value of the transmission resolution over time. In the example of FIG. 5(a), for example, when switching the target value of the transmission resolution from solid line L2, which corresponds to the rated transmission image quality, to solid line L3, the transmission condition setting unit 37 may gradually decrease the target value of the transmission resolution while limiting the amount of change per predetermined unit of time by the upper limit of change rate of the transmission resolution, rather than changing the target value of the transmission resolution from solid line L2 to solid line L3 in an instant. Conversely, when switching the target value of the transmission resolution from dashed line L1 to solid line L3, the transmission condition setting unit 37 may gradually increase the target value of the transmission resolution while limiting the amount of change per predetermined unit of time by the upper limit of change rate of the transmission resolution, rather than changing the target value of the transmission resolution from dashed line L1 to solid line L3 in an instant.

Similarly, the transmission condition setting unit 37 may reduce or increase the target value of the transmission frame rate so that the absolute value of the rate of change over time of the target value of the transmission frame rate does not exceed a predetermined upper limit rate of change. The upper limit rate of change of the transmission frame rate is the upper limit of the absolute value of the rate of change over time of the target value of the transmission frame rate. In the example of FIG. 5(b), for example, when switching the target value of the transmission frame rate from solid line L5 corresponding to the rated transmission image quality to solid line L6, the transmission condition setting unit 37 may gradually decrease the target value of the transmission frame rate while limiting the amount of change per predetermined unit time by the upper limit rate of change of the transmission frame rate, rather than changing the target value of the transmission frame rate from solid line L5 to solid line L6 in an instant. Conversely, when switching the target value of the transmission frame rate from dashed line L4 to solid line L6, the transmission condition setting unit 37 may gradually increase the target value of the transmission frame rate while limiting the amount of change per predetermined unit time by the upper limit rate of change of the transmission frame rate, rather than changing the target value of the transmission frame rate from dashed line L4 to solid line L6 in an instant.

The transmission condition setting unit 37 may set a target value for the transmission resolution based on the reception resolution of the video data received by the remote instruction device 1. The reception resolution is the resolution value for the video data of the surrounding environment image received by the remote instruction device 1. The reception resolution will be described in more detail below.

The transmission condition setting unit 37 receives, for example, the reception resolution transmitted from the remote instruction server 10 of the remote instruction device 1 via the communication network N. The transmission condition setting unit 37 compares, for example, the target value of the transmission resolution of the video data of the surrounding environment image transmitted to the remote instruction device 1 with the reception resolution (actual value) received from the remote instruction device 1, and determines whether the reception resolution is lower than the target value of the transmission resolution of the video data. If the reception resolution is lower than the target value of the transmission resolution of the video data, the transmission condition setting unit 37 may maintain the target value of the transmission frame rate and set the target value of the transmission resolution based on the reception resolution. For example, if the reception resolution is lower than the target value of the transmission resolution of the video data and is equal to or higher than the minimum transmission resolution, the transmission condition setting unit 37 may set the resolution value of the reception resolution to the target value of the transmission resolution and maintain the target value of the transmission frame rate.

The display information transmitting unit 38 transmits the display information acquired by the display information acquiring unit 36 to the remote instruction device 1. The display information transmitting unit 38 transmits the display information to the remote instruction device 1 via the communication network N using the communication unit 26. For video data of the surrounding environment image, the display information transmitting unit 38 transmits the video data based on the transmission conditions set by the transmission condition setting unit 37 so that the transmission resolution and transmission frame rate are their respective target values.

[Configuration of remote instruction device]
The remote instruction device 1 receives video data transmitted from the communication device 4 of the vehicle 2 via the communication network N. The configuration of the remote instruction device 1 will be described with reference to the drawings. FIG. 3 is a block diagram showing an example of the hardware configuration of a remote instruction server. FIG. 4 is a block diagram showing an example of the configuration of the remote instruction device and the functional configuration of the remote instruction server. As shown in FIGS. 3 and 4, the remote instruction device 1 has a remote instruction server 10 and a commander interface 3.

First, we will explain the hardware configuration of the remote instruction server 10. The remote instruction server 10 is configured as a general-purpose computer equipped with, for example, a processor 10a, a memory unit 10b, a communication unit 10c, and a user interface 10d. The user of the user interface 10d refers to the user (administrator, etc.) of the remote instruction device 1.

The processor 10a runs various operating systems to control the remote instruction server 10. The processor 10a is a computing unit such as a CPU that includes a control device, an arithmetic unit, a register, etc. The processor 10a controls the memory unit 10b, the communication unit 10c, and the user interface 10d. The memory unit 10b is configured to include at least one of memory and storage. Memory is a recording medium such as ROM or RAM. Storage is a recording medium such as an HDD.

The communication unit 10c is a communication device for communicating via the communication network N. The communication unit 10c can be a network device, network controller, network card, etc. The user interface 10d is the input/output unit of the remote instruction device 1 for users such as administrators. The user interface 10d includes output devices such as a display and speaker, and input devices such as a touch panel. Note that the remote instruction device 1 does not necessarily have to be installed in a facility having a remote cockpit, but may be mounted on a mobile object such as a vehicle.

The commander interface 3 is the input/output section of the remote instruction device 1 for the remote commander R. The commander interface 3 has an output section 3a and an instruction input section 3b.

The output unit 3a is a device that outputs information used to remotely instruct the vehicle 2 to the remote commander R. The output unit 3a includes, for example, multiple displays that output images. The output unit 3a may also include a speaker that outputs sound. The speaker may be, for example, a headset speaker worn on the head of the remote commander R. The output unit 3a may also provide information to the remote commander R by vibration, for example, using a vibration actuator provided on the seat.

The multiple displays display images of the surrounding environment for the remote commander R. As an example, the multiple displays display an image of the scenery in front of vehicle 2 captured by a camera on vehicle 2. The multiple displays may also display images of the side or rear of vehicle 2 captured by a camera on vehicle 2.

The instruction input unit 3b is a device to which remote instructions are input by the remote commander R. The instruction input unit 3b may have an operating device with a known configuration, such as an operating lever, an operating pedal, or an operating button.

Next, the functional configuration of the remote instruction server 10 will be described. As shown in FIG. 6, the remote instruction server 10 has an information receiving unit 11, a display image acquisition unit 12, a display control unit 13, and an information transmitting unit 14.

The information receiving unit 11 receives driving information that allows the remote commander R to provide remote support for the vehicle 2. The driving information includes video data of images of the surrounding environment. The driving information may also include information related to the autonomous driving of the vehicle 2, such as the route the vehicle 2 will take when driving autonomously.

The information receiving unit 11 may acquire the reception resolution of the video data. The reception resolution is, for example, an actual value provided by the communications carrier of the communications network N. The reception resolution may also be an estimated value obtained based on information provided by the communications carrier of the communications network N. The information receiving unit 11 may also acquire the reception resolution based on measured values of the amount of data transmitted and received during communication.

The display image acquisition unit 12 acquires an image to be displayed to the remote commander R based on the received driving information. The display image acquisition unit 12 acquires, for example, an image of the surrounding environment from the received video data as the image to be displayed to the remote commander R.

The display control unit 13 controls the display of images on multiple displays for the remote commander R. For example, the display control unit 13 causes the remote commander R to display images of the surrounding environment obtained from video data on multiple displays. The display control unit 13 may also provide the remote commander R with various types of information related to remote support.

The information transmission unit 14 transmits various information and data related to remote instructions from the remote commander R to the communication device 4 of the vehicle 2. For example, when the remote commander R inputs a remote instruction into the instruction input unit 3b of the commander interface 3, the information transmission unit 14 transmits the input remote instruction to the vehicle 2.

The information transmitting unit 14 may transmit the reception resolution to the communication device 4 via the communication network N. For example, the information transmitting unit 14 transmits the reception resolution acquired by the information receiving unit 11 to the communication device 4 of the vehicle 2 via the communication network N.

[Communication Device and Remote Instruction System Processing]
Next, an example of the processing of the communication device 4 and the remote instruction system 100 will be described with reference to Figs. 6 and 7. Fig. 6 is a sequence diagram showing an example of the processing of the remote instruction system. Fig. 7 is a flowchart showing an example of the processing for setting transmission conditions. The processing shown in Figs. 6 and 7 is executed, for example, while the vehicle 2 is being driven.

As shown in FIG. 6, in the remote instruction system 100, the remote driving ECU 30 of the vehicle 2 recognizes the surrounding environment of the vehicle 2 using the surrounding environment recognition unit 32 in S10. The surrounding environment recognition unit 32 recognizes the surrounding environment of the vehicle 2 based on, for example, detection data from the external sensor 22.

In S11, the remote driving ECU 30 of the vehicle 2 acquires video data of the surrounding environment image using the display information acquisition unit 36. The display information acquisition unit 36 acquires video data of the surrounding environment image based on the image captured by the camera of the external sensor 22.

In S12, the remote driving ECU 30 sets the transmission conditions. As the processing of S12, the remote driving ECU 30 performs the processing of the flowchart in Figure 7, for example.

As shown in FIG. 7, in S30, the remote driving ECU 30 acquires the vehicle speed using the driving condition recognition unit 33. The driving condition recognition unit 33 acquires the vehicle speed of the vehicle 2 as the driving condition of the vehicle 2, for example, based on the detection results of the internal sensor 23.

In S31, the remote driving ECU 30 acquires communication quality information using the transmission condition setting unit 37. The transmission condition setting unit 37 acquires communication quality information, for example, based on information provided by the communication carrier of the communication network N, such as the actual value of the transmission frame rate and the actual value of the transmission resolution. The transmission condition setting unit 37 may also acquire an estimated value of the transmission frame rate and an estimated value of the transmission resolution as communication quality information.

In S32, the remote driving ECU 30 uses the transmission condition setting unit 37 to determine whether or not to suppress the transmission image quality. The transmission condition setting unit 37 determines whether or not to suppress the transmission image quality, including the transmission frame rate and transmission resolution, based on, for example, the vehicle speed of the vehicle 2 and communication quality information of the communication network N.

If the transmission condition setting unit 37 determines in S33 that the transmission image quality is to be suppressed (S33: YES), the remote driving ECU 30 proceeds to processing S34. If the transmission condition setting unit 37 determines in S33 that the transmission image quality is not to be suppressed (S33: NO), the remote driving ECU 30 proceeds to processing S37.

In S34, the remote driving ECU 30 determines, via the transmission condition setting unit 37, whether the target value for transmission resolution has reached the minimum transmission resolution. If the remote driving ECU 30 determines in S34 that the target value for transmission resolution has reached the minimum transmission resolution (S34: YES), it proceeds to processing of S36. If the remote driving ECU 30 determines in S34 that the target value for transmission resolution has not reached the minimum transmission resolution (S34: NO), it proceeds to processing of S35.

In S35, the remote driving ECU 30 causes the transmission condition setting unit 37 to maintain the target value of the transmission frame rate while reducing the target value of the transmission resolution. The transmission condition setting unit 37 maintains the target value of the transmission frame rate while reducing the target value of the transmission resolution, for example, by switching the target value of the transmission resolution shown in FIG. 5(a) while not switching the target value of the transmission frame rate shown in FIG. 5(b). The transmission condition setting unit 37 may also reduce the target value of the transmission resolution so that the absolute value of the time rate of change of the target value of the transmission resolution does not exceed a predetermined upper limit rate of change. The remote driving ECU 30 then ends the processing of FIG. 7 and returns to the processing of FIG. 6.

In S36, the remote driving ECU 30 causes the transmission condition setting unit 37 to maintain the target value of the transmission resolution while reducing the target value of the transmission frame rate. The transmission condition setting unit 37 maintains the target value of the transmission resolution while reducing the target value of the transmission frame rate, for example, by not switching the target value of the transmission resolution shown in FIG. 5(a) and switching the target value of the transmission frame rate shown in FIG. 5(b). The transmission condition setting unit 37 may also reduce the target value of the transmission frame rate so that the absolute value of the time rate of change of the target value of the transmission frame rate does not exceed a predetermined upper limit rate of change. The remote driving ECU 30 then ends the processing of FIG. 7 and returns to the processing of FIG. 6.

In S37, the remote driving ECU 30 increases the target value of the transmission frame rate and the target value of the transmission resolution using the transmission condition setting unit 37. The transmission condition setting unit 37 increases the target value of the transmission resolution and the target value of the transmission frame rate, for example, by switching the target value of the transmission resolution shown in FIG. 5(a) and switching the target value of the transmission frame rate shown in FIG. 5(b). Thereafter, the remote driving ECU 30 ends the processing of FIG. 7 and returns to the processing of FIG. 6.

Returning to FIG. 6, in S13, the remote driving ECU 30 of the vehicle 2 transmits video data via the display information transmission unit 38. The display information transmission unit 38 transmits video data of the surrounding environment image to the remote instruction device 1 via the communication network N, for example, using the communication unit 26.

In S20, the remote instruction server 10 of the remote instruction device 1 receives video data via the information receiving unit 11. The information receiving unit 11 receives, for example, video data of an image of the surrounding environment transmitted from the communication device 4 of the vehicle 2 via the communication network N.

In S21, the remote instruction server 10 acquires the reception resolution using the information receiving unit 11. The remote instruction server 10 acquires the resolution value for the received video data of the surrounding environment image as the reception resolution, for example, based on the performance value provided by the communication carrier of the communication network N.

In S22, the remote instruction server 10 transmits the reception resolution via the information transmission unit 14. The information transmission unit 14 transmits the acquired reception resolution to the communication device 4 of the vehicle 2 via the communication network N, for example.

In S14, the remote driving ECU 30 of the vehicle 2 receives the reception resolution using the transmission condition setting unit 37. The transmission condition setting unit 37 receives the reception resolution transmitted, for example, from the remote instruction server 10 of the remote instruction device 1 via the communication network N.

In S15, the remote driving ECU 30 uses the transmission condition setting unit 37 to determine whether the reception resolution is lower than the target value for the transmission resolution of the video data. The transmission condition setting unit 37, for example, compares the target value for the transmission resolution of the video data of the surrounding environment image transmitted to the remote instruction device 1 in S13 with the reception resolution (actual value) received from the remote instruction device 1 in S14, and determines whether the reception resolution is lower than the target value for the transmission resolution of the video data.

If the transmission condition setting unit 37 determines in S15 that the reception resolution is lower than the target value for the transmission resolution of the video data (S15: YES), the remote driving ECU 30 proceeds to processing in S16. If the transmission condition setting unit 37 determines in S15 that the reception resolution is equal to or higher than the target value for the transmission resolution of the video data (S15: NO), the remote driving ECU 30 ends the processing of Figure 6. The remote driving ECU 30 may restart the processing of Figure 6 after a predetermined time has elapsed.

In S16, the remote driving ECU 30 maintains the target value of the transmission frame rate and sets the target value of the transmission resolution based on the reception resolution using the transmission condition setting unit 37. For example, if the reception resolution is equal to or greater than the minimum transmission resolution indicated by the dashed line L1 in FIG. 5(a), the transmission condition setting unit 37 may set the resolution value of the reception resolution to the target value of the transmission resolution and maintain the target value of the transmission frame rate shown in FIG. 5(b). In this case, as the transmission condition setting process, the processes of S12 and S16 are performed each time the process of FIG. 6 is executed. Note that, for example, if the reception resolution is less than the minimum transmission resolution indicated by the dashed line L1 in FIG. 5(a), the transmission condition setting unit 37 may not perform the process of S16. Thereafter, the remote driving ECU 30 ends the process of FIG. 6. The remote driving ECU 30 may restart the process of FIG. 6 after a predetermined time has elapsed.

As described above, the communication device 4 determines whether to suppress the transmission image quality of the video data, including the transmission frame rate and transmission resolution, based on the vehicle speed of the vehicle 2 and communication quality information of the communication network N. If it is determined that the transmission image quality should be suppressed, the target value of the transmission frame rate is maintained and the target value of the transmission resolution is reduced. This reduces the amount of transmission data without lowering the transmission frame rate of the video data, while taking into account both the vehicle speed of the vehicle 2 and the communication quality information of the communication network N. This prevents, for example, the transmission frame rate from being insufficient compared to the vehicle speed. This makes it possible to achieve both good visibility of the video data by the remote commander R and a reduction in the amount of transmission data.

The transmission condition setting unit 37 reduces or increases the target value of the transmission resolution so that the absolute value of the time rate of change of the target value of the transmission resolution does not exceed a predetermined upper limit. This makes it possible to reduce the impact on the visibility of the video data displayed by the remote commander R of sudden changes in the transmission resolution or transmission frame rate. Specifically, if a decision is made not to reduce the transmission image quality after the transmission image quality has been reduced, causing a sudden increase in the target value of the transmission resolution or the target value of the transmission frame rate, the target value will be too high for the communication quality information, which could actually cause the image of the surrounding environment to freeze due to packet loss of the video data. Therefore, by reducing sudden changes in the transmission resolution or transmission frame rate, it is possible to reduce the impact on the visibility of the video data displayed by the remote commander R. Note that image freeze refers to, for example, a state in which the image is not updated (the last displayed image remains displayed) continuing for a predetermined period of time (e.g., one second).

When the transmission condition setting unit 37 determines that the transmission image quality should be reduced if the target value of the transmission resolution has reached the preset minimum transmission resolution, it maintains the target value of the transmission resolution and reduces the target value of the transmission frame rate. This makes it possible to reduce the amount of transmission data without causing the transmission resolution to fall below the minimum transmission resolution.

When the receiving resolution of the video data is lower than the target value for the transmission resolution, the remote instruction system 100 maintains the target value for the transmission frame rate and sets the target value for the transmission resolution based on the receiving resolution. This allows the amount of data transmitted to be efficiently reduced by utilizing information about the receiving resolution.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments.

In the above embodiment, the transmission condition setting unit 37 reduced or increased the target value of the transmission resolution so that the absolute value of the time rate of change of the target value of the transmission resolution does not exceed a predetermined upper limit rate of change, but this example is not limiting. For example, if there is little possibility of a high target value being set for the communication quality information, such as when a temporary deterioration in communication quality has been resolved, the transmission condition setting unit 37 may reduce or increase the target value of the transmission resolution beyond the upper limit rate of change.

In the above embodiment, when the transmission condition setting unit 37 determines that the transmission image quality should be suppressed because the target value of the transmission resolution has reached the minimum transmission resolution, it maintains the target value of the transmission resolution and reduces the target value of the transmission frame rate. However, this is not limited to this example. For example, when the transmission condition setting unit 37 determines that the transmission image quality should be suppressed because the target value of the transmission resolution has reached the minimum transmission resolution, it may limit the imaging range of the vehicle 2 included in the video data using a known method, instead of reducing the target value of the transmission frame rate, and reduce the volume of the video data.

In the above embodiment, when the reception resolution is lower than the target transmission resolution of the video data and the reception resolution is equal to or greater than the minimum transmission resolution indicated by dashed line L1 in FIG. 5(a), the transmission condition setting unit 37 maintains the target transmission frame rate and sets the target transmission resolution based on the reception resolution. However, this example is not limited to this. The transmission condition setting unit 37 may set the target transmission resolution based on the reception resolution even if the reception resolution is lower than the minimum transmission resolution indicated by dashed line L1 in FIG. 5(a). Alternatively, the transmission condition setting unit 37 may not have the function of using the reception resolution as the target transmission resolution. For example, in the process of FIG. 6, steps S13, S14, S20, S21, and S22 may be omitted. That is, in the example of FIG. 6, the communication device 4 may perform steps S10 to S13. In this case, step S12 is performed as the transmission condition setting process each time the process of FIG. 6 is executed.

In the above embodiment, a vehicle 2 with an autonomous driving function was exemplified as a vehicle capable of providing remote support, but the autonomous driving function is not required. The vehicle may have a driving assistance function instead of an autonomous driving function, or may have only a manual driving function. In short, the vehicle must be configured to be able to provide remote support in response to remote instructions from a remote commander.

In the above embodiment, for example, a cloud server may be interposed between the vehicle 2 and the remote instruction device 1. The cloud server, for example, relays various information and data transmitted from the vehicle 2 so that it is transmitted to the remote instruction device 1. The cloud server, for example, relays various information and data transmitted from the remote instruction device 1 so that it is transmitted to the vehicle 2.

The functions of the remote driving ECU 30 may be realized, for example, by controlling the actuators 25 of the vehicle 2 in response to vehicle control requests from an ADK [Autonomous Driving Kit] connected to the vehicle 2 via a communication interface.

1... remote instruction device, 2... vehicle, 4... communication device, 11... information receiving unit, 14... information transmitting unit, 37... transmission condition setting unit, 38... display information transmitting unit (transmitting unit), 100... remote instruction system, N... communication network, R... remote commander.

Claims (4)

A communication device mounted on a vehicle capable of performing remote support in response to a remote instruction from a remote commander, the communication device transmitting video data from the vehicle via a communication network,
a transmission condition setting unit that sets transmission conditions including a target value of a transmission frame rate of the video data and a target value of a transmission resolution of the video data based on a speed of the vehicle;
a transmitting unit that transmits the video data based on the transmission conditions so that the transmission resolution and the transmission frame rate reach the respective target values,
The transmission condition setting unit
determining whether to suppress the transmission image quality of the video data, including the transmission frame rate and the transmission resolution, based on the vehicle speed of the vehicle and communication quality information of the communication network;
When it is determined that the transmission image quality is to be suppressed, the communication device suppresses the transmission image quality by maintaining the target value of the transmission frame rate and reducing the target value of the transmission resolution. The communication device of claim 1, wherein the transmission condition setting unit reduces or increases the target value of the transmission resolution so that the absolute value of the time rate of change of the target value of the transmission resolution does not exceed a predetermined upper limit rate of change. The communication device according to claim 1 or 2, wherein when the transmission condition setting unit determines that the transmission image quality should be suppressed if the target value of the transmission resolution has reached a predetermined minimum transmission resolution, it maintains the target value of the transmission resolution and reduces the target value of the transmission frame rate. 3. A remote instruction system including the communication device according to claim 1 or 2 and a remote instruction device that receives the video data transmitted from the communication device via a communication network,
The remote instruction device
an information receiving unit that acquires a reception resolution of the video data;
an information transmitting unit that transmits the reception resolution to the communication device via the communication network;
A remote instruction system in which, when the receiving resolution is smaller than the target value of the transmission resolution of the video data, the transmission condition setting unit maintains the target value of the transmission frame rate and sets the target value of the transmission resolution based on the receiving resolution.