JP7790315B2 - Vehicle display control device, method, and program - Google Patents

Description

This disclosure relates to a vehicle display control device, a vehicle display control method, and a vehicle display control program.

Patent document 1 describes a technology that uses indicator lines (arrows) to indicate which real-world targets an object displayed on a display corresponds to, thereby informing the user which targets the assistance system is capturing.

Japanese Patent Application Laid-Open No. 2022-013827

In assistance systems such as advanced driver assistance systems (ADAS), the operation state of the assistance system changes depending on the reliability of sensor fusion (reliability of the target detection state). For example, in bad weather, this reliability decreases, causing some of the functions provided by the assistance system to become inoperable. In contrast, the technology described in Patent Document 1 displays a certain object on the display regardless of the operation state of the assistance system (target detection state), which could lead the user to mistakenly believe that the assistance system is operating normally based on the display state of the object on the display.

This disclosure was made in consideration of the above, and aims to provide a vehicle display control device, vehicle display control method, and vehicle display control program that can prevent misidentification of the operating state of an assistance system that switches depending on the reliability of the target detection state.

A vehicle display control device according to a first aspect includes a calculation unit that calculates the reliability of the detection state of a target object around the vehicle from detection information from a plurality of sensors that each detect the target object, and a control unit that changes the display mode of a depiction corresponding to the target object displayed on a display unit based on the reliability calculated by the calculation unit.

In the first aspect, the reliability of the detection state of the target is calculated from the detection information of multiple sensors that each detect targets around the vehicle, and the display mode of the drawing corresponding to the target displayed on the display unit is changed based on the calculated reliability. As a result, when the operating state of the assistance system changes, the user can recognize that the operating state of the assistance system has changed from the change in the display mode of the drawing corresponding to the target, thereby preventing misinterpretation of the operating state of the assistance system.

In a second aspect, in the first aspect, the control unit changes the display mode of the drawing corresponding to the target by at least one of lowering the contrast of the drawing as the reliability decreases, changing the color of the drawing according to the reliability, and displaying the drawing in a manner indicating the reliability.

According to the second aspect, the user can intuitively recognize that the operating state of the assistance system has changed from a change in the display mode of the drawing corresponding to the target object.

In a third aspect, in the first or second aspect, the plurality of sensors include a camera and a radar, and the control unit further displays at least one of the following: first information indicating which sensor has a reduced reliability of the detection state; second information indicating the cause of the reduced reliability of the detection state; and third information indicating a function that may not be operational.

According to the third aspect, the user can further understand the details of changes in the operating status of the assistance system by referring to any of the first to third information displayed.

A fourth aspect of the vehicle display control method involves a computer executing a process that includes calculating the reliability of the detection state of a target object around the vehicle from detection information from multiple sensors that each detect the target object, and changing the display mode of a depiction corresponding to the target object displayed on a display unit based on the calculated reliability.

According to the fourth aspect, as with the first aspect, it is possible to prevent misidentification of the operation state of the assistance system, which switches depending on the reliability of the target detection state.

A vehicle display control program according to a fifth aspect causes a computer to execute processing including calculating the reliability of the detection state of a target object around the vehicle based on detection information from a plurality of sensors that each detect the target object, and changing the display mode of a depiction corresponding to the target object displayed on a display unit based on the calculated reliability.

According to the fifth aspect, as with the first aspect, it is possible to prevent misidentification of the operation state of the assistance system, which switches depending on the reliability of the target detection state.

The present disclosure has the advantage of being able to prevent misidentification of the operation state of an assistance system, which switches depending on the reliability of the target detection state.

1 is a block diagram showing a schematic configuration of an in-vehicle system according to an embodiment; FIG. 2 is a functional block diagram of the in-vehicle system. 10 is a flowchart showing a display control process. 10A and 10B are diagrams illustrating an example of changes in the display mode of an object of a preceding vehicle according to the overall reliability of the preceding vehicle. FIG. 10 is an image diagram showing another example of a display mode of an object of a leading vehicle.

An example of an embodiment of the present disclosure will be described in detail below with reference to the drawings. As shown in FIG. 1, an in-vehicle system 10 according to this embodiment includes a camera 12, a camera ECU 14, a radar 34, a radar ECU 36, a lidar 56, a lidar ECU 58, an AD/ADAS ECU 78, a meter ECU 98, a meter display 120, and a system bus 122. Note that, hereinafter, a vehicle equipped with the in-vehicle system 10 will be referred to as the "host vehicle." Note that the camera ECU 14, radar ECU 36, lidar ECU 58, AD/ADAS ECU 78, and meter ECU 98 are an example of a vehicle display control device according to the present disclosure.

The camera 12 captures images of objects around the vehicle and outputs the captured images (detection information). The camera 12 is connected to the camera ECU 14. The camera ECU 14 includes a CPU (Central Processing Unit) 16, a memory 18 such as a ROM (Read Only Memory) or RAM (Random Access Memory), and a non-volatile storage unit 20 such as a HDD (Hard Disk Drive) or SSD (Solid State Drive). The camera ECU 14 also includes an input/output I/F (Interface) 22 and a communication I/F 24. The camera 12 is connected to the input/output I/F 22, and the communication I/F 24 is connected to a system bus 122. The CPU 16, memory 18, storage unit 20, input/output I/F 22, and communication I/F 24 are all connected to an internal bus 26, enabling them to communicate with each other.

A camera processing program 28 is also stored in the memory unit 20 of the camera ECU 14. The camera processing program 28 is read from the memory unit 20 and loaded into the memory 18, and the camera processing program 28 loaded into the memory 18 is executed by the CPU 16, causing the camera ECU 14 to function as the object generation processing unit 30 and reliability calculation unit 32 shown in FIG. 2 .

The object generation processing unit 30 generates objects corresponding to targets around the vehicle detected by the camera 12, based on the image capture results (detection information) input from the camera 12. The reliability calculation unit 32 calculates the reliability of the detection state by the camera 12, based on the image capture results (detection information) input from the camera 12. That is, the camera 12's target detection accuracy decreases in situations such as backlighting or nighttime. The reliability calculation unit 32 determines the current situation and calculates the reliability of the detection state by the camera 12 so that the reliability decreases as the current situation approaches a situation in which the target detection accuracy by the camera 12 decreases.

The radar 34 measures the distance to targets around the vehicle by transmitting search waves (e.g., millimeter waves) toward the vicinity of the vehicle and receiving reflected waves, and outputs the distance measurement results (detection information). The radar 34 is connected to the radar ECU 36. The radar ECU 36 includes a CPU 38, a memory 40 such as ROM or RAM, and a non-volatile storage unit 42 such as an HDD or SSD. The radar ECU 36 also includes an input/output I/F 44 and a communication I/F 46. The input/output I/F 44 is connected to the radar 34, and the communication I/F 46 is connected to the system bus 122. The CPU 38, memory 40, storage unit 42, input/output I/F 44, and communication I/F 46 are all connected to an internal bus 48, allowing them to communicate with each other.

The radar ECU 36 also stores a radar processing program 50 in its storage unit 42. The radar ECU 36 reads the radar processing program 50 from the storage unit 42 and stores it in its memory 40. The radar processing program 50 stored in the memory 40 is then executed by the CPU 38, causing the radar ECU 36 to function as the object generation processing unit 52 and reliability calculation unit 54 shown in FIG. 2.

The object generation processing unit 52 generates objects corresponding to targets around the vehicle detected by the radar 34 based on the ranging results (detection information) input from the radar 34. The reliability calculation unit 54 calculates the reliability of the detection state by the radar 34 based on the ranging results (detection information) input from the radar 34. That is, the radar 34's target detection accuracy decreases in situations such as rainy weather or when dirt adheres to the detection unit. The reliability calculation unit 54 determines the current situation and calculates the reliability of the detection state by the radar 34 so that the reliability decreases as the current situation approaches a situation in which the target detection accuracy by the radar 34 decreases.

The LIDAR 56 measures the distance to targets around the vehicle by transmitting search waves (e.g., laser light) toward the vicinity of the vehicle and receiving reflected waves, and outputs the distance measurement results (an example of detection information). The LIDAR 56 is connected to the LIDAR ECU 58. The LIDAR ECU 58 includes a CPU 60, a memory 62 such as ROM or RAM, and a non-volatile storage unit 64 such as an HDD or SSD. The LIDAR ECU 58 also includes an input/output I/F 66 and a communication I/F 68. The LIDAR 56 is connected to the input/output I/F 66, and the communication I/F 68 is connected to a system bus 122. The CPU 60, memory 62, storage unit 64, input/output I/F 66, and communication I/F 68 are all connected to an internal bus 70, allowing them to communicate with each other.

The memory unit 64 of the rider ECU 58 also stores a rider processing program 72. The rider processing program 72 is read from the memory unit 64 and loaded into the memory 62, and the rider processing program 72 loaded into the memory 62 is executed by the CPU 60, causing the rider ECU 58 to function as the object generation processing unit 74 and reliability calculation unit 76 shown in FIG. 2 .

The object generation processing unit 74 generates objects corresponding to targets around the vehicle detected by the LIDAR 56 based on the distance measurement results (detection information) input from the LIDAR 56. The reliability calculation unit 76 calculates the reliability of the detection state by the LIDAR 56 based on the distance measurement results (detection information) input from the LIDAR 56. That is, the LIDAR 56's target detection accuracy decreases in situations where, for example, dirt has adhered to the detection unit. The reliability calculation unit 76 determines the current situation and calculates the reliability of the detection state by the LIDAR 56 so that the reliability decreases as the current situation approaches a situation in which the target detection accuracy by the LIDAR 56 decreases.

The AD/ADAS ECU 78 includes a CPU 80, a memory 82 such as ROM or RAM, and a non-volatile storage unit 84 such as an HDD or SSD. The AD/ADAS ECU 78 also includes a communication I/F 86, which is connected to a system bus 122. The CPU 80, memory 82, storage unit 84, and communication I/F 86 are each connected to an internal bus 88 and are capable of communicating with each other.

The AD/ADAS ECU 78 also stores an ADAS processing program 90 in its storage unit 84. The ADAS processing program 90 is read from the storage unit 84 and loaded into the memory 82, and the ADAS processing program 90 loaded into the memory 82 is executed by the CPU 80, causing the AD/ADAS ECU 78 to function as the I/O layer processing unit 92, sensor fusion processing unit 94, and driving assistance control unit 96 shown in FIG. 2 .

The I/O layer processing unit 92 acquires object information and reliability information from each sensor ECU (camera ECU 14, radar ECU 36, and lidar ECU 58) and outputs the acquired information to the sensor fusion processing unit 94. The sensor fusion processing unit 94 performs sensor fusion processing to calculate the position and speed of targets present around the vehicle based on the information acquired by the I/O layer processing unit 92 from each sensor ECU, and calculates the overall reliability of the calculated detection state of targets present around the vehicle (also known as the probability of target presence). The sensor fusion processing unit 94, together with the reliability calculation units 32, 54, and 76, are examples of calculation units in this disclosure.

The driving assistance control unit 96 is a functional block that functions as an advanced driving assistance system (assistance system) and has the ability to perform multiple driving assistance controls based on the results of sensor fusion processing (target detection results) by the sensor fusion processing unit 94. Examples of driving assistance controls performed by the driving assistance control unit 96 include adaptive cruise control (ACC), blind spot monitoring, automatic braking, and a lane keeping assist system (LKA). However, if the reliability of the target detection state decreases, the accuracy of each driving assistance control also decreases. Furthermore, for example, the control accuracy of the blind spot monitoring decreases more quickly in response to a decrease in the reliability of the target detection state compared to other driving assistance controls. For example, the degree to which control accuracy is affected by a decrease in the reliability of the target detection state differs for each individual driving assistance control.

For this reason, the driving assistance control unit 96 sets a separate threshold for each driving assistance control as to whether to execute driving assistance control based on the overall reliability of the target detection state calculated by sensor fusion processing. As a result, the driving assistance control unit 96 executes all driving assistance controls when the overall reliability of the target detection state is very high (for example, 95% or higher), and does not execute some driving assistance controls when the overall reliability of the target detection state drops slightly (for example, about 70%). Furthermore, the driving assistance control unit 96 does not execute multiple driving assistance controls when the overall reliability of the target detection state drops further (for example, about 50%), and does not execute all driving assistance controls when the overall reliability of the target detection state is extremely low (for example, about 10%).

The meter ECU 98 includes a CPU 100, a memory 102 such as ROM or RAM, and a non-volatile storage unit 104 such as an HDD or SSD. The meter ECU 98 also includes an input/output I/F 108 and a communication I/F 106. A meter display 120 is connected to the input/output I/F 108, and a system bus 122 is connected to the communication I/F 106. The CPU 100, memory 102, storage unit 104, input/output I/F 108, and communication I/F 106 are each connected to an internal bus 110 and are capable of communicating with each other.

The storage unit 104 of the meter ECU 98 also stores a display processing program 112 and an object reliability index MAP 114. The meter ECU 98 functions as the meter display unit 118 shown in Figure 2 by reading the display processing program 112 from the storage unit 104 and loading it into the memory 102, and then executing the display processing program 112 loaded into the memory 102 by the CPU 100.

In the object reliability index MAP 114, information defining the relationship between the overall reliability of the detection state of the target and the display mode of the drawing of the object corresponding to the target to be displayed on the meter display 120 is pre-registered. The meter display unit 118 changes the display mode of the drawing of the object corresponding to the target to be displayed on the meter display 120 based on the information stored in the object reliability index MAP 114 in association with the overall reliability calculated by the sensor fusion processing unit 94.

The meter display unit 118 is an example of a control unit in the present disclosure. The camera processing program 28, radar processing program 50, lidar processing program 72, ADAS processing program, and display processing program 112 are examples of vehicle display control programs in the present disclosure.

Next, as an operation of this embodiment, the display control process executed by the in-vehicle system 10 according to this embodiment while, for example, the ignition switch of the host vehicle is on will be described with reference to FIG. 3. Note that, in the following, a preceding vehicle traveling ahead of the host vehicle will be used as an example of a target present around the host vehicle, but targets in this disclosure are not limited to preceding vehicles.

In step 200 of the display control process, the camera ECU 14 acquires detection information (photographed results) input from the camera 12. In step 202, the object generation processing unit 30 of the camera ECU 14 generates an object corresponding to the preceding vehicle detected by the camera 12 based on the detection information acquired from the camera 12. In step 204, the reliability calculation unit 32 of the camera ECU 14 calculates the reliability of the detection state by the camera 12.

In the next step 206, the radar ECU 36 acquires the detection information (ranging results) input from the radar 34. In step 208, the object generation processing unit 52 of the radar ECU 36 generates an object corresponding to the preceding vehicle detected by the radar 34 based on the detection information acquired from the radar 34. In step 210, the reliability calculation unit 54 of the radar ECU 36 calculates the reliability of the detection state by the radar 34.

In step 212, the LIDAR ECU 58 acquires the detection information (distance measurement results) input from the LIDAR 56. In step 214, the object generation processing unit 74 of the LIDAR ECU 58 generates an object corresponding to the preceding vehicle detected by the LIDAR 56. In step 216, the reliability calculation unit 76 of the LIDAR ECU 58 calculates the reliability of the detection state by the LIDAR 56.

In step 218, the I/O layer processing unit 92 of the AD/ADAS ECU 78 acquires detection information, the object of the preceding vehicle, and the reliability of the detection state from the camera ECU 14, radar ECU 36, and lidar ECU 58. In step 220, the sensor fusion processing unit 94 of the AD/ADAS ECU 78 performs sensor fusion processing to calculate the position, speed, and overall reliability of the detection state of the preceding vehicle based on the information acquired from the camera ECU 14, radar ECU 36, and lidar ECU 58.

In step 222, the meter display unit 118 of the meter ECU 98 acquires the overall reliability of the detection state of the preceding vehicle from the AD/ADAS ECU 78. The meter display unit 118 then executes processing to change the display mode of the object corresponding to the preceding vehicle displayed on the meter display 120 based on the information stored in the object reliability index MAP 114 in association with the acquired overall reliability of the detection state of the preceding vehicle.

Specifically, when the overall reliability of the detection state of the preceding vehicle is very high (e.g., 95% or higher), the contrast of the object 130 corresponding to the preceding vehicle is maximized and the intensity of the display color (e.g., red) is maximized, as shown in Figure 4(A). In this case, a user viewing the meter display 120 can recognize that all driving assistance controls are operating from the contrast and intensity of the display color of the displayed object 130.

Furthermore, if the overall reliability of the detection state of the preceding vehicle drops slightly from the state shown in Figure 4(A) (for example, to about 70%), the contrast of the object 130 corresponding to the preceding vehicle is reduced by a predetermined amount and the intensity of the display color (for example, red) is reduced by a predetermined amount, as shown in Figure 4(B). In this case, a user viewing the meter display 120 can recognize that some of the driving assistance controls are inactive from the contrast and intensity of the display color of the displayed object 130.

Furthermore, if the overall reliability of the detection state of the preceding vehicle further decreases from the state shown in Figure 4(B) (for example, to approximately 50%), the contrast of the object 130 corresponding to the preceding vehicle is further decreased by a predetermined amount and the intensity of the display color (for example, red) is further decreased by a predetermined amount, as shown in Figure 4(C). In this case, a user viewing the meter display 120 can recognize that multiple driving assistance controls are inactive from the contrast and intensity of the display color of the displayed object 130.

Furthermore, if the overall reliability of the detection state of the preceding vehicle drops extremely (for example, to around 10%), the contrast of the object corresponding to the preceding vehicle is further reduced by a predetermined amount and the intensity of the display color (for example, red) is further reduced by a predetermined amount, making the object difficult to see, as shown in Figure 4(D). In this case, a user viewing the meter display 120 can recognize from the display state of the object that all driving assistance controls are inactive.

In the next step 224, for example, the meter ECU 98 determines whether the vehicle's ignition switch has been turned off. If the determination in step 224 is negative, the process returns to step 200, and steps 200 to 224 are repeated until the determination in step 224 is positive. If the determination in step 224 is positive, the display control process ends.

As described above, in this embodiment, the reliability calculation units 32, 54, 76 and the sensor fusion processing unit 94 calculate the reliability of the detection state of the target from the detection information of the camera 12, radar 34, and lidar 56, which respectively detect targets around the vehicle, and the meter display unit 118 changes the display mode of the drawing corresponding to the target displayed on the meter display 120 based on the calculated reliability. As a result, when the operating state of the assistance system changes, the user can recognize that the operating state of the assistance system has changed from the change in the display mode of the drawing corresponding to the target, thereby preventing misinterpretation of the operating state of the assistance system.

In addition, in this embodiment, the meter display unit 118 changes the display mode of the drawing corresponding to the target by lowering the contrast of the drawing as the reliability decreases and by changing the color of the drawing according to the reliability. This allows the user to intuitively recognize that the operating state of the assistance system has changed from the change in the display mode of the drawing corresponding to the target.

While the above embodiment describes a configuration in which the contrast and display color intensity of the object 130 corresponding to the preceding vehicle are changed in accordance with a change in the overall reliability of the detection state of the preceding vehicle, the present disclosure is not limited to this. For example, as the overall reliability of the detection state of the preceding vehicle decreases, the hue of the display color of the object 130 corresponding to the preceding vehicle may be switched, for example, from blue to yellow to red. As an example, as shown in FIG. 5(A), a character string 132 indicating the overall reliability of the detection state of the preceding vehicle may be added to the object 130 corresponding to the preceding vehicle. Alternatively, the overall reliability of the detection state of the preceding vehicle may be indicated by an indicator or the like instead of the character string 132. These configurations also allow the user to intuitively recognize that the operating state of the assistance system has changed.

Furthermore, the character string added to the object 130 corresponding to the preceding vehicle is not limited to the character string 132 indicating the overall reliability of the detection state of the preceding vehicle. As an example, as shown in FIG. 5(B), at least one of a first character string 134 indicating which sensor has a reduced reliability of the detection state of the preceding vehicle, a second character string 136 indicating the cause of the reduced reliability of the detection state of the preceding vehicle, and a third character string 138 indicating a function that may not be operational may be further displayed. This allows the user to grasp detailed changes in the operational state of the assistance system by referring to any of the further displayed first character string 134 to third character string 138. Note that the first character string 134 is an example of first information in the present disclosure, the second character string 136 is an example of second information in the present disclosure, and the third character string 138 is an example of third information in the present disclosure.

In addition, in the above embodiment, a preceding vehicle traveling ahead of the host vehicle was described as an example of a target in this disclosure, but the target in this disclosure is not limited to a preceding vehicle. The target in this disclosure may also be, for example, a vehicle traveling in a lane adjacent to the lane in which the host vehicle is traveling, or a pedestrian or bicycle present in the vicinity of the host vehicle.

In the above embodiment, the camera ECU 14 is provided with the object generation processing unit 30 and reliability calculation unit 32, the radar ECU 36 is provided with the object generation processing unit 52 and reliability calculation unit 54, and the lidar ECU 58 is provided with the object generation processing unit 74 and reliability calculation unit 76. However, this is not limited to this, and at least one of the object generation processing unit and the reliability calculation unit may be provided in the AD/ADAS ECU 78.

Furthermore, in the above embodiment, a meter display 120 is used as an example of a display unit in the present disclosure, but this is not limited to this. The display unit may be a HUD (Head Up Display) or a center display provided in the center of an instrument panel, for example.

In addition, in the above embodiment, of the camera processing program 28, radar processing program 50, LIDAR processing program 72, ADAS processing program 90, and display processing program 112, which are examples of a vehicle display control program according to the present disclosure, the camera processing program 28 is pre-stored (installed) in the storage unit 20, the radar processing program 50 in the storage unit 42, the LIDAR processing program 72 in the storage unit 64, the ADAS processing program 90 in the storage unit 84, and the display processing program 112 in the storage unit 104. However, the vehicle display control program according to the present disclosure can also be provided in a form recorded on a non-transitory recording medium such as an HDD, SSD, or DVD.

Furthermore, the vehicle display control device disclosed herein can also be applied to vehicles capable of autonomous driving.

10 In-vehicle system 12 Camera 14 Camera ECU
32 Reliability calculation unit (calculation unit)
34 Radar 36 Radar ECU
54 Reliability calculation unit (calculation unit)
56 Rider 58 Rider ECU
76 Reliability calculation unit (calculation unit)
78 AD/ADAS ECU
94 Sensor fusion processing unit (calculation unit)
98 Meter ECU
118 Meter display unit (control unit)
130 Objects

Claims (5)

a calculation unit that calculates reliability of detection states of targets around the vehicle based on detection information from a plurality of sensors that detect the targets;
a control unit that changes a display mode of a drawing corresponding to the target displayed on a display unit based on the reliability calculated by the calculation unit, and causes the display unit to display third information indicating a driving assistance control that may not be executed among a plurality of driving assistance controls that a driving assistance control unit can execute based on the target detection results by the plurality of sensors ; and
A display control device for a vehicle including: A display control device for a vehicle as described in claim 1, wherein the control unit changes the display mode of the drawing corresponding to the target by at least one of lowering the contrast of the drawing as the reliability decreases, changing the color of the drawing according to the reliability, and displaying the drawing in a manner indicating the reliability. 3. A display control device for a vehicle as described in claim 1 or claim 2, wherein the plurality of sensors include a camera and a radar, and the control unit further displays at least one of first information indicating which sensor has a reduced reliability of the detection state, and second information indicating a factor in the reduced reliability of the detection state. A display control method for a vehicle in which a computer performs processing including calculating the reliability of the detection state of targets around the vehicle from detection information from a plurality of sensors that each detect the targets, changing the display mode of a drawing corresponding to the target displayed on a display unit based on the calculated reliability, and displaying third information on the display unit that indicates driving assistance controls that may not be executed out of a plurality of driving assistance controls that a driving assistance control unit can execute based on the target detection results from the plurality of sensors . On the computer,
A display control program for a vehicle for executing a process including: calculating the reliability of the detection state of a target from detection information from a plurality of sensors that each detects a target around the vehicle; changing the display mode of a drawing corresponding to the target displayed on a display unit based on the calculated reliability; and displaying on the display unit third information indicating driving assistance controls that may not be executed out of a plurality of driving assistance controls that can be executed by a driving assistance control unit based on the target detection results from the plurality of sensors.