JP7831399B2 - Information processing device - Google Patents

Description

This invention relates to an information processing device for diagnosing the driving of a vehicle driver.

Patent Document 1 discloses a technology that can provide diagnostic results tailored to a driver's driving skill and effectively utilize those results. Specifically, Patent Document 1 discloses a driving skill diagnostic device that determines the driver's proficiency level and increases the number of diagnostic items displayed as the driver's proficiency level increases. When increasing the number of diagnostic items displayed, for example, it adds diagnostic items that were not previously displayed in order of poor results, or adds diagnostic items that were not previously displayed in order of high relevance to the previously provided diagnostic items.

Japanese Patent Publication No. 2014-031050

Simply presenting the driver with the driving assessment results after the drive was insufficient for the driver to fully understand the strengths and weaknesses of their own driving.

This invention was made in consideration of the above facts, and aims to provide an information processing device that allows the driver to understand the results of safety checks while driving a vehicle.

The information processing device according to the first embodiment includes: an acquisition unit that acquires vehicle speed, steering angle, yaw rate, and a photograph of the driver of the vehicle from the vehicle; an extraction unit that extracts driving scenes for a period in which the vehicle speed, steering angle, and yaw rate acquired by the acquisition unit satisfy predetermined extraction conditions; an evaluation unit that evaluates the driver's safety check based on the photographed images acquired by the acquisition unit at a predetermined time before and after the start of the driving scene extracted by the extraction unit; and a control unit that controls the notification unit to notify the evaluation of the safety check by the evaluation unit from the notification unit while the vehicle is in operation . The evaluation unit performs the evaluation by duplicating data for a predetermined evaluation time longer than the first hour, using photographed images and vehicle information for a predetermined evaluation time, every predetermined first hour, and outputs the earlier evaluation result first when duplicated .

According to the first embodiment, the vehicle speed, steering angle, yaw rate, and a captured image of the vehicle's driver are acquired from the vehicle by the acquisition unit.

The extraction unit extracts driving scenes during periods when the vehicle speed, steering angle, and yaw rate acquired by the acquisition unit meet predetermined extraction conditions.

The evaluation unit evaluates the driver's safety checks based on images captured by the acquisition unit at predetermined time intervals before and after the start of the driving scene extracted by the extraction unit.

Furthermore, the control unit controls the notification unit to announce the safety check evaluation performed by the evaluation unit while the vehicle is in operation. This makes it possible for the driver to be aware of the safety check evaluation results while the vehicle is running.

The information processing device according to the second embodiment, in the information processing device according to the first embodiment, has the control unit that notifies the driver's driving evaluation, including the safety confirmation evaluation, from the notification unit at predetermined intervals, and when the evaluation unit has performed the safety confirmation evaluation, it controls the notification unit to notify the safety confirmation evaluation at the next notification timing.

According to the second embodiment, the driver can be made aware of the safety check evaluation at an early stage while the vehicle is in operation.

The information processing device according to the third embodiment, in the information processing device according to the first or second embodiment, has an evaluation unit that detects the driver's behavior, including at least one of the following, based on the captured image: whether or not the vehicle's rearview mirror is checked, whether or not the vehicle's right side mirror is checked, whether or not the vehicle's left side mirror is checked, and whether or not the driver turns their head. Based on the detected behavior, the device evaluates the driver's safety check.

According to the third embodiment, safety checks can be evaluated based on the driver's safety check actions.

The information processing device according to the fourth embodiment is an information processing device according to any one of the first to third embodiments, wherein the acquisition unit further acquires a direction indicator signal from the vehicle, and the extraction unit extracts the driving scene in which the vehicle speed, steering angle, yaw rate, and direction indicator signal acquired by the acquisition unit satisfy the extraction conditions.

According to the fourth aspect, the driving scenarios subject to safety verification evaluation can be carefully selected.

The information processing device according to the fifth embodiment, in an information processing device according to any one of the first to fourth embodiments, prohibits the evaluation of the driver's safety checks around a predetermined time before and after the start of the driving scene subject to the error information, if the driver's behavior detected from the captured image acquired by the acquisition unit includes predetermined error information.

According to the fifth aspect, the accuracy of safety confirmation evaluation can be improved.

As described above, the present invention provides an information processing device that allows the driver to understand the results of safety checks while driving a vehicle.

This figure shows the schematic configuration of the driving diagnostic device according to this embodiment. This figure shows an example of a display unit and an example of the arrangement of the display unit. This diagram shows the configuration of the diagnostic control unit of the driving diagnostic device according to this embodiment. This is a functional block diagram showing the functional configuration of the diagnostic control unit of the driving diagnostic device according to this embodiment. This diagram illustrates the conventional operational diagnostics and the operational diagnostics of this embodiment. This flowchart shows an example of the processing flow performed in the diagnostic control unit of the driving diagnostic device according to this embodiment.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. Figure 1 is a diagram showing a schematic configuration of the driving diagnostic device according to this embodiment. In this embodiment, as an example of an information processing device, a driving diagnostic device mounted on a vehicle 12 and used to evaluate the driver's safety will be described.

The driving diagnostic device 10 according to this embodiment is mounted on the vehicle 12 and performs driving diagnostics on the driver operating the vehicle 12. This embodiment describes an example of a driving diagnostic that evaluates safety checks for lane changes and right/left turns, but it may also perform driving diagnostics that evaluate other safety checks and driving operations such as stopping or reversing.

The driving diagnostic device 10 is designed to be connectable to a server 14, such as a cloud server, via a communication network 16. The diagnostic results from the driving diagnostic device 10 may be transmitted to the server 14, which then manages the driving diagnostic results for each vehicle 12 and creates driving diagnostic reports, etc.

The driving diagnostic device 10 includes a diagnostic control unit 20, a vehicle information detection unit 22, an imaging unit 24, a communication unit 26, and a display unit 28 as an example of a notification unit.

The vehicle information detection unit 22 detects vehicle information related to the vehicle 12. In this embodiment, vehicle speed, steering angle, yaw rate, and direction indicator signals are acquired as examples of vehicle information, but other vehicle information may also be detected. For example, vehicle information such as accelerator opening, vehicle 12 position information, acceleration, distance to obstacles around the vehicle, and route may be detected. Specifically, the vehicle information detection unit 22 can be fitted with multiple types of sensors and devices that acquire information representing the conditions of the surrounding environment of the vehicle 12. Examples of sensors and devices include sensors mounted on the vehicle 12 such as a vehicle speed sensor, accelerator opening sensor, steering angle sensor, yaw rate sensor, and acceleration sensor, as well as switches or sensors that detect direction indicator signals, stop lamp signals, brake hydraulic pressure, etc., a GNSS (Global Navigation Satellite System) device, an in-vehicle communication device, a navigation system, and a radar device. The GNSS device receives GNSS signals including time information from multiple GNSS satellites to determine the position of the vehicle 12. The accuracy of the positioning of the GNSS device improves as the number of receivable GNSS signals increases. The on-board communication device is a communication device that performs at least one of vehicle-to-vehicle communication with other vehicles 12 and vehicle-to-infrastructure communication with roadside devices via the communication unit 26. The navigation system includes a map information storage unit that stores map information and performs processing to display the position of the vehicle 12 on a map and to guide the vehicle to a destination based on the position information obtained from the GNSS device and the map information stored in the map information storage unit. The radar device also includes a plurality of radars with different detection ranges that detect objects such as pedestrians and other vehicles 12 that are present around the vehicle 12 and acquire the relative position and relative speed of the detected objects and the vehicle 12. The radar device also incorporates a processing unit that processes the detection results of surrounding objects. The processing unit, based on changes in relative position and relative speed of individual objects included in the most recent multiple detection results, excludes noise, roadside objects such as guardrails, etc., from the monitoring targets, and tracks and monitors pedestrians, other vehicles 12, etc. as the target objects. The radar device then outputs information such as the relative position and relative speed of each target object.

In this embodiment, the imaging unit 24 is mounted inside the vehicle and captures images of the driver, generating image data representing the captured video. The imaging unit 24 may utilize an infrared sensor to detect characteristic features of the driver, such as their eyes and face, using infrared light. Furthermore, the imaging unit 24 may also capture images of at least one area around the vehicle 12, including the front, side, and rear.

The communication unit 26 establishes communication with the server 14 via the communication network 16 and transmits and receives various types of information, such as image information obtained by the imaging unit 24 and vehicle information detected by the vehicle information detection unit 22.

The display unit 28 provides various information to the occupants by displaying information. In this embodiment, it notifies the occupants of the results of the driving diagnosis by displaying the results, and also notifies them of the driving diagnosis results by outputting an audible signal. For example, as shown in Figure 2, the display unit 28 can be a multimedia display 32 located in the center of the instrument panel 30, or a meter display 36 located on the instrument panel 30 opposite the steering wheel 34. Figure 2 shows an example of the display unit 28 and an example of its arrangement.

As shown in Figure 3, the diagnostic control unit 20 is composed of a general-purpose microcomputer including a CPU (Central Processing Unit) 20A, ROM (Read Only Memory) 20B, RAM (Random Access Memory) 20C, storage 20D, interface (I/F) 20E, and bus 20F. Figure 3 is a block diagram showing the configuration of the diagnostic control unit 20 of the driving diagnostic device 10 according to this embodiment.

Furthermore, the diagnostic control unit 20 performs a driving diagnosis by having the CPU 20A execute a program stored in the ROM 20B. This diagnosis evaluates the driver's safety checks during at least one of the driving scenarios—lane changes and right/left turns—based on the vehicle information detected by the vehicle information detection unit 22 and the images captured by the imaging unit 24 at the time of vehicle information detection.

Specifically, the diagnostic control unit 20 has the functions of an acquisition unit 40, an extraction unit 42, an evaluation unit 44, and a control unit 46, as shown in Figure 4, by having the CPU 20A execute a program stored in the ROM 20B. Figure 4 is a functional block diagram showing the functional configuration of the diagnostic control unit 20 of the driving diagnostic device 10 according to this embodiment.

The acquisition unit 40 acquires vehicle information detected by the vehicle information detection unit 22 and images captured by the imaging unit 24. In this embodiment, the acquisition unit 40 acquires vehicle speed, steering angle, yaw rate, direction indicator signal, and the imaging results from the imaging unit 24.

The extraction unit 42 extracts driving scenes during periods when the vehicle speed, steering angle, yaw rate, and direction indicator signal acquired by the acquisition unit 40 meet predetermined extraction conditions. In this embodiment, as will be described in detail later, when evaluating safety checks during lane changes, driving scenes of lane changes are extracted; and when evaluating safety checks during right or left turns, driving scenes of right or left turns are extracted.

The evaluation unit 44 evaluates the driver's safety checks based on the captured images acquired by the acquisition unit 40 at a predetermined time before and after the start of the driving scene extracted by the extraction unit 42. For example, the evaluation unit 44 detects the driver's behavior based on the captured images, including at least one of the following: whether the driver checked the rearview mirror of the vehicle 12, whether the driver checked the right side mirror of the vehicle, whether the driver checked the left side mirror of the vehicle 12, and whether the driver turned their head. Based on the detected behavior, the evaluation evaluates the driver's safety checks. This allows for the evaluation of safety checks based on the driver's safety check actions.

The control unit 46 controls the display unit 28 to notify the vehicle 12 of the safety check evaluation performed by the evaluation unit 44 while the vehicle 12 is in operation. For example, it displays the safety check evaluation results during lane changes or right/left turns on the display unit 28 and also controls the system to announce the evaluation results audibly.

Next, we will explain the details of the driving diagnostics for safety checks during lane changes and during right and left turns, using specific examples.

First, regarding the driving assessment for safety checks during lane changes, we will explain the method for extracting driving scenarios and the method for evaluating safe driving.

The acquisition unit 40 acquires vehicle speed, turn signal, steering angle, yaw rate, GPS timestamp, and images captured by the imaging unit 24 as signals necessary for driving diagnostics during lane changes to ensure safety. Note that the turn signal may be omitted, although this will reduce the accuracy of extracting driving scenes (described later). However, including the turn signal allows for the selection of driving scenes to be evaluated for safety.

The extraction unit 42 extracts lane change driving scenes, for example, using a yaw rate of 0.7 [deg/sec] or higher, or a yaw rate of -0.7 [deg/sec] or lower, as the start condition for the driving scene. Conversely, the end condition for the driving scene is a predetermined time (for example, 5 seconds) after the start point. The yaw rate is then processed using a 1.0 [kHz] LPF (Low Pass Filter).

Furthermore, lane change driving scenes are extracted using the following conditions for extracting driving scenes: the average vehicle speed in the scene is 20 km/h or higher, the turn signal is on, the minimum steering angle is -120 degrees or higher or 120 degrees or lower, the minimum yaw angle is -40 degrees or lower, or the maximum yaw angle is 40 degrees or higher, and the minimum yaw angle is -100 degrees or higher, or the maximum yaw angle is 100 degrees or lower.

The evaluation unit 44 then detects from the captured images whether the driver checked the rearview mirror, driver's side door mirror, passenger's side door mirror, and performed head movements during a predetermined time (e.g., 4 seconds) before the start of the driving scene extracted by the extraction unit 42, and a predetermined time (e.g., 5 seconds) after the start. It then diagnoses whether safety checks were performed before changing lanes. For example, the evaluation unit calculates the driver's head movement coordinates, eye movement coordinates (up/down and left/right) from the captured images to diagnose whether safety checks such as checking the rearview mirror, driver's side door mirror, passenger's side door mirror, and head movements were performed.

If none of these pre-lane change safety checks are performed, the relevant driving scene will be extracted. However, if the driver's behavior includes an error signal containing predetermined error information (e.g., undetectable), the diagnosis will be prohibited as it would result in a misdiagnosis.

Next, we will explain the method for extracting driving scenarios and evaluating safety checks during right and left turns as part of the driving diagnostic process.

The acquisition unit 40 acquires vehicle speed, turn signal signals, steering angle, yaw rate, GPS timestamp, and images captured by the imaging unit 24 as signals necessary for driving diagnostics during safe turns.

The extraction unit 42 extracts driving scenes involving right and left turns. For example, the start condition for a driving scene is a yaw rate of 0.7 [deg/sec] or higher, or a yaw rate of -0.7 [deg/sec] or lower. Conversely, the end condition for a driving scene is a predetermined time (e.g., 5 seconds) after the start point. The yaw rate is then processed using a 1.0 [kHz] LPF (Low Pass Filter).

Furthermore, driving scenes involving right or left turns are extracted using the following conditions: the maximum yaw rate is 0.8 [deg/sec] or greater, the minimum yaw rate is -0.8 [deg/sec] or less, the maximum vehicle speed in the scene is 15 [km/h] or greater, the turn signal is on, and the absolute value of the maximum steering angle is 150 [deg] or greater.

The evaluation unit 44 then detects from the captured images whether the driver checked the rearview mirror, driver's side door mirror, passenger's side door mirror, and performed head movements during a predetermined time (e.g., 4 seconds) before the start of the driving scene extracted by the extraction unit 42, and a predetermined time (e.g., 5 seconds) after the start. It then diagnoses whether safety checks were performed before turning left or right. For example, the evaluation unit calculates the driver's head movement coordinates, eye movement coordinates (up/down and left/right) from the captured images to diagnose whether safety checks such as checking the rearview mirror, driver's side door mirror, passenger's side door mirror, and head movements were performed.

If none of these safety checks are performed before turning left or right, the driving scene will be extracted as an example of insufficient safety checks.

Furthermore, if the driver's behavior includes predetermined error information (error signals such as "undetectable"), the diagnosis will be prohibited as this would result in a misdiagnosis. Specifically, if the captured image acquired by the acquisition unit 40 contains predetermined error information, the evaluation unit 44 prohibits the evaluation of the driver's safety checks around a predetermined time before and after the start of the driving scene affected by the error information. This improves the accuracy of the safety check evaluation.

Incidentally, in conventional driving diagnostics, as shown in the upper part of Figure 5, driving diagnostics were performed and reports were created at predetermined intervals, such as 10 minutes. Therefore, when actual safety checks were insufficient, feedback could not be given to the driver. Even when feedback was given, the driver might not remember or understand what went wrong.

Therefore, the driving diagnostic device 10 according to this embodiment performs driving diagnostics in real time and provides feedback to the driver.

To perform real-time diagnosis, as shown in the lower part of Figure 5, the driving diagnosis is performed by overlapping data from a predetermined time period (e.g., 10 seconds) with a predetermined time interval (e.g., 1 second). Specifically, the diagnostic control unit 20 extracts driving scenes such as lane changes and right/left turns from captured images and vehicle information for a predetermined diagnostic time period longer than the first hour (e.g., 10 seconds) at predetermined intervals (e.g., 1 second), and performs a driving diagnosis regarding safety checks. Since overlapping diagnoses may result in the same diagnosis being performed, the earlier diagnosis result is prioritized in the output of duplicate diagnostic results.

Next, we will describe the specific processes performed by the operation diagnostic device 10 configured according to this embodiment as described above. Figure 6 is a flowchart showing an example of the processing flow performed by the diagnostic control unit 20 of the operation diagnostic device 10 according to this embodiment. Note that the processing in Figure 6 starts, for example, every 1 [sec] as the first time period.

In step 100, the CPU 20A acquires vehicle information and captured images, then proceeds to step 102. Specifically, the acquisition unit 40 acquires the vehicle information detected by the vehicle information detection unit 22 and the captured images taken by the image capture unit 24. In this embodiment, the acquisition unit 40 acquires vehicle speed, steering angle, yaw rate, and the image capture results from the image capture unit 24.

In step 102, the CPU 20A extracts driving scenes and proceeds to step 104. Specifically, the extraction unit 42 extracts driving scenes during periods when the vehicle speed, steering angle, and yaw rate acquired by the acquisition unit 40 satisfy predetermined extraction conditions. In this embodiment, when evaluating safety checks during lane changes, driving scenes of lane changes are extracted according to the conditions described above. Similarly, when evaluating safety checks during right and left turns, driving scenes of right and left turns are extracted according to the conditions described above.

In step 104, the CPU 20A makes a safety check and proceeds to step 106. Specifically, the evaluation unit 44 evaluates the driver's safety check based on the captured images acquired by the acquisition unit 40 at a predetermined time before and after the start of the driving scene extracted by the extraction unit 42. For example, the evaluation unit 44 detects the driver's behavior based on the captured images, including at least one of the following: whether the driver checked the rearview mirror of the vehicle 12, whether the driver checked the right side mirror of the vehicle, whether the driver checked the left side mirror of the vehicle 12, and whether the driver turned their head. Based on the detected behavior, the evaluation unit 44 evaluates the driver's safety check.

In step 106, the CPU 20A determines whether or not there is insufficient safety check. If the determination is negative, the process proceeds to step 108; if it is positive, the process proceeds to step 110.

In step 108, the CPU 20A terminates the series of processes by notifying the driver that safe driving has occurred. Specifically, the control unit 46 performs control to notify the driver of the safety confirmation diagnosis result while the vehicle 12 is in operation by displaying a message indicating safe driving on the display unit 28, as part of the safety confirmation evaluation by the evaluation unit 44. For example, at least one of a message and an icon image indicating safe driving is displayed on the display unit 28, and an audio output is also produced.

On the other hand, in step 110, the CPU 20A notifies the driver of insufficient safety checks and terminates the series of processes. Specifically, the control unit 46 performs control to inform the driver of the safety check diagnosis result by displaying on the display unit 28 that there were insufficient safety checks, as evaluated by the evaluation unit 44, while the vehicle 12 is in operation. For example, at least one of a message and an icon image indicating insufficient safety checks is displayed on the display unit 28, and an audio output is also produced.

By processing the data in this way, the driver is notified of the evaluation of their safety checks, making it possible for the driver to be aware of the safety check evaluation results while driving the vehicle.
Furthermore, by performing driving diagnostics with predetermined time intervals and accumulating data for predetermined time periods, it is possible to evaluate the driver's deceleration operations in real time and inform the driver.

Furthermore, by performing driving diagnostics with predetermined time intervals and accumulating data for predetermined time periods, it is possible to evaluate the driver's deceleration operations in real time and inform the driver.

In the above embodiment, driving diagnostics are performed to check for safety during right and left turns and lane changes, but the driving scenes are not limited to these two. For example, driving diagnostics for safety checks in other driving scenes such as when reversing or when coming to a complete stop, as well as driving diagnostics for accelerator operation and driving diagnostics for brake operation, may also be performed. Furthermore, when driving diagnostics are performed for multiple driving scenes, driving evaluations may be performed at the same time. In this case, the control unit 46 may notify the driver's driving evaluation, including the evaluation of safety checks, from the display unit 28 at predetermined notification timings, and if the evaluation unit 44 has performed an evaluation of safety checks, it may be controlled to notify the evaluation of safety checks from the display unit 28 at the next notification timing. For example, if, within a certain diagnostic time (e.g., 10 [sec]), driving evaluations include an evaluation of accelerator operation and a driving evaluation of safety checks when turning right or left, the diagnostic result that takes priority according to a predetermined priority (e.g., the driving evaluation of safety checks when turning right or left) is notified at the notification timing when the diagnostic time has elapsed. This allows drivers to understand driving evaluations, such as safety checks, at an early stage while the vehicle is in operation.

Furthermore, while the above embodiment describes an example where driving diagnostics are performed using a driving diagnostic device 10 mounted on the vehicle 12, the invention is not limited to this. For example, the in-vehicle device may be made communicable with an information processing terminal such as a smartphone, and the functions of the diagnostic control unit 20 may be provided on the information processing terminal. Alternatively, vehicle information and captured images may be transmitted to the server 14 to perform driving diagnostics, and the diagnostic results may be transmitted to the diagnostic control unit 20 for display on the display unit 28.

Furthermore, while the processing performed by the diagnostic control unit 20 in each of the above embodiments has been described as software processing performed by executing a program, it is not limited to this. For example, it may be processing performed by hardware such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). Alternatively, it may be processing that combines both software and hardware. Also, if it is software processing, the program may be stored on various storage media and distributed.

Furthermore, the present invention is not limited to the above, and can be implemented in various modifications without departing from its spirit.

10. Operational diagnostic device (information processing device)
12 Vehicle 20 Diagnostic control unit 22 Vehicle information detection unit 24 Imaging unit 28 Display unit (notification unit)
32 Multimedia display 36 Meter display 40 Acquisition unit 42 Extraction unit 44 Evaluation unit 46 Control unit

Claims (5)

An acquisition unit that acquires vehicle speed, steering angle, yaw rate, and a captured image of the vehicle's driver from the vehicle,
An extraction unit extracts driving scenes during a period in which the vehicle speed, steering angle, and yaw rate acquired by the acquisition unit satisfy predetermined extraction conditions.
An evaluation unit evaluates the driver's safety check based on the captured images acquired by the acquisition unit at a predetermined time before and after the start of the driving scene extracted by the extraction unit,
A control unit that controls the notification of the safety confirmation evaluation by the evaluation unit from the notification unit while the vehicle is in operation,
Equipped with ,
The evaluation unit performs the evaluation at predetermined intervals of a predetermined first hour, based on captured images and vehicle information for a predetermined evaluation time longer than the first hour, duplicating the data for the evaluation time, and outputs the earlier evaluation result first in case of duplicate evaluation results . The information processing apparatus according to claim 1, wherein the control unit notifies the driver's driving evaluation, including the safety check evaluation, from the notification unit at predetermined intervals, and when the evaluation unit performs the safety check evaluation, the notification unit notifies the safety check evaluation at the next notification timing. The information processing device according to claim 1, wherein the evaluation unit detects the driver's behavior, including at least one of the following based on the captured image: whether or not the vehicle's rearview mirror is checked, whether or not the vehicle's right side mirror is checked, whether or not the vehicle's left side mirror is checked, and whether or not the driver turns their head; and evaluates the driver's safety check based on the detected behavior. The information processing apparatus according to claim 1, wherein the acquisition unit further acquires a direction indicator signal from the vehicle, and the extraction unit extracts the driving scene in which the vehicle speed, steering angle, yaw rate, and direction indicator signal acquired by the acquisition unit satisfy the extraction conditions. The information processing device according to claim 1, wherein the evaluation unit, if the driver's behavior detected from the captured image acquired by the acquisition unit includes predetermined error information, prohibits the evaluation of the driver's safety checks around the predetermined time before and after the start of the driving scene subject to the error information.