JP7697482B2 - Driver monitoring device, driver monitoring method, and computer program for driver monitoring - Google Patents

Description

The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring.

Patent Document 1 describes how, in a vehicle equipped with an autonomous driving function, if it is estimated that the driver is continuously performing actions other than those permitted when the autonomous driving function is operating, an information presentation device is caused to issue a warning to the driver.

Patent document 2 describes that when determining whether or not a driver is in a state of difficulty in driving, if an abnormality is detected in at least one of the driver's driving operation and the vehicle's running state in addition to an abnormality in the driver's condition, the driver is determined to be in a state of difficulty in driving.

Patent document 3 describes detecting the driver's state and changing the timing of notifying the driver of a change in the vehicle's driving mode (e.g., from an automatic driving mode to a manual driving mode) depending on the driver's state.

Patent document 4 describes a system that monitors the driver's line of sight and state, and when it detects that the driver is looking at a device such as a smartphone when requesting the driver to switch from an automated driving control mode to a manual driving mode, stops the function of the device.

JP 2021-152888 A JP 2019-166968 A International Publication No. 2017/085981 JP 2021-049891 A

When a vehicle achieves Level 3 autonomous driving, the driver will be permitted to operate a device such as a smartphone if they are able to take over driving operations upon request from the system. However, actions such as falling asleep at the wheel that would prevent the driver from quickly taking over driving operations will be prohibited.

For this reason, even when Level 3 autonomous driving is implemented, it is necessary to monitor the driver's condition using an imaging device installed in the vehicle to capture an image of the driver. However, based on the image generated by such an imaging device, it is difficult to distinguish between a state in which the driver is operating a terminal and an abnormal posture due to the driver falling asleep or suddenly becoming ill. For this reason, when the driver is operating a terminal, there is a risk of erroneously determining that the driver is not in a driving standby state in which he or she can perform driving operations.

In view of the above problems, the object of the present invention is to prevent the driver from erroneously determining that the vehicle is not in a driving standby state in which the driver can perform driving operations.

The gist of this disclosure is as follows:

(1) A driver monitoring device having a determination unit that determines whether or not a driver of a vehicle is in a driving standby state in which the driver can perform driving operations, and a notification unit that notifies the driver of a warning when it is determined that the driver is not in the driving standby state, the determination unit acquiring an image generated by an imaging device provided in the vehicle to capture an image of the driver and driver information acquired by a terminal operable by the driver, and judging whether or not the driver is in the driving standby state based on the image and the driver information.

(2) The driver monitoring device described in (1) above, in which the driver information includes an image generated by a camera mounted on the terminal or information input by the driver to the terminal.

(3) The driver monitoring device according to (1) or (2) above, wherein the determination unit determines whether the driver is in the driving standby state based on the image using a predetermined determination criterion, determines whether the driver is in an awake state based on the driver information, and relaxes the determination criterion when it is determined that the driver is in an awake state.

(4) The driver monitoring device according to (1) or (2) above, wherein the determination unit determines whether the driver is in the standby state based on the image using a predetermined determination criterion, and if it is determined that the driver is not in the standby state based on the image, determines whether the driver is in an awake state based on the driver information, and if it is determined that the driver is in an awake state, invalidates the determination result based on the image.

(5) The driver monitoring device according to any one of (1) to (4) above, wherein the determination unit acquires the driver information from the terminal when it determines based on the image that the driver is not in the standby state, and does not acquire the driver information from the terminal when it determines based on the image that the driver is in the standby state.

(6) The driver monitoring device according to any one of (1) to (5) above, further comprising a terminal control unit that limits or prohibits the driver from operating the terminal when autonomous driving of level 2 or lower is being performed in the vehicle.

(7) The driver monitoring device according to any one of (1) to (6) above, wherein the terminal is a smartphone.

(8) A driver monitoring method executed by a computer, the driver monitoring method including: acquiring an image generated by an imaging device provided in a vehicle to capture an image of the driver of the vehicle; acquiring driver information acquired by a terminal operable by the driver; determining whether the driver is in a driving standby state in which the driver can perform driving operations based on the image and the driver information; and notifying the driver of a warning when it is determined that the driver is not in the driving standby state.

(9) A computer program for driver monitoring that causes a computer to execute the following: acquiring an image generated by an imaging device provided in a vehicle to capture an image of the driver of the vehicle; acquiring driver information acquired by a terminal operable by the driver; determining whether the driver is in a driving standby state in which he or she can perform driving operations based on the image and the driver information; and notifying the driver of a warning when it is determined that the driver is not in the driving standby state.

The present invention can prevent the driver from erroneously determining that the vehicle is not in a driving standby state in which the driver can perform driving operations.

FIG. 1 is a schematic configuration diagram of a vehicle control system including a driver monitoring device according to a first embodiment of the present invention. FIG. 2 is a diagram showing a schematic view of the interior of a vehicle equipped with a driver monitor camera. FIG. 3 is a functional block diagram of a processor of the ECU in the first embodiment. FIG. 4 is a flowchart showing a control routine for the judgment criterion setting process in the first embodiment of the present invention. FIG. 5 is a flowchart showing a control routine for a warning process in the first embodiment of the present invention. FIG. 6 is a flowchart showing a control routine for a warning process in the second embodiment of the present invention. FIG. 7 is a functional block diagram of a processor of an ECU in the third embodiment. FIG. 8 is a flowchart showing a control routine of a terminal operation determination process in the third embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the drawings. In the following description, similar components are given the same reference numbers.

First Embodiment
A first embodiment of the present invention will be described below with reference to Figures 1 to 5. Figure 1 is a schematic configuration diagram of a vehicle control system 1 including a driver monitoring device according to a first embodiment of the present invention. The vehicle control system 1 is mounted on a vehicle and executes various controls of the vehicle.

As shown in FIG. 1, the vehicle control system 1 includes a driver monitor camera 2, a surrounding information detection device 3, a Global Navigation Satellite System (GNSS) receiver 4, a map database 5, a navigation device 6, a vehicle behavior detection device 7, an actuator 8, a human machine interface (HMI) 9, a communication device 10, and an electronic control unit (ECU) 20. The driver monitor camera 2, the surrounding information detection device 3, the GNSS receiver 4, the map database 5, the navigation device 6, the vehicle behavior detection device 7, the actuator 8, the HMI 9, and the communication device 10 are electrically connected to the ECU 20 via an in-vehicle network or the like that complies with standards such as the Controller Area Network (CAN).

The driver monitor camera 2 captures an image of the vehicle driver and generates an image showing the driver. The output of the driver monitor camera 2, i.e., the image generated by the driver monitor camera 2, is transmitted to the ECU 20. A specific example of the configuration of the driver monitor camera 2 is described below.

The driver monitor camera 2 has a camera and a floodlight. The camera is composed of a lens and an image sensor, for example a CMOS (complementary metal oxide semiconductor) camera or a CCD (charge-coupled device) camera. The floodlight is an LED (light-emitting diode), for example two near-infrared LEDs arranged on both sides of the camera. By irradiating the driver with near-infrared light, the driver's face can be photographed without causing discomfort to the driver even in low-light conditions such as at night. In addition, a bandpass filter that removes light with wavelength components other than near-infrared may be provided inside the camera, and a visible light cut filter that removes light with red wavelength components irradiated from the near-infrared LED may be provided in front of the floodlight.

Figure 2 is a diagram showing the interior of a vehicle 30 equipped with a driver monitor camera 2. The driver monitor camera 2 is installed in the cabin of the vehicle 30 to capture an image of the driver of the vehicle 30. For example, as shown in Figure 2, the driver monitor camera 2 is installed on the upper part of the steering column 31 of the vehicle 30. In Figure 2, the projection range of the driver monitor camera 2 is shown by a dashed line. In this embodiment, the driver monitor camera 2 captures the face of the driver. The driver monitor camera 2 may be installed on the steering wheel 32, rearview mirror, meter panel, meter hood, etc. of the vehicle 30. The driver monitor camera 2 is an example of an imaging device installed in the vehicle to capture an image of the driver.

The surrounding information detection device 3 acquires data (images, point cloud data, etc.) about the surroundings of the vehicle 30 and detects surrounding information about the vehicle 30 (e.g., surrounding vehicles, lanes, pedestrians, bicycles, traffic lights, signs, etc.). For example, the surrounding information detection device 3 includes a camera (monocular camera or stereo camera), a millimeter wave radar, a lidar (LIDAR: Laser Imaging Detection And Ranging), or an ultrasonic sensor (sonar), or any combination of these. The surrounding information detection device 3 may further include an illuminance sensor, a rain sensor, etc. The output of the surrounding information detection device 3, i.e., the surrounding information about the vehicle 30 detected by the surrounding information detection device 3, is transmitted to the ECU 20.

The GNSS receiver 4 detects the current position of the vehicle 30 (e.g., the latitude and longitude of the vehicle 30) based on positioning information obtained from multiple (e.g., three or more) positioning satellites. Specifically, the GNSS receiver 4 captures multiple positioning satellites and receives radio waves transmitted from the positioning satellites. The GNSS receiver 4 then calculates the distance to the positioning satellite based on the difference between the transmission time and reception time of the radio waves, and detects the current position of the vehicle 30 based on the distance to the positioning satellite and the position (orbit information) of the positioning satellite. The output of the GNSS receiver 4, i.e., the current position of the vehicle 30 detected by the GNSS receiver 4, is transmitted to the ECU 20. A GPS (Global Positioning System) receiver is an example of a GNSS receiver.

The map database 5 stores map information. The ECU 20 acquires the map information from the map database 5. Note that the map database may be provided outside the vehicle 30 (e.g., a server, etc.), and the ECU 20 may acquire the map information from outside the vehicle 30.

The navigation device 6 sets a driving route for the vehicle 30 to the destination based on the current position of the vehicle 30 detected by the GNSS receiver 4, the map information in the map database 5, input by an occupant (e.g., the driver) of the vehicle 30, etc. The driving route set by the navigation device 6 is transmitted to the ECU 20.

The vehicle behavior detection device 7 detects behavior information of the vehicle 30. The vehicle behavior detection device 7 includes, for example, a vehicle speed sensor that detects the speed of the vehicle 30, a yaw rate sensor that detects the yaw rate of the vehicle 30, etc. The output of the vehicle behavior detection device 7, i.e., the vehicle behavior information detected by the vehicle behavior detection device 7, is transmitted to the ECU 20.

The actuators 8 operate the vehicle 30. For example, the actuators 8 include a drive device (e.g., at least one of an internal combustion engine and an electric motor) for accelerating the vehicle 30, a brake actuator for braking (deceleration) the vehicle 30, a steering actuator for steering the vehicle 30, etc. The ECU 20 controls the actuators 8 to control the behavior of the vehicle 30.

The HMI 9 transmits information between the vehicle 30 and an occupant of the vehicle 30 (e.g., the driver). The HMI 9 has an output section (e.g., a display, a speaker, a light source, a vibration unit, etc.) that provides information to the occupant of the vehicle 30, and an input section (e.g., a touch panel, an operation button, an operation switch, a microphone, etc.) into which information is input by the occupant of the vehicle 30. The output of the ECU 20 is notified to the occupant of the vehicle 30 via the HMI 9, and the input from the occupant of the vehicle 30 is transmitted to the ECU 20 via the HMI 9. The HMI 9 is an example of an input device, an output device, or an input/output device.

The communication device 10 is capable of communicating with the outside of the vehicle 30, and enables communication between the vehicle 30 and the outside of the vehicle 30. For example, the communication device 10 includes a wide-area wireless communication module that enables wide-area wireless communication between the vehicle 30 and the outside of the vehicle 30 (e.g., a server) via a communication network such as a carrier network or the Internet network, and a short-range wireless communication module that enables short-range wireless communication based on a communication standard such as Bluetooth (registered trademark) or ZigBee (registered trademark).

On the other hand, the terminal 11 shown in FIG. 1 is carried by the driver of the vehicle 30. The terminal 11 is brought into the vehicle 30 by the driver and can be operated by the driver. The terminal 11 is paired with the vehicle 30 in advance, and the ECU 20 can communicate with the terminal 11 via a short-range wireless communication module of the communication device 10. In this embodiment, the terminal 11 is a smartphone. Note that the terminal 11 may be electrically connected to the ECU 20 by a wire.

The ECU 20 executes various controls of the vehicle. As shown in FIG. 1, the ECU 20 includes a communication interface 21, a memory 22, and a processor 23. The communication interface 21 and the memory 22 are connected to the processor 23 via signal lines. In this embodiment, one ECU 20 is provided, but multiple ECUs may be provided for each function.

The communication interface 21 has an interface circuit for connecting the ECU 20 to an in-vehicle network. The ECU 20 is connected to other in-vehicle devices via the communication interface 21.

The memory 22 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores programs, data, etc. that are used when various processes are executed by the processor 23.

The processor 23 has one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further have an arithmetic circuit such as a logical arithmetic unit or a numerical arithmetic unit.

The vehicle control system 1 functions as an autonomous driving system and realizes autonomous driving of the vehicle 30. In this embodiment, the vehicle control system 1 is capable of implementing level 3 autonomous driving in a pre-set operational design domain (ODD). Note that the autonomous driving levels in this specification are based on the definitions in SAE (Society of Automotive Engineers) J3016.

In situations outside the operation design area, the vehicle control system 1 performs autonomous driving of level 2 or lower. In autonomous driving of level 1 or level 2, the vehicle control system 1 activates driving assistance functions such as adaptive cruise control (ACC), which automatically controls the vehicle speed depending on whether there is a preceding vehicle, and lane keeping assist (LKA) or lane tracing assist (LTA), which automatically controls the steering of the vehicle so that the vehicle remains within the lane. Autonomous driving of level 0 corresponds to manual driving in which the acceleration, deceleration (braking), and steering of the vehicle 30 are all performed by the driver.

On the other hand, in level 3 autonomous driving, the system takes the initiative in driving the vehicle 30, and the driver is freed from the obligation to monitor the surroundings. Therefore, when level 3 autonomous driving is implemented in the vehicle 30, the driver is permitted to operate the terminal 11 if the driver is in a state where he or she can take over driving operations in response to a request from the system. On the other hand, actions such as falling asleep at the wheel that prevent the driver from quickly taking over driving operations are prohibited. For this reason, even when level 3 autonomous driving is implemented, it is necessary to monitor the driver's condition using an imaging device such as the driver monitor camera 2.

In this embodiment, the ECU 20 provided in the vehicle 30 functions as a driver monitoring device that monitors the driver of the vehicle 30. FIG. 3 is a functional block diagram of the processor 23 of the ECU 20 in the first embodiment. In this embodiment, the processor 23 has a determination unit 25 and a notification unit 26. The determination unit 25 and the notification unit 26 are functional modules that are realized by the processor 23 of the ECU 20 executing a computer program stored in the memory 22 of the ECU 20. Note that each of these functional modules may be realized by a dedicated arithmetic circuit provided in the processor 23.

The determination unit 25 determines whether the driver of the vehicle 30 is in a standby state in which the driver can perform driving operations. In particular, in this embodiment, the determination unit 25 determines whether the driver is in a standby state when level 3 autonomous driving is being performed in the vehicle 30. For example, the determination unit 25 determines that the driver is not in a standby state when an abnormal posture due to the driver falling asleep, leaving the seat, or sudden illness is detected. The notification unit 26 notifies the driver of a warning when the determination unit 25 determines that the driver is not in a standby state. This makes it possible to prompt the driver to return to a standby state, thereby improving the safety of the autonomous driving of the vehicle 30.

The determination unit 25 acquires an image (hereinafter referred to as a "monitoring image") generated by the driver monitor camera 2, and determines whether or not the driver is in a standby state based on the monitoring image. As described above, in level 3 autonomous driving, the driver can operate the terminal 11 while the vehicle 30 is traveling as a legitimate action. However, it is difficult to distinguish, based on the monitoring image, between a state in which the driver is operating the terminal 11 and an abnormal posture due to the driver falling asleep or suddenly becoming ill. On the other hand, the terminal 11 can acquire information other than the monitoring image as information about the driver.

Therefore, the determination unit 25 obtains driver information obtained by the terminal 11 in addition to the monitoring image, and determines whether the driver is in a standby state based on the monitoring image and the driver information. This makes it possible to prevent erroneous determination that the driver is not in a standby state.

In this embodiment, the determination unit 25 determines whether the driver is in a standby state based on a monitoring image using a predetermined determination criterion. The determination unit 25 also determines whether the driver is in an awake state based on the driver information acquired by the terminal 11, and relaxes the determination criterion when it is determined that the driver is in an awake state. This makes it possible to prevent a driver who is operating the terminal 11 as a legitimate act from being erroneously determined not to be in a standby state.

The above-mentioned control will be described in detail below with reference to Figures 4 and 5. Figure 4 is a flowchart showing a control routine for the judgment criterion setting process in the first embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20.

First, in step S101, the determination unit 25 of the processor 23 determines whether or not level 3 autonomous driving is being implemented in the vehicle 30. Level 3 autonomous driving is implemented when the driver requests the operation of autonomous driving in a preset operation design area. If it is determined that level 3 autonomous driving is not being implemented, this control routine ends. On the other hand, if it is determined that level 3 autonomous driving is being implemented, this control routine proceeds to step S102.

In step S102, the determination unit 25 acquires driver information from the terminal 11 by short-range wireless communication or the like. The driver information is information about the driver acquired by the terminal 11, and includes, for example, an image generated by a camera (e.g., an in-camera) mounted on the terminal 11, or information input by the driver to the terminal 11 (e.g., information input by the driver's finger or voice).

Next, in step S103, the determination unit 25 determines whether the driver is awake based on the driver information. For example, the determination unit 25 determines whether the driver is awake based on an image generated by a camera of the terminal 11 (hereinafter referred to as a "camera image"). In this case, for example, the determination unit 25 calculates the degree of eye opening of the driver based on the camera image, and determines that the driver is awake if the eye opening degree is equal to or greater than a predetermined value, and determines that the driver is not awake if the eye opening degree is less than the predetermined value. Furthermore, if the driver's eyes are not detected from the camera image, the determination unit 25 determines that it is unclear whether the driver is awake.

The determination unit 25 may determine whether the driver is awake based on input information to the terminal 11 by the driver. In this case, the determination unit 25 determines that the driver is awake when the driver inputs (touch input, button input, voice input, etc.) to the terminal 11 within a predetermined time, and determines that it is unclear whether the driver is awake when the driver does not input to the terminal 11 within the predetermined time. The determination unit 25 may also determine whether the driver is awake based on a camera image and input information to the terminal 11 by the driver. In this case, for example, the determination unit 25 determines that the driver is awake when the degree of eye opening of the driver is equal to or greater than a predetermined value and the driver inputs to the terminal 11 within the predetermined time. The determination unit 25 may also determine whether the driver is awake based on the output of an acceleration sensor mounted on the terminal 11.

If it is determined in step S103 that the driver is not in an awake state, or if it is determined that it is unclear whether the driver is in an awake state, this control routine proceeds to step S104. In step S104, the determination unit 25 initializes a determination criterion for determining whether the driver is in a driving standby state. That is, the determination unit 25 sets the determination criterion to a preset condition. The determination criterion is, for example, a threshold range for the driver's facial orientation (up-down and left-right directions). After step S104, this control routine ends.

On the other hand, if it is determined in step S103 that the driver is in an awake state, the control routine proceeds to step S105. In step S105, the determination unit 25 relaxes the determination criteria for determining whether the driver is in a standby state. That is, the determination unit 25 changes the determination criteria so that it is easier to determine that the driver is in a standby state. When the driver operates the terminal 11 on his/her lap, the driver's face faces downward. For this reason, for example, the determination unit 25 relaxes the determination criteria by increasing the threshold value of the downward direction of the face direction. In addition, in the monitoring image, the driver's eyes may be blocked by the driver's hand or arm holding the terminal 11 or the terminal 11. For this reason, the determination unit 25 may relax the determination criteria so that the driver is determined to be in a standby state as long as the driver's face is detected from the monitoring image. After step S105, the control routine ends.

Figure 5 is a flowchart showing a control routine for warning processing in the first embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20.

First, in step S201, similar to step S101 in FIG. 4, the determination unit 25 of the processor 23 determines whether or not level 3 autonomous driving is being performed in the vehicle 30. If it is determined that level 3 autonomous driving is not being performed, this control routine ends. On the other hand, if it is determined that level 3 autonomous driving is being performed, this control routine proceeds to step S202.

In step S202, the determination unit 25 acquires a monitoring image. The monitoring image is repeatedly generated by the driver monitor camera 2 at a predetermined shooting period (for example, 1/30 to 1/10 seconds), and the determination unit 25 acquires the monitoring image from the driver monitor camera 2.

Next, in step S203, the determination unit 25 determines whether the driver is in a standby state based on the monitoring image using a predetermined determination criterion. For example, the determination unit 25 detects the driver's facial direction from the monitoring image, and determines that the driver is in a standby state when the driver's facial direction is within a predetermined threshold range, and determines that the driver is not in a standby state when the driver's facial direction is outside the predetermined threshold range. At this time, the determination criterion set in step S104 or S105 is used. In other words, if the driver is determined to be in an awake state, the determination criterion is relaxed.

If it is determined in step S203 that the driver is in a standby state, this control routine ends. On the other hand, if it is determined in step S203 that the driver is not in a standby state, this control routine proceeds to step S204.

In step S204, the notification unit 26 of the processor 23 notifies the driver of a visual, auditory, or tactile warning via the HMI 9. An example of a visual warning is a warning light emitted from a light source of the HMI 9. An example of an auditory warning is a warning voice or warning sound output from a speaker of the HMI 9. An example of a tactile warning is a vibration output from a vibration unit of the HMI 9 (e.g., vibration of the steering wheel 32 or seat belt). The notification unit 26 may notify the driver of two or more types of warnings (e.g., a visual warning and an auditory warning). After step S204, this control routine ends.

The control routine of FIG. 4 may be omitted, and steps S102 to S105 of FIG. 4 may be executed between steps S201 and S202 of FIG. 5. The determination unit 25 may acquire driver information from the terminal 11 and determine whether the driver is awake or not based on the driver information only when it is determined that the driver is not in a standby state based on the monitoring image. In this case, the determination unit 25 relaxes the determination criteria when it is determined that the driver is in an awake state, and uses the relaxed determination criteria to determine again whether the driver is in a standby state or not based on the monitoring image. That is, the control routine of FIG. 4 may be omitted, and steps S102, S103, and S105 of FIG. 4 and step S203 of FIG. 5 may be executed between steps S203 and S204 of FIG. 5. By acquiring driver information from the terminal 11 only when it is determined that the driver is not in a standby state, the amount of power consumed by the terminal 11 and the ECU 20 can be reduced.

Second Embodiment
The driver monitoring device according to the second embodiment is basically similar in configuration and control to the driver monitoring device according to the first embodiment, except for the points described below. Therefore, the second embodiment of the present invention will be described below, focusing on the differences from the first embodiment.

In the second embodiment, the determination unit 25 determines whether the driver is in a standby state based on the monitoring image using a predetermined determination criterion, and if it is determined that the driver is not in a standby state based on the monitoring image, it determines whether the driver is in an awake state based on the driver information acquired by the terminal 11. If the determination unit 25 determines that the driver is in an awake state, it invalidates the determination result based on the monitoring image and determines that the driver is in a standby state. This makes it possible to prevent a driver who is operating the terminal 11 as a legitimate act from being erroneously determined to be not in a standby state.

In addition, when the determination unit 25 determines based on the monitoring image that the driver is not in a standby state, it acquires driver information from the terminal 11, and when the determination unit 25 determines based on the monitoring image that the driver is in a standby state, it does not acquire driver information from the terminal 11. This makes it possible to reduce the amount of power consumed by the terminal 11 and the ECU 20.

The above-mentioned control will be described in detail below with reference to FIG. 6. FIG. 6 is a flowchart showing a control routine for warning processing in the second embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20.

Steps S301 to S303 are executed in the same manner as steps S201 to S203 in FIG. 5, and in step S303, it is determined based on the monitored image whether the driver is in a standby state, using the initialized criteria. If it is determined in step S303 that the driver is not in a standby state, this control routine proceeds to step S304.

In step S304, similar to step S102 in FIG. 4, the determination unit 25 acquires driver information from the terminal 11. Next, in step S305, similar to step S103 in FIG. 4, the determination unit 25 determines whether the driver is awake or not based on the driver information.

If it is determined in step S305 that the driver is in an awake state, the control routine proceeds to step S306. In step S306, the determination unit 25 invalidates the determination result based on the monitoring image. In other words, the determination unit 25 determines that the driver is in a standby state. After step S306, the control routine ends.

On the other hand, if it is determined in step S305 that the driver is not awake, the control routine proceeds to step S307. In this case, the determination result based on the monitoring image is maintained. Therefore, in step S307, similar to step S204 in FIG. 5, the notification unit 26 notifies the driver of a warning via the HMI 9. After step S307, the control routine ends. Note that step S304 may be executed between steps S301 and S303.

Third Embodiment
The driver monitoring device according to the third embodiment is basically similar in configuration and control to the driver monitoring device according to the first embodiment, except for the points described below. Therefore, the third embodiment of the present invention will be described below, focusing on the differences from the first embodiment.

Figure 7 is a functional block diagram of the processor 23 of the ECU 20 in the third embodiment. In the third embodiment, the processor 23 has a terminal control unit 27 in addition to the determination unit 25 and notification unit 26. The determination unit 25, notification unit 26, and terminal control unit 27 are functional modules that are realized by the processor 23 of the ECU 20 executing a computer program stored in the memory 22 of the ECU 20. Note that each of these functional modules may be realized by a dedicated arithmetic circuit provided in the processor 23.

The terminal control unit 27 controls the terminal 11 by sending control signals to the terminal 11. In autonomous driving at level 3, the driver is permitted to operate the terminal 11, but in autonomous driving at level 2 or lower, the driver is generally not permitted to operate the terminal 11. However, in autonomous driving at level 2 or lower, the driver may feel the urge to operate the terminal 11. This is particularly noticeable when the driver is operating the terminal 11 in autonomous driving at level 3.

Therefore, in the third embodiment, the terminal control unit 27 restricts the driver's operation of the terminal 11 when autonomous driving of level 2 or lower is being performed in the vehicle 30. This makes it possible to prevent the driver's attention from being distracted during autonomous driving of level 2 or lower.

In the third embodiment, in addition to the control routines of Figs. 4 and 5, a control routine of Fig. 8 is executed. Fig. 8 is a flowchart showing a control routine of a terminal operation determination process in the third embodiment of the present invention. This control routine is repeatedly executed at a predetermined execution interval by the processor 23 of the ECU 20.

First, in step S401, the terminal control unit 27 of the processor 23 determines whether or not level 3 autonomous driving is being implemented in the vehicle 30. If it is determined that level 3 autonomous driving is being implemented, this control routine proceeds to step S402.

In step S402, the terminal control unit 27 permits the driver to operate the terminal 11. At this time, the terminal control unit 27 may transmit a control signal to the terminal 11 to permit the driver to operate the terminal 11. After step S402, this control routine ends.

On the other hand, if it is determined in step S401 that level 3 autonomous driving is not being implemented, that is, if level 2 or lower autonomous driving is being implemented, this control routine proceeds to step S403. In step S403, the determination unit 25 restricts the operation of the terminal 11 by the driver. Specifically, the determination unit 25 transmits a control signal to the terminal 11 to restrict the operation of the terminal 11. When the terminal 11 receives the control signal, the processor of the terminal 11 executes control to restrict the operation of the terminal 11. The operation of the terminal 11 that is restricted includes, for example, operations using the driver's fingers, and operations using the driver's voice (for example, hands-free calling, listening to the radio or music, etc.) are excluded from the restrictions. After step S403, this control routine ends.

The terminal control unit 27 may prohibit the driver from operating the terminal 11 when autonomous driving of level 2 or lower is being performed in the vehicle 30. In this case, in step S403, the terminal control unit 27 transmits a control signal to the terminal 11 to prohibit the operation of the terminal 11. As a result, the processor of the terminal 11 executes control to prohibit the operation of the terminal 11.

<Other embodiments>
Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims. For example, part of the configuration of the vehicle control system 1 shown in FIG. 1 may be omitted.

The terminal 11 that can be operated by the driver is not limited to a smartphone, but may be another mobile terminal such as a tablet terminal or a mobile game console, or a navigation device 6 provided in the vehicle 30. In addition, the above-mentioned control may be executed to monitor the driver's condition in situations other than Level 3 autonomous driving.

Also, a server provided outside the vehicle 30 may function as the driver monitoring device. In this case, an image generated by an imaging device such as the driver monitor camera 2 and driver information acquired by the terminal 11 are transmitted from the vehicle 30 to the server via a communication network, and when the determination unit of the server determines that the driver is not in a driving standby state, the notification unit of the server notifies the driver of a warning via the ECU 20 of the vehicle 30.

In addition, a computer program that causes a computer to realize the functions of each part of the processor 23 of the ECU 20 or the processor of the server may be provided in a form stored in a computer-readable recording medium. The computer-readable recording medium is, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

The above-described embodiments can be implemented in any combination. For example, when the second and third embodiments are combined, in the third embodiment, the control routine of FIG. 6 is executed instead of the control routines of FIG. 4 and FIG. 5.

2 Driver monitor camera 11 Terminal 20 Electronic control unit (ECU)
23 Processor 25 Determination unit 26 Notification unit 30 Vehicle

Claims (10)

a determination unit that determines whether or not the vehicle is in a driving standby state in which the driver of the vehicle can perform driving operations;
a notification unit that notifies the driver of a warning when it is determined that the driver is not in the driving standby state,
The determination unit acquires an image generated by an imaging device provided in the vehicle to capture an image of the driver and driver information acquired by a terminal operable by the driver, and determines whether the driver is in the driving standby state based on the image and the driver information ;
The driver monitoring device, wherein the judgment unit acquires the driver information from the terminal when it determines that the driver is not in a driving standby state based on the image, and does not acquire the driver information from the terminal when it determines that the driver is in a driving standby state based on the image . A driver monitoring device, comprising:
a determination unit that determines whether or not the vehicle is in a driving standby state in which the driver of the vehicle can perform driving operations;
a notification unit that notifies the driver of a warning when it is determined that the driver is not in the driving standby state;
having
The determination unit acquires an image generated by an imaging device provided in the vehicle to capture an image of the driver and driver information acquired by a terminal operable by the driver, and determines whether the driver is in the driving standby state based on the image and the driver information;
The driver monitoring device further has a terminal control unit that restricts or prohibits the driver from operating the terminal when autonomous driving of level 2 or lower is being implemented in the vehicle. The driver monitoring device according to claim 1 , wherein the driver information includes an image generated by a camera mounted on the terminal, or information input by the driver to the terminal. The driver monitoring device according to claim 1 or 2, wherein the determination unit determines whether the driver is in the driving standby state based on the image using a predetermined determination criterion, determines whether the driver is in an awake state based on the driver information, and relaxes the determination criterion when it is determined that the driver is in an awake state. The driver monitoring device according to claim 1 or 2, wherein the determination unit determines whether the driver is in the standby state based on the image using a predetermined determination criterion, and when it is determined that the driver is not in the standby state based on the image, it determines whether the driver is in an awake state based on the driver information, and when it is determined that the driver is in an awake state, it invalidates the determination result based on the image. The driver monitoring device according to claim 1 or 2, wherein the terminal is a smartphone. 1. A computer-implemented method for driver monitoring, comprising:
acquiring an image generated by an imaging device provided in the vehicle to capture an image of a driver of the vehicle;
acquiring driver information acquired by a terminal operable by the driver;
determining whether the driver is in a driving standby state in which the driver can perform driving operations based on the image and the driver information;
notifying the driver of a warning when it is determined that the driver is not in the driving standby state ;
When it is determined based on the image that the driver is not in the driving standby state, the driver information is acquired from the terminal, and when it is determined based on the image that the driver is in the driving standby state, the driver information is not acquired from the terminal.
A driver monitoring method comprising: 1. A computer-implemented method for driver monitoring, comprising:
acquiring an image generated by an imaging device provided in the vehicle to capture an image of a driver of the vehicle;
acquiring driver information acquired by a terminal operable by the driver;
determining whether the driver is in a driving standby state in which the driver can perform driving operations based on the image and the driver information;
notifying the driver of a warning when it is determined that the driver is not in the driving standby state;
When level 2 or lower automated driving is being performed in the vehicle, restricting or prohibiting the driver from operating the terminal;
A driver monitoring method comprising: acquiring an image generated by an imaging device provided in the vehicle to capture an image of a driver of the vehicle;
acquiring driver information acquired by a terminal operable by the driver;
determining whether the driver is in a driving standby state in which the driver can perform driving operations based on the image and the driver information;
notifying the driver of a warning when it is determined that the driver is not in the driving standby state ;
When it is determined based on the image that the driver is not in the driving standby state, the driver information is acquired from the terminal, and when it is determined based on the image that the driver is in the driving standby state, the driver information is not acquired from the terminal.
A computer program for driver monitoring, which causes a computer to execute the above. acquiring an image generated by an imaging device provided in the vehicle to capture an image of a driver of the vehicle;
acquiring driver information acquired by a terminal operable by the driver;
determining whether the driver is in a driving standby state in which the driver can perform driving operations based on the image and the driver information;
notifying the driver of a warning when it is determined that the driver is not in the driving standby state;
When level 2 or lower autonomous driving is being performed in the vehicle, restricting or prohibiting the driver from operating the terminal;
A computer program for driver monitoring, which causes a computer to execute the above.

Images