JP7522602B2 - Vehicle Driving Assistance Systems - Google Patents

Description

The present invention relates to a vehicle driving assistance system.

Research is currently being conducted into autonomous driving, which does not depend on a driver or other passenger to drive, and into assisting passengers in driving. Autonomous driving is an advanced form of driving assistance.
Meanwhile, research is also being conducted on passenger monitoring devices for passengers in vehicles such as automobiles. Patent Document 1 discloses a technique for monitoring passengers and outputting an alarm or the like.
It is expected that enabling these technologies to work organically together will increase the convenience and safety of vehicles such as automobiles.

Japanese Patent Application Publication No. 11-043009

In a device for monitoring a passenger, such as a driver, in a vehicle, an imaging device for capturing an image of the passenger can be suitably used for monitoring the passenger. The state of the passenger can be determined based on the image captured by the imaging device.
However, when monitoring the state of an occupant by capturing an image of the occupant with an imaging device, the determination is based on the image captured by the imaging device, which may result in problems such as the following.
For example, in a vehicle, depending on the driving environment, for example, strong light may shine on the face of the occupant or the interior of the vehicle may become completely dark. In this case, the occupant monitoring device may not be able to correctly judge the state of the occupant's face based on the captured image. And the driving assistance system of the vehicle that executes control based on the judgment result of the occupant monitoring device may also not be able to execute control based on the correct judgment.

For this reason, it is required that the vehicle occupant monitoring device is able to judge the facial state of the occupant as much as possible even in the above-mentioned captured image. In order to make such an image judgment possible, for example, the criteria for judging the facial state of the occupant are adjusted so that the face of the occupant that is not clearly visible in the captured image can be judged as the face of the occupant. However, it is expected that the criteria for judging the facial state of the occupant when such an adjustment is made will be looser than the original judgment criteria. As a result, for example, even if the occupant is in a correct driving state where he or she can drive a car, there is a growing concern that the quality of the captured image and the loose judgment criteria may erroneously judge that the driving state is inappropriate. In addition, for example, even if the occupant is not in a correct driving state where he or she can drive a car, there is a growing concern that the quality of the captured image and the loose judgment criteria may erroneously judge that the driving state is appropriate.

As such, there is a demand for vehicles to be improved.

A driving assistance system for a vehicle according to an embodiment of the present invention includes an imaging device capable of outputting a first captured image of an occupant of the vehicle, a warning unit that outputs a warning to the occupant, and a control unit that can execute control using the imaging device and the warning unit, wherein the control unit determines the driving state of the occupant based on a face recognition unit that executes recognition processing of at least a part of the face of the occupant for the first captured image obtained from the imaging device, and a second captured image in which at least a part of the face can be recognized by the face recognition unit, among the first captured images that can be output by the imaging device, and when the occupant is in a state where he or she is able to drive, and a loss judgment unit that generates a second warning request if a third captured image in which at least a part of the face cannot be recognized by the face recognition unit occurs consecutively at a frequency equal to or greater than a predetermined continuous loss judgment period, among the first captured images that the imaging device can output, and the warning unit outputs a warning to the occupant based on the first warning request of the occupant judgment unit and the second warning request of the loss judgment unit.

Preferably, the occupant determination unit determines whether the occupant is looking away or falling asleep based on the state of the eyes of the occupant's face in the second captured image.

Preferably, the loss determination unit determines whether or not a plurality of third captured images in which the face cannot be recognized occur as the third captured image in which the face cannot be recognized, at least for the first captured image in which the occupant determination unit cannot determine the driving state of the occupant.

Preferably, when the loss determination unit is unable to obtain the first captured image from the imaging device, it determines that a third captured image in which a face cannot be recognized by the face recognition unit has occurred, and determines whether multiple third captured images in which a face cannot be recognized have occurred.

Preferably, the loss determination unit generates the second alarm request when the face recognition unit continues to be unable to recognize a face in the first captured image that the imaging device can output.

Preferably, the loss determination unit includes a first loss determination unit that generates a third alarm request as the second alarm request when a state where the face cannot be recognized by the face recognition unit occurs continuously and continuously, and a second loss determination unit that generates a fourth alarm request as the second alarm request when a state where the face cannot be recognized by the face recognition unit occurs continuously and intermittently, and the alarm unit outputs an alarm to the occupant based on the third alarm request of the first loss determination unit and the fourth alarm request of the second loss determination unit as the second alarm request.

Preferably, the second loss determination unit determines that the loss is occurring intermittently and repeatedly for a period longer than the period for which the first loss determination unit determines continuity.

Preferably, the vehicle has a detection unit that detects whether or not the occupant is in a state where he or she can operate an operating member operated by the occupant when driving the vehicle, and when the first alarm request of the occupant determination unit or the second alarm request of the loss determination unit is generated, the warning unit determines whether or not the occupant is in a state where he or she can operate the operating member based on the detection result of the detection unit, and when it cannot be determined that the occupant is in a state where he or she can operate the operating member, outputs a warning requesting the occupant to operate the operating member.

Preferably, the vehicle has a driving control device that controls the driving of the vehicle, and the driving control device decelerates or stops the vehicle if the detection result of the detection unit does not change when the occupant is in a state in which the operation member can be operated by the alarm unit after the alarm unit outputs an alarm.

In the present invention, first, a face recognition unit performs a recognition process of at least a part of the face of the occupant based on a first captured image of an imaging device capturing an image of the vehicle occupant. Then, the occupant determination unit determines the driving state of the occupant for a second captured image in which at least a part of the face is recognized by the face recognition unit, among the first captured images that can be output by the imaging device. This makes it unnecessary for the occupant determination unit to determine the driving state of the occupant for a first captured image of poor quality in which the face cannot be recognized, for example. The occupant determination unit can determine the driving state of the occupant based on the second captured image of good quality, assuming that the second captured image is captured so that the occupant's face can be correctly recognized among the first captured images that can be output by the imaging device. The determination of the driving state of the occupant by the occupant determination unit can be optimized for that determination. It is expected that the accuracy of the determination by the occupant determination unit will be improved by the optimized determination. In particular, when determining whether an occupant is looking away or falling asleep based on the state of the eyes, which are part of the occupant's face in the captured image, the size and resolution of the image of the eyes in the image are often not high enough, and the occupant determination unit is required to determine the state of the eyes with high accuracy based on limited information about the eyes. For example, when determining the state of the eyes, such as the degree of opening or the size of the pupils, a judgment with significantly higher accuracy is required compared to simply determining whether the eyes are captured in the captured image. In the present invention, by performing advance recognition of the captured image by the face recognition unit, it is possible to enable the occupant determination unit to make such a judgment with high accuracy. Then, the warning unit can output a warning to the occupant based on the first warning request from the occupant determination unit.

Moreover, in the present invention, a loss determination unit is provided in parallel with the occupant determination unit. When a plurality of third captured images in which at least a part of the face cannot be recognized by the face recognition unit occur among the first captured images output from the imaging device, the loss determination unit generates a second warning request in the same manner as the occupant determination unit. This allows the loss determination unit to generate a second warning request for the third captured image not related to the judgment of the occupant determination unit among the first captured images output from the imaging device. For example, when the third captured image not related to the judgment of the occupant determination unit indicates a possibility that the occupant is in an inappropriate driving state, such as looking away or falling asleep, the loss determination unit can generate a second warning request based on the existence of such a possibility without directly judging the state of the occupant based on the captured image. By such parallel cooperative processing between the occupant determination unit and the loss determination unit, when there is a possibility that the occupant is in an inappropriate driving state, such as looking away or falling asleep, an alarm regarding that can be output almost reliably. In the present invention, regardless of the imaging environment or state of the imaging device, if there is a possibility that the occupant is in an inappropriate driving state, such as looking away or falling asleep, a warning about this can be output almost reliably.

In this way, the present invention can achieve two mutually contradictory processing requirements: correctly determining whether an occupant is in an inappropriate driving state, and broadly outputting an alarm regarding the possibility that an occupant is in an inappropriate driving state. As a vehicle occupant monitor, the present invention can correctly determine the driving state of an occupant, while prioritizing safety by broadly outputting an alarm regarding an inappropriate driving state, including cases where such a state is possible, and is therefore suitable for use in vehicles.

FIG. 1 is a schematic explanatory diagram of an automobile according to an embodiment of the present invention. FIG. 2 is a schematic explanatory diagram of a control system for the automobile of FIG. FIG. 3 is an explanatory diagram of an automobile driving support system realized by the control system of FIG. FIG. 4 is a flow chart of the control flow of the passenger monitoring device of FIG. FIG. 5 is a flow chart of the control flow of the alarm device of FIG. FIG. 6 is a flow chart of the control flow of the driving control device of FIG. FIG. 7 is an explanatory diagram of an example of a process for generating a warning request in a case where captured images in which a face cannot be recognized are repeatedly generated.

The following describes an embodiment of the present invention with reference to the drawings.

FIG. 1 is a schematic explanatory diagram of an automobile 1 according to an embodiment of the present invention.
The automobile 1 in FIG. 1 is an example of a vehicle. The automobile 1 has a vehicle compartment 3 in the center of a vehicle body 2, in which a seat 4 for an occupant is disposed. An occupant, such as a driver, in the automobile 1 operates the driving of the automobile 1 while seated in the seat 4. This allows the automobile 1 to drive automatically or manually. When the automobile 1 drives based on an operation for driving by an occupant, the automobile 1 may perform driving assistance. Such an automobile 1 assists both automatic driving and manual driving.

Fig. 2 is a schematic explanatory diagram of the control system 10 of the automobile 1 of Fig. 1. Fig. 2 shows the control system 10 of the automobile 1 as well as a traffic information system 100 that communicates with the control system 10.
In the automobile 1, the driving control device is required to perform automatic driving without relying on the driver or other occupants, and to assist the occupants in driving. In addition, the automobile 1 is required to identify and monitor the occupants in the automobile 1 using the occupant monitoring device 81. By organically linking these technologies, it is considered possible to identify the driver and provide advanced driving assistance tailored to the driver. It is expected that the convenience and safety of the automobile 1 will be improved.
In FIG. 2, a plurality of control devices constituting the control system 10 of the automobile 1 are shown as representatives represented by control ECUs (Electronic Control Units) incorporated in each of them.

The control system 10 of the automobile 1 in FIG. 2 specifically includes a drive ECU 11, a steering ECU 12, a braking ECU 13, a drive ECU (automatic driving/driving assistance ECU) 14 as a device for controlling the running of the automobile 1 by driving assistance during automatic driving or manual driving, a driving operation ECU 15, a detection ECU 16, an occupant monitoring ECU 17 of an occupant monitoring device, an alarm ECU 18 of an alarm device, an external communication ECU 19, and a UI ECU 20. These multiple control ECUs are directly connected to a cable 26, and are connected to a central gateway (CGW) 27 as a relay device by a vehicle network 25 such as a CAN (Controller Area Network) or a LIN (Local Interconnect Network) used in the automobile 1. Each control ECU can communicate with each other by encrypted data to which its own ID as the sender and the ID of the destination device are added. Each control ECU may simultaneously transmit encrypted data without specifying a destination to multiple other control ECUs, like a broadcast. Each control ECU receives encrypted data whose destination is its own ID and uses it for its own control. A control ECU may also receive encrypted data whose destination is a specific ID other than its own and use it for its own control. If the encrypted data received from each of the multiple cables 26 is intended for a control ECU connected to another cable 26, the central gateway 27 outputs the received encrypted data to the other cable 26. This allows the multiple control ECUs shown in FIG. 2 to transmit and receive encrypted data to each other.

The drive ECU 11 receives the encrypted data through the vehicle network 25, and controls a drive source, such as an engine or a motor (not shown), and a transmission provided in the automobile 1. This enables the automobile 1 to accelerate and run.
The steering ECU 12 receives the encrypted data through the vehicle network 25 and controls a steering device (not shown) provided in the automobile 1. This enables the automobile 1 to change the direction of travel.
The brake ECU 13 receives the encrypted data through the vehicle network 25 and controls a brake device (not shown) provided in the vehicle 1. This allows the traveling vehicle 1 to decelerate and stop. The brake ECU 13 may also communicate with the drive ECU 11 through the vehicle network 25 to slow down the rotation of the drive source or change the gear ratio of the transmission to decelerate the traveling vehicle 1.

The driving operation ECU 15 is connected to operation members such as a steering wheel 31, a brake pedal 32, an accelerator pedal 33, and a shift lever 34 for controlling the driving of the automobile 1. The driving operation ECU 15 outputs encrypted data including operation information about the operated operation members to the drive ECU 14 and the like via the vehicle network 25.

The detection ECU 16 is connected to, for example, a speed sensor 41, an acceleration sensor 42, an external camera 43, and a hands-on sensor 44. The speed sensor 41 detects the speed of the automobile 1. The acceleration sensor 42 detects the acceleration of the automobile 1. The external camera 43 captures the surroundings outside the automobile 1. The external camera 43 may capture, for example, the front, side, or rear of the automobile 1. The hands-on sensor 44 is provided, for example, on the steering wheel 31, and detects changes in capacitance or pressure caused by the hands of the occupant operating the steering wheel 31. The detection ECU 16 acquires the physical quantities of the connected sensors as physical quantities detected in the automobile 1 during the running of the automobile 1, and outputs them to the drive ECU 14, etc. through the vehicle network 25. The detection ECU 16 may also analyze the external image captured by the external camera 43 to detect other moving objects around the automobile 1, and output information about the detected other moving objects to the drive ECU 14, etc. through the vehicle network 25.

The external communication ECU 19 communicates with the outside of the automobile 1. For example, the external communication ECU 19 communicates with the communication relay station 101 of the traffic information system 100 to send and receive communication data with the server device 102. The external communication ECU 19 also sends and receives communication data with other automobiles 110 traveling near the automobile 1 or with mobile terminals (not shown) of pedestrians, for example, by V2X communication. When the external communication ECU 19 receives navigation data to the destination of the automobile 1, traffic information, information on the situation around the automobile 1, and the like through these communications, it may output the information to the drive ECU 14, etc., through the vehicle network 25.

The UIECU 20 is connected to, for example, a display device 51 and an operation device 52 .
The operation device 52 may have a touch panel and a plurality of keys that are overlaid on the display screen of the display device 51. When the UIECU 20 receives encrypted data related to display, it causes an image to be displayed on the display device 51. The UIECU 20 may generate, for example, navigation data based on the operation of the operation device 52 corresponding to the display on the display device 51, and output the data to the drive ECU 14 or the like via the vehicle network 25.

The drive ECU 14 controls the driving of the automobile 1 by switching between automatic driving and driving assistance. The drive ECU 14 controls the driving of the automobile 1 based on various information acquired through the vehicle network 25. For example, in the case of automatic driving, the drive ECU 14 outputs control information to the drive ECU 11, steering ECU 12, and braking ECU 13 so that the automobile travels to the destination while checking the safety of the surroundings according to the navigation data. In the case of driving assistance, the drive ECU 14 outputs control information to the drive ECU 11, steering ECU 12, and braking ECU 13, with the amount of operation adjusted according to the surrounding safety, etc., based on the operation information.

The occupant monitoring ECU 17 is connected to an internal camera 61, a microphone 62, a timer 63, and a memory 64.

The internal camera 61 is provided in the passenger compartment 3 to capture images of the interior of the automobile 1. The internal camera 61 is provided on the inner surface of the passenger compartment 3, for example, on the dashboard in front of the occupants as shown in FIG. 1 . As a result, the images captured by the internal camera 61 show the occupants, such as the driver, who are riding in the automobile 1. When capturing images of the entire interior of the automobile 1, the internal camera 61 may be a wide-angle camera or a 360-degree camera. The internal camera 61 repeatedly captures images of the occupants in the passenger compartment 3 at predetermined intervals.
The microphone 62 records the sounds inside the car 1 .
The timer 63 measures the time and duration.
The memory 64 records programs, setting data, and the like for passenger monitoring.

The occupant monitoring ECU 17 reads and executes the program from the memory 64. In this way, the occupant monitoring ECU 17 functions as a control unit of the occupant monitoring device 81.
The occupant monitoring ECU 17 , which serves as a control unit of the occupant monitoring device 81 , monitors the occupants, including the driver, who are in the automobile 1 , based on the images captured by the internal camera 61 and the sounds recorded by the microphone 62 .
The occupant monitoring ECU 17 identifies the occupant riding in the automobile 1 based on an image captured by the internal camera 61, for example.
The occupant monitoring ECU 17 may output setting information of the automobile 1 regarding the identified occupant to the drive ECU 14 or the like via the vehicle network 25 .
The occupant monitoring ECU 17 monitors the state of the identified occupant or the driver or other person responsible for operation, and generates an alarm request as necessary and outputs it via the vehicle network 25. The occupant monitoring ECU 17 may determine whether the occupant is dozing off based on, for example, the degree to which the occupant's eyes are open, or determine whether the occupant is looking away based on, for example, the direction of the occupant's face or eyes, and generate and output an alarm request when it determines that the occupant is dozing off or looking away.

The alarm ECU 18 is connected to a meter panel 71 and a speaker 72 .
The meter panel 71 is provided on an inner surface of the vehicle interior 3, for example, on a dashboard in front of the occupants. The meter panel 71 has a liquid crystal panel for displaying warnings and light-emitting elements.
The speakers 72 are provided on the inner surface of the vehicle interior 3, for example, on the doors, dashboard, pillars, roof, etc.
For example, when a warning request is output to the vehicle network 25, the warning ECU 18 acquires it and outputs a warning sound, a warning display, and a warning message corresponding to the warning request from the meter panel 71 and the speaker 72. This allows the occupants in the passenger compartment 3 to recognize the warning issued by the automobile 1 and to respond to the warning.

Incidentally, the passenger monitoring device 81 that monitors passengers such as the driver in the automobile 1 often uses an internal camera 61 that captures images of the passengers in order to monitor the riding conditions of the passengers. In this case, the passenger monitoring ECU 17 of the passenger monitoring device 81 determines the state of the passengers based on the images captured by the internal camera 61.
However, when the state of the occupant is determined based on capturing an image of the occupant by the internal camera 61, the determination is based on the image captured by the internal camera 61, which may result in problems such as the following.
For example, in the automobile 1, depending on the driving environment, for example, a strong light may shine on the face of the occupant or the inside of the automobile may become pitch black. In this case, the occupant monitoring ECU 17 may not be able to correctly judge the state of the occupant based on the captured image. In particular, when judging the orientation or openness of the eyes, which have a small area in the captured image, it is highly likely that it will be difficult to judge such things in the captured image if a strong light shines on the face of the occupant or the inside of the automobile becomes pitch black. In addition, the automobile 1, which executes warnings or control based on the judgment results of the occupant monitoring ECU 17, may also not be able to execute appropriate warnings or control based on correct judgments.
For this reason, the occupant monitoring ECU 17 is required to determine the state of the occupant as much as possible even for the above-mentioned inappropriate captured image. In order to enable such image determination, it is conceivable to determine the state of the occupant by determining, for example, the face or eyes of the occupant that are not clearly shown in the captured image as the face or eyes of the occupant. The criteria for determining the state of the occupant by the occupant monitoring ECU 17 are adjusted to be less strict than when the determination is made based on the face or eyes of the occupant that are clearly shown in the captured image. For this reason, in a determination made after such adjustment, as a result of the adjustment, there is an increased concern that, for example, the occupant may be erroneously determined to be in an inappropriate driving state even though he or she is in a correct driving state capable of driving the automobile 1. In addition, there is an increased concern that, for example, the occupant may be erroneously determined to be in an appropriate driving state even though he or she is not in a correct driving state capable of driving the automobile 1.
In this way, the occupant monitoring ECU 17 of the occupant monitoring device 81 is required to appropriately determine the occupant's riding status based on the image captured by the internal camera 61, even if the quality of the image is poor, so that the automobile 1 can appropriately execute warnings and control.
In addition, the driving assistance system 80 of the automobile 1 is required not to output too many warnings, so as to avoid, for example, making erroneous or overly sensitive decisions during automatic driving that would require the occupants to frequently or excessively perform certain movements or operations, such as a hands-on position.
Thus, further improvements are required in the driving assistance system 80 of the automobile 1.

FIG. 3 is an explanatory diagram of a driving assistance system 80 for the automobile 1 which is realized by the control system 10 of FIG.
3 includes an occupant monitoring device 81, an alarm device 87, a driving control device 89, and a hands-on sensor 44. The occupant monitoring device 81, the alarm device 87, and the driving control device 89 are connected to each other via the vehicle network 25.
The hands-on sensor 44 detects changes in capacitance and pressure caused by the hands of the occupant operating the steering wheel 31, and can thereby detect whether the steering wheel 31, which the occupant operates to manually drive the automobile 1, is in a state where the occupant can place their hands on it and operate it.

The passenger monitoring device 81 has a face recognition unit 82, an inattentiveness determination unit 83, a dozing-off determination unit 84, a continuous loss determination unit 85, and a frequent loss determination unit 86 as functions of the passenger monitoring ECU 17 that executes a program. In addition, FIG. 3 also shows an internal camera 61 and a timer 63.

The internal camera 61 repeatedly captures images of the occupants in the vehicle compartment 3 at predetermined intervals and outputs the captured images.

The face recognition unit 82 performs a recognition process on the image captured by the internal camera 61 to recognize the face of the occupant or parts of the face, such as the eyes, nose, and mouth. The face recognition unit 82 may check by a simple process whether the captured image can be used for the judgment of the inattentive judgment unit 83 and the dozing judgment unit 84 at the subsequent stages, but preferably recognizes the face, eyes, nose, and mouth of the occupant in the captured image using stricter judgment criteria than the inattentive judgment unit 83 and the dozing judgment unit 84 at the subsequent stages. As a result, the face recognition unit 82 can determine that the captured image is an image in which the driving state of the occupant cannot be determined, at least when the whites and blacks of the eyes are not captured in the image part of the eyes related to the judgment of the inattentive judgment unit 83 and the dozing judgment unit 84 at the subsequent stages. Or, the captured image can be determined to be an image in which the inattentive judgment unit 83 and the dozing judgment unit 84 at the subsequent stages may be confused in their judgment. If the face, eyes, nose, and mouth of the occupant can be properly recognized in the captured image, the face recognition unit 82 outputs a permission signal indicating that the occupant has been recognized. If the occupant cannot be recognized, a non-permission signal may be output.

When an authorization signal is input, the inattentiveness judgment unit 83 judges whether the occupant is looking away from the image captured by the internal camera 61. For example, the inattentiveness judgment unit 83 extracts a portion of the captured image that captures the occupant's face or eyes, and judges the direction of the occupant's face or eyes based on the extracted partial image. At this time, the inattentiveness judgment unit 83 may extract or judge using the recognition result of the previous face recognition unit 82, or may extract and judge from scratch using its own processing. Then, if the occupant's face or eyes are not facing in the direction of travel of the automobile 1, for example, forward, it outputs a warning request regarding inattentiveness. In this way, the inattentiveness judgment unit 83 judges whether the occupant is looking away based on the state of the face or eyes in the captured image.

When an authorization signal is input, the dozing judgment unit 84 judges whether the occupant is dozing based on the image captured by the internal camera 61. For example, the dozing judgment unit 84 extracts a portion of the captured image that captures the occupant's eyes, and judges the degree to which the occupant's eyes are open based on the extracted partial image. At this time, the dozing judgment unit 84 may extract or judge using the recognition result of the preceding face recognition unit 82, or may extract and judge from scratch using its own processing. Then, if the degree to which the occupant's eyes are open is equal to or less than a predetermined threshold, a warning request about dozing is output. In this way, the dozing judgment unit 84 judges whether the occupant is dozing based on the state of the eyes in the captured image.

In this way, the inattentiveness determination unit 83 and the drowsiness determination unit 84 determine whether the occupant is in an inappropriate driving state, such as looking away or falling asleep, based on the captured image in which the occupant's face, etc., can be recognized, and generate and output an alarm request if the occupant is in an inappropriate driving state. Here, an inappropriate driving state means, for example, that the occupant is not in a driving posture that would be taken when operating the vehicle 1, or is not in a state that is defined as being immediately operable.

The continuous loss determination unit 85 counts the number of captured images of the internal camera 61 in a state where the permission signal is not input, for example. The count value may be recorded in the memory 64. When an image captured by the internal camera 61 occurs in a state where the permission signal is input, the continuous loss determination unit 85 may reset the count value. In this way, the continuous loss determination unit 85 can count the number of consecutive captured images in a state where the permission signal is not input. Note that the state where the permission signal is not input includes a case where a captured image is not output at the timing when the internal camera 61 outputs a captured image. This is because the face recognition unit 82 cannot output the permission signal in this case as well. Then, when the count value indicating the consecutive number becomes equal to or greater than a threshold number of times corresponding to a predetermined continuous loss determination period, the continuous loss determination unit 85 outputs a warning request that the captured images are consecutively inappropriate and therefore it is not possible to correctly determine whether the driver is looking away or dozing. The continuous loss determination period may be continued by the timer 63.
In this way, the continuous loss determination unit 85 generates and outputs an alarm request when a state in which the face cannot be continuously recognized by the face recognition unit 82 for multiple captured images that should be output from the internal camera 61 occurs repeatedly and continuously for a certain number of times or for a certain period of time.

The frequent loss determination unit 86 counts the number of captured images of the internal camera 61 in a state where the permission signal is not input, for example. The count value may be recorded in the memory 64. Even if an image captured by the internal camera 61 occurs in a state where the permission signal is input, the frequent loss determination unit 86 does not reset the count value, but continues counting from the current value. As a result, even if a state where the permission signal is not input occurs intermittently, the continuous loss determination unit 85 can count the number of captured images in a state where the permission signal is not input throughout the entire period. Note that the state where the permission signal is not input includes a case where an image is not output at the timing when the internal camera 61 outputs an image. Then, when the count value in the most recent frequent loss determination period becomes equal to or exceeds the threshold number of times, the intermittent loss determination unit outputs a warning request that the captured images are intermittently inappropriate and therefore it is not possible to correctly determine whether the driver is looking away or dozing. The most recent frequent loss determination period may be continued by the timer 63. In this way, the frequent loss determination unit 86 generates and outputs an alarm request when a state in which the face recognition unit 82 is unable to continuously recognize a face in multiple captured images that should be output from the internal camera 61 occurs intermittently and repeatedly for a certain number of recent times or for a certain cumulative period of time.

In this way, the continuous loss determination unit 85 and the frequent loss determination unit 86 generate and output an alarm request when the face recognition unit 82 is unable to repeatedly recognize a face in multiple captured images that are continuously output from the internal camera 61.

The warning device 87 has a warning unit 88 as a function of the warning ECU 18 that executes a program.

When an alarm request is received from the occupant monitoring device 81, the alarm unit 88 judges whether or not the occupant is in a state in which he or she can operate the operating member based on the detection of the hands-on sensor 44. If it is not possible to determine that the occupant is in a state in which he or she can operate the operating member, the alarm unit 88 selects an alarm according to the alarm request and outputs the alarm. For example, the alarm unit 88 outputs an alarm to the occupant based on the alarm request to correct an inappropriate driving state such as looking away from the vehicle or falling asleep. The alarm is output to the occupant from the speaker 72 and the meter panel 71. The alarm here may be, for example, a hands-on alarm to correct an inappropriate driving state such as looking away from the vehicle or falling asleep. Furthermore, if it is possible to determine that the occupant is in a state in which he or she can operate the operating member, the alarm unit 88 does not output an alarm. This makes it possible to reduce unnecessary alarm output.

The driving control device 89 has a driving control unit 90 as a function of the drive ECU 14 that executes the program.

The driving control unit 90 controls the driving of the automobile 1 during automatic driving and manual driving of the automobile 1. For example, during automatic driving of the automobile 1, the driving control unit 90 controls the driving of the automobile 1 without depending on the operation of the occupant.
After the alarm device 87 outputs an alarm or after the occupant monitoring device 81 outputs an alarm request, the driving control unit 90 judges whether or not the occupant is in a state capable of operating the operating member based on the detection of the hands-on sensor 44. If it is not judged that the occupant is in a state capable of operating the operating member, the driving control unit 90 executes a driving control of the automobile 1 that is predetermined according to the content of the alarm. For example, the driving control unit 90 decelerates the traveling automobile 1 and stops the automobile 1 on a safe shoulder of the road or the like.

FIG. 4 is a flow chart of the control flow of the occupant monitoring device 81 of FIG.
The occupant monitoring ECU 17 repeatedly executes the process shown in FIG.

In step ST1, the occupant monitoring ECU 17 determines whether it is time to obtain an image from the internal camera 61 and output an image. The occupant monitoring ECU 17 may determine whether it is time to obtain an image based on the time measured by the timer 63 or the time elapsed since the previous image acquisition timing. If it is not time to obtain an image, the occupant monitoring ECU 17 repeats this process. If it is time to obtain an image, the occupant monitoring ECU 17 advances the process to step ST2.

In step ST2, the occupant monitoring ECU 17 acquires a new captured image that the internal camera 61 outputs.

In step ST3, the occupant monitoring ECU 17, as the face recognition unit 82, performs a recognition process for the occupant's face or parts of the face, such as the eyes, nose, and mouth, for the newly acquired captured image. The occupant monitoring ECU 17 performs a recognition process for the whites and blacks of the eyes, particularly for the eyes. In the captured image, for example, due to the imaging environment, the occupant's face or parts of the face, such as the eyes, nose, and mouth, may not be captured clearly and identifiable. In addition, the occupant monitoring ECU 17 may not be able to capture an image at the capture image capture timing due to, for example, a malfunction of the internal camera 61. The occupant monitoring ECU 17 performs a recognition process for the occupant, including these situations. When the occupant monitoring ECU 17 can recognize the occupant's face or parts of the face, such as the eyes, nose, and mouth, for the captured image, it outputs an authorization signal indicating that the occupant has been recognized to the inattentive judgment unit 83, the dozing judgment unit 84, the continuous loss judgment unit 85, and the frequent loss judgment unit 86, which are involved in the following subsequent processing. In other cases, the occupant monitoring ECU 17 does not output an authorization signal. Alternatively, the occupant monitoring ECU 17 may output a non-authorization signal. These authorization signals may be output, for example, by being recorded in the memory 64 or by being directly transmitted by inter-program communication.

In step ST4, the occupant monitoring ECU 17 determines whether or not the occupant's face or parts of the face such as the eyes, nose, and mouth have been recognized in the captured image as the face recognition unit 82. If the occupant's face or parts of the face such as the eyes, nose, and mouth have been recognized in the captured image, the occupant monitoring ECU 17 proceeds to step ST5.
If the occupant's face or the like cannot be recognized, the occupant monitoring ECU 17 advances the process to step ST8.

In step ST5, the occupant monitoring ECU 17, which serves as the inattentiveness determination unit 83 to which the permission signal has been input, determines whether the occupant is looking away from the image captured by the internal camera 61. If it determines that the occupant is looking away from the vehicle, the occupant monitoring ECU 17 advances the process to step ST10. If it does not determine that the occupant is looking away from the vehicle, the occupant monitoring ECU 17 advances the process to step ST6.

In step ST6, the occupant monitoring ECU 17, which serves as the dozing determination unit 84 to which the permission signal has been input, determines whether the occupant is dozing based on the image captured by the internal camera 61. If it determines that the occupant is dozing, the occupant monitoring ECU 17 proceeds to step ST10. If it does not determine that the occupant is dozing, the occupant monitoring ECU 17 ends the process of FIG. 4.
The occupant monitoring ECU 17 may perform the dozing off determination in step ST6 before the inattentive driving determination in step ST5.

In step ST7, the occupant monitoring ECU 17 records in the memory 64 the image loss for which the face recognition unit 82 was unable to recognize the face. The occupant monitoring ECU 17 may, for example, count the number of consecutive captured images as loss images for which the face recognition was unable to be performed, and record the count value in the memory 64. In addition, for example, the occupant monitoring ECU 17 may accumulate and record in the memory 64 the acquisition execution time of the captured image as a loss image for which the face recognition unit 82 was unable to recognize the face.

In step ST8, the occupant monitoring ECU 17, which serves as the continuous loss judgment unit 85 to which no permission signal has been input, judges whether or not a certain number of loss images for which facial recognition has not been possible by the facial recognition unit 82 have occurred consecutively, which corresponds to a predetermined continuous loss judgment period.
The occupant monitoring ECU 17 compares the number of consecutively occurring loss images with a judgment threshold corresponding to a consecutive loss judgment period. If the number of consecutively occurring loss images is equal to or greater than the judgment threshold, the occupant monitoring ECU 17 proceeds to step ST10. Otherwise, the occupant monitoring ECU 17 proceeds to step ST9.

In step ST9, the occupant monitoring ECU 17, which is the frequent loss determination unit 86 to which the permission signal has not been input, determines whether loss images for which the face recognition unit 82 has not been able to recognize have occurred intermittently with a certain frequency or more during the most recent frequent loss determination period that is set in advance. The occupant monitoring ECU 17 compares the count value of loss images during the most recent frequent loss determination period with a threshold value for the number of occurrences of loss images that is set in advance for the frequent loss determination period. If the count value of loss images during the most recent frequent loss determination period is equal to or greater than the threshold value for the number of occurrences, the occupant monitoring ECU 17 proceeds to step ST10. Otherwise, the occupant monitoring ECU 17 ends the process of FIG. 4.

In step ST10, the occupant monitoring ECU 17 outputs a hands-on warning request to the warning ECU 18 and the drive ECU 14 via the vehicle network 25. The occupant monitoring ECU 17 may output the warning request so that the cause of the warning request can be distinguished.

FIG. 5 is a flow chart of the control flow of the alarm device 87 of FIG.
The warning ECU 18, as a warning unit 88 of the warning device 87, repeatedly executes the process shown in FIG.

In step ST21, the warning ECU 18 determines whether or not a hands-on warning request has been received from the occupant monitoring device 81. If a warning request has not been received, the warning ECU 18 repeats this process. If a warning request has been received, the warning ECU 18 advances the process to step ST22.

In step ST22, the warning ECU 18 acquires the detection value of the hands-on sensor 44. If the steering wheel 31 is gripped by the occupant's hands, the detection ECU 16 outputs "hands-on" as the detection value of the hands-on sensor 44. If the steering wheel 31 is not gripped by the occupant's hands, the detection ECU 16 outputs "hands-off" as the detection value of the hands-on sensor 44. The warning ECU 18 acquires the detection value of either of these sensors.

In step ST23, the warning ECU 18 determines whether or not the occupant is in a hands-on state in which the occupant can immediately start manual operation based on the sensor detection value acquired in step ST22. If the occupant is in a hands-on state, the warning ECU 18 ends the process in FIG.
The warning ECU 18 does not output a hands-on warning based on the warning request, and ends the process in Fig. 5. If the occupant is not in a hands-on state, the warning ECU 18 advances the process to step ST24.

In step ST24, the warning ECU 18 outputs a hands-on warning. The warning ECU 18 may output a notification that a hands-on warning has been output to the drive ECU 14 via the vehicle network 25.

FIG. 6 is a flow chart of the control flow of the cruise control device 89 of FIG.
The drive ECU 14, as the driving control unit 90 of the driving control device 89, repeatedly executes the process shown in FIG.

In step ST31, the drive ECU 14 determines whether the automobile 1 has started to move. If the automobile 1 has not started to move, the drive ECU 14 repeats this process. If the automobile 1 has started to move, the drive ECU 14 advances the process to step ST32.

In step ST32, the drive ECU 14 acquires driving information.
The driving information may include, for example, a hands-on warning request from the occupant monitoring device 81, whether or not a hands-on warning is output from the warning device 87, a detection value from the hands-on sensor 44, and information for determining whether or not autonomous driving is possible. The information for determining whether or not autonomous driving is possible may include, for example, information on the state of the occupants, an image captured by the external camera 43, information on other moving objects in the vicinity, navigation data, and information on the course based thereon.

In step ST33, the drive ECU 14 determines whether a hands-on warning has been issued. If the warning device 87 has output a hands-on warning, or if the occupant monitoring device 81 has output a hands-on warning request, the drive ECU 14 determines that a hands-on warning has been issued and proceeds to step ST34. Otherwise, the drive ECU 14 proceeds to step ST36.

In step ST34, the drive ECU 14 determines whether the occupant is in a hands-on state where the occupant can immediately start manual operation based on the detection value of the hands-on sensor 44. If the occupant is in a hands-on state, the drive ECU 14 advances the process to step ST36. If the occupant is not in a hands-on state even after the warning, the drive ECU 14 advances the process to step ST35.

In step ST35, the drive ECU 14 executes automatic deceleration and stop control to bring the automobile 1 to a safe stop. The drive ECU 14 decelerates the traveling automobile 1 while checking the safety of the surroundings, and stops the automobile 1 on a safe roadside or the like. This allows the automobile 1 to decelerate and stop if the occupants are not in a hands-on state even after the warning.

In step ST36, the drive ECU 14 determines whether autonomous driving is possible. If the drive ECU 14 has navigation data and route information based on the navigation data, as well as information on other moving objects in the vicinity, and autonomous driving is possible in a state where reliable safety can be ensured, including backup from the occupants, the drive ECU 14 determines that autonomous driving is possible and proceeds to step ST37.

In step ST37, the drive ECU 14 executes autonomous automatic driving control.

In step ST38, the drive ECU 14 outputs a hands-off request to the vehicle network 25 because the vehicle 1 is in an automatic driving state. The alarm unit 88 acquires and outputs the hands-off request through the vehicle network 25. This allows the occupant to take their hands off the steering wheel 31, for example.

In step ST39, the drive ECU 14 outputs a hands-on request to the vehicle network 25 because the automobile 1 is no longer capable of automatic driving by automatic driving.
The alarm unit 88 receives and outputs a hands-on request via the vehicle network 25. This allows the occupant to grip the steering wheel 31 with their hands.

In step ST40, the drive ECU 14 controls the driving of the vehicle 1 to assist the driving operation based on the driving operation of the occupant.

FIG. 7 is an explanatory diagram of an example of a process for generating and outputting a warning request in a case where captured images in which a face cannot be recognized are repeatedly generated.
For captured images in which a face cannot be recognized, the face recognition unit 82 does not output a permission signal, and therefore the inattentive judgment unit 83 and the dozing judgment unit 84 do not execute their respective processes. In contrast, the continuous loss judgment unit 85 and the frequent loss judgment unit 86 execute their respective processes based on the continuous occurrence of loss images.
In FIG. 7, time flows from left to right.

7A to 7C are timing charts showing an example of a situation in which image loss, which makes it impossible for the face recognition unit 82 to recognize a face, occurs continuously.
7A shows the acquisition cycle of the captured image. Each arrow corresponds to one cycle in which the internal camera 61 outputs the captured image. The occupant monitoring ECU 17 executes a process of acquiring the captured image from the internal camera 61 for each imaging cycle indicated by an arrow.
7B shows an output signal of the recognition result by the face recognition unit 82 for the acquired captured images. In this example, the face recognition unit 82 outputs a recognition OK permission signal as an output signal for the first two captured images. For the next three captured images, the face recognition unit 82 outputs a recognition NG non-permission signal as an output signal. For the next three captured images, the face recognition unit 82 outputs a recognition OK permission signal as an output signal.
FIG. 7C shows an output signal for requesting a warning outputted by the continuous loss determining unit 85 based on continuous image loss.
In this example, the continuous loss determination unit 85 outputs a warning request when three or more image losses occur consecutively. The period during which three or more captured images are output consecutively is the period for determining continuous losses. The determination threshold for the count value is 3.

7D to 7F are timing charts showing an example of a situation in which image loss, which prevents face recognition by the face recognition unit 82, occurs intermittently.
FIG. 7D shows the acquisition period of captured images, similar to FIG.
7E shows an output signal of the recognition result by the face recognition unit 82 for the acquired captured image. In this example, the face recognition unit 82 outputs a recognition OK permission signal as an output signal for the first captured image. For the next captured image, it outputs a recognition NG disallowance signal as an output signal. After that, it alternately outputs a recognition OK permission signal and a recognition NG disallowance signal as an output signal for each captured image.
FIG. 7F shows an output signal for requesting a warning outputted by the frequent loss determining unit 86 based on intermittent image loss.
In this example, the frequent loss determining unit 86 outputs a warning request when four or more image losses occur intermittently in the most recent frequent loss determination period of seven or more. The determination threshold for the count value is four.
In addition, the frequent loss determination period is longer than the consecutive loss determination period. The determination threshold of the count value during the frequent loss determination period is greater than the determination threshold of the count value during the consecutive loss determination period. This prevents the frequent loss determination unit 86 from outputting a warning request before the consecutive loss determination unit 85 when, for example, four or more consecutive image losses have occurred. The consecutive loss determination unit 85 can output a significant warning request based on its own determination, without being influenced by the output of a warning request by the frequent loss determination unit 86.

As described above, in this embodiment, based on the captured images of the internal camera 61, which can periodically and repeatedly capture images of the occupants of the automobile 1, the face recognition unit 82 first performs a recognition process on the occupant's face or parts of the face, such as the eyes, nose, and mouth, as a pre-processing. Then, the occupant determination unit, such as the inattentive determination unit 83 and the drowsiness determination unit 84, determines that the occupant is not in an inappropriate driving state, such as looking away or falling asleep, for the captured images of the internal camera 61 in which the face recognition unit 82 recognizes the face, etc. This eliminates the need for the occupant determination unit to determine that the occupant is not in an inappropriate driving state, such as looking away or falling asleep, for the captured images of the internal camera 61 that are of poor quality. The occupant determination unit only needs to determine that the occupant is not in an inappropriate driving state, such as looking away or falling asleep, based on some of the captured images of good quality, on the assumption that the occupant's face, etc., are correctly recognized in the captured images of the internal camera 61. The occupant determination unit's determination of whether or not the driver is in an inappropriate driving state, such as looking away or falling asleep, can be adapted to the content deemed appropriate for that determination. The accuracy of the determination by the occupant determination unit is increased so as to be appropriate for that determination. In particular, when determining whether the driver is looking away or falling asleep based on the eyes in a captured image, the size and resolution of the eye image in the image often cannot be sufficiently high, and the occupant determination unit is required to determine the state of the eyes with high accuracy based on the eye information, which is scarce in the image. For example, a determination with significantly higher accuracy is required based on the state of the eyes, such as the degree to which the eyes are open and the size of the pupils, compared to simply determining whether or not the eyes are detected. In the present invention, the occupant determination unit can be made to be capable of such high-accuracy determination.

Moreover, in this embodiment, loss determination units such as a continuous loss determination unit 85 and a frequent loss determination unit 86 that generate a warning request in the same manner as the occupant determination unit are provided in parallel with the occupant determination unit capable of making such a highly accurate judgment. These loss determination units generate a warning request when a captured image in which the face or parts of the face, such as the eyes, nose, and mouth, cannot be recognized by the face recognition unit 82 is continuously and repeatedly generated among the multiple captured images that can be output from the internal camera 61. This allows the loss determination unit to generate a warning request by the loss determination unit for captured images not related to the judgment of the occupant determination unit among the captured images of the internal camera 61. When there is a possibility that the occupant is in an inappropriate driving state, such as looking away or falling asleep, in the captured images not related to the judgment of the occupant determination unit among the captured images of the internal camera 61, the loss determination unit can generate a warning request based on the existence of such a possibility without directly judging the state of the occupant.
For example, for an image of the eyes in a state where the occupant determination unit cannot make a highly accurate determination, the face recognition unit 82 determines that the eyes, which are part of the occupant's face, cannot be recognized, so that an alarm request can be generated and output based on the judgment of the loss determination unit in the case of an image or situation where the eye condition cannot be directly determined from the image. For low-quality images among the images captured by the internal camera 61 that are difficult for the occupant determination unit to determine, an alarm request can be generated based on the judgment of the loss determination unit. The occupant can deal with the alarm based on the judgment of the loss determination unit in the same way as in the case of an alarm based on the judgment of the occupant determination unit. A warning can be output in a situation where a warning request should be generated based on the direct and indirect judgment of the driving state of the occupant, which is multilayered by the occupant determination unit and the loss determination unit. The occurrence of a situation where a warning request is not generated in a situation where a warning request should be generated can be effectively suppressed.
In addition, for example, if the internal camera 61 breaks down, it becomes impossible to obtain captured images from the internal camera 61. In such a case, the face recognition unit 82 does not output an authorization signal, as in the case where a face cannot be recognized in the captured images. As a result, a loss determination unit such as the continuous loss determination unit 85 or the frequent loss determination unit 86 can generate an alarm request based on the continuation of a situation where captured images cannot be obtained from the internal camera 61. In this embodiment, an alarm can be issued to inform the occupant not only of the occupant's condition, but also of the occupant assistance function based on the image from the internal camera 61 not functioning properly.

By such parallel cooperative processing between the occupant determination unit and the loss determination unit, in this embodiment, when there is a possibility that the occupant is in an inappropriate driving state such as looking away from the vehicle or falling asleep, it is possible to notify the occupant of this by issuing an alarm, regardless of the environment or state of the internal camera 61. In this embodiment, it is possible to achieve at a high level two mutually contradictory requirements: accurate determination that the occupant is not in an inappropriate driving state such as looking away from the vehicle or falling asleep, and reliable output of an alarm regarding the possibility that the occupant is in an inappropriate driving state such as looking away from the vehicle or falling asleep. The driving assistance system 80 of this embodiment can be suitably used in the automobile 1 because it can achieve at a high level both accurate recognition and determination regarding occupant monitoring in the automobile 1 and ensuring high safety.
The passenger alarm system for the automobile 1, which is composed of the passenger monitoring device 81 and the alarm device 87, can be expected to provide the same effects as those described above.

The above embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible without departing from the gist of the invention.

For example, in the above-described embodiment, the occupant monitoring device 81 and the alarm device 87 are formed separately and connected to each other via the vehicle network 25 .
Alternatively, for example, the occupant monitoring device 81 and the alarm device 87 may be provided in the automobile 1 as an occupant alarm device for the automobile 1, and the whole may be connected to the vehicle network 25. In this case, the occupant monitoring device 81 will have an alarm unit 88 and the like.

1...Automobile (vehicle), 14...Drive ECU (automatic driving/driving assistance ECU), 17...Occupant monitoring ECU, 18...Alarm ECU, 31...Steering wheel, 44...Hands-on sensor, 61...Interior camera, 63...Timer, 64...Memory, 80...Driving assistance system, 81...Occupant monitoring device, 82...Facial recognition unit, 83...Inattentive judgment unit (occupant judgment unit), 84...Drowsiness judgment unit (occupant judgment unit), 85...Continuous loss judgment unit (first loss judgment unit), 86...Frequent loss judgment unit (second loss judgment unit), 87...Alarm device, 88...Alarm unit, 89...Driving control device, 90...Driving control unit

Claims (9)

an imaging device capable of outputting a first captured image capturing an occupant of a vehicle;
an alarm unit that outputs an alarm to an occupant;
a control unit capable of executing control using the imaging device and the alarm unit,
The control unit is
a face recognition unit that executes a recognition process for at least a part of a face of an occupant on the first captured image acquired from the imaging device;
an occupant determination unit that determines a driving state of an occupant based on a second captured image in which at least a part of a face can be recognized by the face recognition unit, among the first captured images that can be output by the imaging device, and generates a first warning request when the occupant is not in a state in which he or she is capable of driving; and
a loss determination unit that generates a second alarm request when a third captured image in which at least a part of a face cannot be recognized by the face recognition unit occurs consecutively a certain number of times corresponding to a predetermined continuous loss determination period, or when the third captured image occurs intermittently with a certain frequency or more during a predetermined most recent frequent loss determination period;
It functions as
The alarm unit is
outputting an alarm to an occupant based on the first alarm request of the occupant determination unit and the second alarm request of the loss determination unit;
Vehicle driver assistance system.
the occupant determination unit determines whether the occupant is looking away or falling asleep based on a state of the eyes of the occupant's face in the second captured image.
A vehicle driving assistance system according to claim 1.
The loss determination unit determines whether or not a plurality of the third captured images in which a face cannot be recognized are generated as the third captured image in which a face cannot be recognized, at least for the first captured image in which the occupant determination unit cannot determine the driving state of the occupant.
3. A vehicle driving assistance system according to claim 1 or 2.
When the first captured image cannot be acquired from the imaging device, the loss determination unit determines whether a plurality of the third captured images in which a face cannot be recognized by the face recognition unit have occurred, assuming that the third captured image in which a face cannot be recognized by the face recognition unit has occurred.
A vehicle driving assistance system according to any one of claims 1 to 3.
The loss determination unit is
generating the second alarm request when a situation occurs continuously where the face cannot be recognized by the face recognition unit in the first captured image that can be output by the imaging device;
A vehicle driving assistance system according to any one of claims 1 to 4.
As the loss determination unit,
a first loss determination unit that generates a third alarm request as the second alarm request when a state in which the face cannot be recognized by the face recognition unit occurs continuously and continuously;
a second loss determination unit that generates a fourth alarm request as the second alarm request when a state in which the face cannot be recognized by the face recognition unit occurs continuously in an intermittent and repeated manner,
The alarm unit is
outputting an alarm to an occupant based on the third alarm request of the first loss determination unit and the fourth alarm request of the second loss determination unit as the second alarm request;
A vehicle driving assistance system according to any one of claims 1 to 5.
The second loss determination unit determines that the loss occurs intermittently and repeatedly for a period longer than a period for which the first loss determination unit determines continuity.
A vehicle driving assistance system according to claim 6.
a detection unit that detects whether an operating member operated by a passenger when driving the vehicle is in a state where the passenger can operate the operating member;
The alarm unit is
when the first warning request from the occupant determination unit or the second warning request from the loss determination unit is generated, determining whether or not an occupant is in a state in which the occupant can operate the operating member based on a detection result from the detection unit;
When it is not possible to determine that the operation member is in an operable state, a warning is output to the occupant requesting the occupant to operate the operation member.
A vehicle driving assistance system according to any one of claims 1 to 7.
A driving control device for controlling driving of the vehicle,
The driving control device decelerates or stops the vehicle when the detection result of the detection unit does not change if the occupant is in a state capable of operating the operation member after the alarm unit outputs the alarm.
A vehicle driving assistance system according to claim 8 .

Images