JP7073682B2 - In-vehicle alarm device - Google Patents

Description

Embodiments of the present invention relate to an in-vehicle alarm device.

In recent years, development of a face detection technique for detecting the position and orientation of a face included in an captured still image or moving image and the state of facial parts such as eyes and mouth has been promoted. For example, in Patent Document 1, by executing face detection processing on image data acquired at any time with the driver as an image pickup target, it is detected whether the driver is in an inattentive state, a dozing state, or the like, and the state is predetermined. An in-vehicle alarm device that issues a warning to the driver when the continuation is continued for an hour or longer is disclosed.

Japanese Unexamined Patent Publication No. 2009-93674

However, a processing resource (CPU (Central Processing Unit)) that can be allocated to image processing from the viewpoint of installation space, power consumption, etc., while the real-time performance of the alarm is required from the viewpoint of safety, such as an in-vehicle alarm device. ) Under conditions where resources, etc.) are limited, it may not be possible to obtain sufficient face detection accuracy. In such a case, there is a certain probability that a false detection of a driver's inattentiveness, dozing (closed eyes), or the driver's absence (hereinafter collectively referred to as a careless state) will occur, and as a result, There is a possibility that it will be an annoying alarm device in which false alarms frequently occur for the driver and passengers.

Therefore, in the following embodiments, it is an object to provide an in-vehicle alarm device capable of reducing the occurrence of false alarms.

The vehicle-mounted alarm device according to the embodiment of the present invention has, for example, a first processing unit that executes a first face detection process for each image data input at a predetermined frame rate, and the above-mentioned image data. A second processing unit that executes a second face detection process different from the first face detection process, an alarm output unit that issues an alarm, and a face detection process for input image data when predetermined conditions are met. Is switched from the first face detection process by the first processing unit to the second face detection process by the second processing unit, and is detected from the input image data as a result of the second face detection process. When the state of the person is inadvertent, it includes a state determination unit that controls the alarm output unit and issues the alarm. With such a configuration, for example, it is possible to comprehensively judge the results of different face detection processes, or to adopt a face detection process with high detection accuracy for the second face detection process, thereby making it possible. , It is possible to reduce the occurrence of false detection. As a result, it becomes possible to realize an in-vehicle alarm device with few false alarms.

The predetermined condition is that the careless state of the person detected by the first face detection process from each of the image data input at the predetermined frame rate continues for a predetermined time or longer. According to such a configuration, for example, when the driver's careless state is detected by the first face detection process while the first face detection process executed in normal times is a process with a short processing time. , It becomes possible to configure to execute the second face detection process with high detection accuracy. As a result, it is possible to realize an in-vehicle alarm device with few false alarms while maintaining real-time performance by accelerating the timing of starting a reaction to a driver's careless state.

The predetermined condition is that, in the automatic control of the vehicle, a control schedule for changing the vehicle lane in which the vehicle travels has occurred, or a control schedule for turning the vehicle to the right or left has occurred. According to such a configuration, for example, when it is necessary to ensure the normal state of the driver, it is possible to issue an alarm based on the result of the second face detection process having high detection accuracy, so that an error can be issued. It is possible to realize an in-vehicle alarm device with few alarms.

The face detection accuracy of the second face detection process is higher than the face detection accuracy of the first face detection process. According to such a configuration, for example, it is possible to issue an alarm based on the result of the second face detection process having high detection accuracy, so that it is possible to realize an in-vehicle alarm device with few false alarms. ..

When the state of the person detected from the input image data is inadvertent as a result of the second face detection process, the state determination unit performs face detection processing on the newly input image data. The second face detection process by the second processing unit is switched to the first face detection process by the first processing unit again, and as a result of the first face detection process, the newly input image data is obtained. When the state of the person detected from the above is the careless state, the alarm output unit is controlled to issue the alarm. According to such a configuration, for example, by adopting a face detection process having a short processing time as the first face detection process, even if the processing time of the second face detection process is long, the latest operation is performed. Since it is possible to issue an alarm based on the state of the person, it is possible to realize an in-vehicle alarm device with few false alarms.

FIG. 1 is a perspective view showing a schematic configuration example of a vehicle equipped with the alarm device according to the first embodiment. FIG. 2 is a diagram showing a schematic configuration example of a dashboard of a vehicle according to the first embodiment. FIG. 3 is a diagram showing an example of a mounting position of the image pickup apparatus according to the first embodiment. FIG. 4 is a block diagram showing an example of the configuration of the control system according to the first embodiment. FIG. 5 is a functional block diagram showing a schematic configuration example of the vehicle-mounted alarm device according to the first embodiment. FIG. 6 is a diagram for explaining an outline of the alarm operation according to the first embodiment. FIG. 7 is a flowchart showing a schematic example of the main flow of the alarm operation according to the first embodiment. FIG. 8 is a flowchart showing an example of the first process executed in the alarm operation according to the first embodiment. FIG. 9 is a flowchart showing an example of the second process executed in the alarm operation according to the first embodiment. FIG. 10 is a flowchart showing an example of a flow when issuing a second process execution request according to the second embodiment. FIG. 11 is a flowchart showing a schematic example of the main flow of the alarm operation according to the second embodiment.

Hereinafter, exemplary embodiments of the present invention will be disclosed. The configurations of the embodiments shown below, as well as the actions, results, and effects produced by such configurations, are examples. The present invention can be realized by a configuration other than the configurations disclosed in the following embodiments, and at least one of various effects based on the basic configuration and derivative effects can be obtained. ..

(First Embodiment)
First, the vehicle-mounted alarm device according to the first embodiment will be described in detail with reference to the drawings. FIG. 1 is a perspective view showing a schematic configuration example of a vehicle equipped with the alarm device according to the first embodiment. FIG. 2 is a diagram showing a schematic configuration example of a dashboard of a vehicle according to the first embodiment. In the present embodiment, the vehicle 1 may be, for example, an automobile (internal combustion engine automobile) having an internal combustion engine such as a gasoline engine as a drive source, or an automobile having an electric motor (also referred to as an electric motor) as a drive source. It may be a vehicle (electric vehicle, fuel cell vehicle, etc.) or a vehicle (hybrid vehicle) driven by both of them. Further, the vehicle 1 can be equipped with various transmission devices, and can also be equipped with various devices (systems, parts, etc.) necessary for driving an internal combustion engine and an electric motor. Furthermore, the method, number, layout, etc. of the devices involved in driving the wheels in the vehicle 1 (hereinafter, the sign of the front wheel is 3F, the sign of the rear wheel is 3R, and the case where they are not distinguished are referred to as wheel 3). , Can be changed as appropriate.

As shown in FIGS. 1 and 2, the vehicle body 2 of the vehicle 1 forms a passenger compartment 2a in which an occupant including a driver rides. In the vehicle interior 2a, a steering unit 4, an acceleration operation unit 5, a braking operation unit 6, a shift operation unit 7, and the like are provided so as to face the driver's seat 2b. In the present embodiment, as an example, the steering unit 4 is a steering wheel protruding from the dashboard (instrument panel) 12. The acceleration operation unit 5 is, for example, an accelerator pedal located under the driver's feet. The braking operation unit 6 is, for example, a brake pedal located under the driver's feet. The shift operation unit 7 is, for example, a shift lever protruding from the center console. The configurations of the steering unit 4, the acceleration operation unit 5, the braking operation unit 6, the speed change operation unit 7, and the like can be variously modified.

A monitor device 11 is provided in the vehicle width direction of the dashboard 12 in the vehicle interior 2a, that is, in the central portion in the left-right direction when viewed from the driver. The monitor device 11 is provided with a display device 8 and an audio output device 9 (see FIG. 4). The display device 8 is, for example, an LCD (Liquid Crystal Display), an OLELD (Organic Electroluminescent Display), or the like. The audio output device 9 is, for example, a speaker. Further, the display device 8 may be covered with a transparent operation input unit 10 (see FIG. 4) such as a touch panel. The occupant can visually recognize the image displayed on the display screen of the display device 8 via the transparent operation input unit 10. Further, the occupant can execute the operation input to the operation input unit 10 by touching, pushing, or moving the position corresponding to the image displayed on the display screen of the display device 8 with a finger or the like. ..

An instrument panel portion 25 is provided at a portion of the dashboard 12 facing the seat 2b. The instrument panel 25 is provided with a speed display unit 25a for displaying the moving speed of the vehicle 1 and a rotation speed display unit 25b for displaying the rotation speed of an output shaft of an internal combustion engine, an electric motor, or the like as a power source. ..

At the position facing the seat 2b in the passenger compartment 2a, it is possible to image the face of the driver sitting in the seat 2b in the correct posture (hereinafter referred to as the normal state) while facing the forward direction of the vehicle 1. An image pickup device 201 is provided. The angle of view and posture of the image pickup apparatus 201 are adjusted so that the face of the driver 302 sitting in the seat 2b in a normal state is located at the center of the shooting angle of view. For example, as shown in FIG. 3, the image pickup apparatus 201 is provided on the ceiling portion 2c of the vehicle body 2. However, the installation position of the image pickup device 201 is not limited to the ceiling portion 2c, and at least both eyes of the face of the driver 302 seated in the seat 2b in a normal state such as the upper part of the windshield 2d of the vehicle body 2 and the handle column 202 are imaged. It is possible to make various changes as long as the position can be changed.

The image pickup device 201 is, for example, a CCD (Charge Coupled Device) camera or the like. For example, the image pickup apparatus 201 photographs the face of the driver 302 at a predetermined frame rate while the vehicle 1 is traveling, and sequentially outputs the image data obtained by the imaging to the ECU 14.

Next, the control system of the vehicle 1 provided with the in-vehicle alarm device according to the present embodiment will be described. FIG. 4 is a block diagram showing an example of the configuration of the control system according to the present embodiment. As illustrated in FIG. 4, the control system 100 includes an ECU (Engine Control Unit) 14, a monitor device 11, a steering system 13, distance measuring units 16 and 17, a brake system 18, a steering angle sensor 19, and an accelerator sensor. 20, a shift sensor 21 and a wheel speed sensor 22 are provided. The ECU 14, the monitor device 11, the steering system 13, the ranging units 16 and 17, the brake system 18, the steering angle sensor 19, the accelerator sensor 20, the shift sensor 21, and the wheel speed sensor 22 are via the in-vehicle network 23 as a telecommunication line. Are electrically connected to each other. The in-vehicle network 23 is constructed as, for example, a CAN (Controller Area Network).

The ECU 14 can control the steering system 13, the brake system 18, and the like by sending a control signal through the in-vehicle network 23. For example, the steering system 13 controls the traveling direction of the vehicle 1 by driving the actuator 13a based on the control signal received from the ECU 14 to give a steering angle to the front wheels 3F. The steering system 13 may be, for example, an electric power steering system, an SBW (Steer By Wire) system, or the like. For example, the brake system 18 drives the actuator 18a based on the control signal received from the ECU 14 to operate a braking device (not shown) to decelerate or stop the vehicle 1. Further, the ECU 14 detects the torque sensor 13b, the brake sensor 18b, the steering angle sensor 19, the ranging unit 16, the ranging unit 17, the accelerator sensor 20, the shift sensor 21, the wheel speed sensor 22, and the like via the in-vehicle network 23. The result, the operation signal of the operation input unit 10 and the like can be received.

The ECU 14 includes, for example, a CPU (Central Processing Unit) 14a, a ROM (Read Only Memory) 14b, a RAM (Random Access Memory) 14c, a display control unit 14d, a voice control unit 14e, and an SSD (Solid State Drive, flash memory) 14f. Etc. The CPU 14a, for example, performs image processing related to an image displayed on the display device 8, determines a movement target position (parking target position, target position) of the vehicle 1, and guides the vehicle 1 (parking route and parking guidance route). (Including), various arithmetic processes and controls such as determination of the presence or absence of interference with an object, automatic control of the vehicle 1, cancellation of automatic control, and the like can be executed. The CPU 14a can read a program installed and stored in a non-volatile storage device such as a ROM 14b, and execute arithmetic processing according to the program. The RAM 14c temporarily stores various data used in the calculation by the CPU 14a. Further, the display control unit 14d mainly executes image processing using the image data obtained by the image pickup unit 15, synthesis of image data displayed by the display device 8, and the like among the arithmetic processing in the ECU 14. Further, the voice control unit 14e mainly executes the processing of the voice data output by the voice output device 9 among the arithmetic processing in the ECU 14. Further, the SSD 14f is a rewritable non-volatile storage unit, and can store data even when the power of the ECU 14 is turned off.

In the above configuration, the CPU 14a, ROM 14b, RAM 14c and the like may be configured by a single chip such as a SoC (System-on-Chip). Further, the ECU 14 may be configured by using another logic operation processor such as a DSP (Digital Signal Processor), a logic circuit, or the like instead of the CPU 14a. Further, the ECU 14 may have a configuration in which an HDD (Hard Disk Drive) is used instead of the SSD 14f. Furthermore, the SSD 14f and the HDD may be configured to be externally attached to the ECU 14.

Further, the imaging unit 15 and the distance measuring units 16 and 17 are configured to acquire data necessary for the ECU 14 to execute automatic control of the vehicle 1. For example, the image pickup unit 15 sequentially photographs the external environment around the vehicle body 2 including the road surface on which the vehicle 1 can move and the area where the vehicle 1 can park, and outputs the image data obtained by this imaging to the ECU 14. .. The ranging units 16 and 17 are, for example, sonars that emit ultrasonic waves and capture the reflected waves, measure the distance between the vehicle 1 and an object existing around the vehicle 1, and the distance obtained by this measurement. Information is output to the ECU 14.

The brake system 18 has, for example, an ABS (Anti-lock Brake System) that suppresses the lock of the brake, a sideslip prevention device (ESC: Electronic Stability Control) that suppresses the sideslip of the vehicle 1 during cornering, and enhances the braking force (ESC: Electronic Stability Control). An electric brake system (which executes brake assist), BBW (Brake By Wire), etc. The brake system 18 applies a braking force to the wheels 3 and thus to the vehicle 1 via the actuator 18a. Further, the brake system 18 can detect signs of brake lock, idling of the wheels 3, skidding, etc. from the difference in rotation between the left and right wheels 3 and execute various controls. The brake sensor 18b is, for example, a sensor that detects the position of the movable portion of the braking operation portion 6. The brake sensor 18b can detect the position of the brake pedal as a movable part. The brake sensor 18b includes a displacement sensor.

The steering angle sensor 19 is, for example, a sensor that detects the steering amount of the steering unit 4 such as the steering wheel. The steering angle sensor 19 is configured by using, for example, a Hall element or the like. The ECU 14 acquires the steering amount of the steering unit 4 by the driver, the steering amount of each wheel 3 at the time of automatic steering, and the like from the steering angle sensor 19 and executes various controls. The rudder angle sensor 19 detects the rotation angle of the rotating portion included in the steering unit 4. The steering angle sensor 19 is an example of an angle sensor.

The accelerator sensor 20 is, for example, a sensor that detects the position of the movable portion of the acceleration operation unit 5. The accelerator sensor 20 can detect the position of the accelerator pedal as a movable portion. The accelerator sensor 20 includes a displacement sensor.

The shift sensor 21 is, for example, a sensor that detects the position of the movable portion of the shift operation unit 7. The shift sensor 21 can detect the position of a lever, an arm, a button, or the like as a movable portion. The shift sensor 21 may include a displacement sensor or may be configured as a switch. Further, at the position of the movable portion of the speed change operation unit 7, for example, a drive range for moving the vehicle 1 forward, a reverse range for moving the vehicle 1 backward, a neutral range in which the wheels 3 are not given forward or backward power, and the vehicle 1 are stopped. It shall include the parking range to be operated.

The wheel speed sensor 22 is a sensor that detects the amount of rotation of the wheel 3 and the number of rotations per unit time. The wheel speed sensor 22 outputs the number of wheel speed pulses indicating the detected rotation speed as a sensor value. The wheel speed sensor 22 may be configured by using, for example, a Hall element or the like. The ECU 14 calculates the amount of movement of the vehicle 1 based on the sensor value acquired from the wheel speed sensor 22, and executes various controls. The wheel speed sensor 22 may be provided in the brake system 18. In that case, the ECU 14 acquires the detection result of the wheel speed sensor 22 via the brake system 18.

The configurations, arrangements, electrical connection forms, etc. of the various sensors and actuators described above are merely examples, and can be variously changed as needed.

Further, in the above configuration, the image pickup device 201, the ECU 14, and the monitor device 11 constitute an in-vehicle alarm device according to the present embodiment. Hereinafter, the configuration and operation of the vehicle-mounted alarm device according to the present embodiment will be described in detail with reference to the drawings.

As a specific method for reducing the occurrence of false alarms in the in-vehicle alarm device, for example, it is possible to detect the driver's careless state (looking aside, dozing (closed eyes), absence, etc.) with higher accuracy. A method of adopting face detection processing and a method of improving the accuracy of face detection by making a comprehensive judgment by taking the logical product of the results of different types of face detection processing can be considered. Therefore, in the present embodiment, a case where a face detection process (hereinafter referred to as a high-precision face detection process) capable of detecting a driver's careless state with higher accuracy is adopted will be described as an example.

As the high-precision face detection process, for example, there is a face detection process using machine learning such as deep learning. However, such high-precision face detection processing generally requires more processing resources. The processing resource is, for example, a value obtained by multiplying the processing time by the processing capacity of the computer. Therefore, when a face detection process having a long processing time such as a high-precision face detection process is adopted, there is a possibility that the real-time property of face detection and an alarm based on the result may be impaired.

As described above, there is a trade-off relationship between the accuracy of the face detection process and the frame rate of the face detection process. Therefore, in a situation where processing resources are limited, such as a control system 100 mounted on a vehicle, it may be difficult to achieve both high face detection accuracy and high frame rate.

Therefore, in the present embodiment, the high-precision face detection process is executed when a certain predetermined condition is satisfied. With such a configuration, it is possible to execute the face detection process having a short processing time in normal times, for example, during a period in which the driver's careless state is not detected. As a result, the face detection process can be executed at a high frame rate (short processing time), so that the timing of starting the reaction to the driver's careless state can be accelerated. As a result, it is possible to realize an in-vehicle alarm device with few false alarms based on the result of high-precision face detection processing without impairing the real-time property.

Further, by configuring the face detection process having a high frame rate in normal times, it is possible to detect the driver's blinking, eye movement, mouth movement, etc. at a high frame rate. This also has the advantage that the results of face detection processing in normal times can be applied to other applications that require face detection processing at a high frame rate, such as drowsiness estimation, gaze point estimation, and lip reading. Be done.

In the following description, a process in which the processing resources required for execution are relatively large and the processing time is relatively long is also referred to as "heavy processing". On the other hand, a process in which the processing resources required for execution are relatively small and the processing time is relatively short is also referred to as "light processing". In addition, "light processing" has a short processing time but low face detection accuracy, while "heavy processing" has a long processing time but high face detection accuracy. Not limited.

The "light processing" in the present embodiment includes, for example, pattern matching (also referred to as template matching) in which feature points extracted from image data are collated with an existing template (face model) (face model fitting) to perform face detection. Alternatively, it is possible to apply face detection processing using relatively shallow deep learning in which the number of hidden layers between the input layer and the output layer is several layers. On the other hand, to the "heavy processing" in the present embodiment, face detection processing using machine learning such as relatively deep deep learning in which the number of hidden layers is more than ten layers can be applied.

For example, when face detection processing by pattern matching is adopted for "light processing", face detection processing using deep learning can be adopted for "heavy processing". Further, for example, when face detection processing using relatively shallow deep learning with several hidden layers is adopted for "light processing", the number of hidden layers is more than ten for "heavy processing". It is possible to adopt the face detection process using the above relatively deep deep learning.

Further, in addition to the combination as described above, it is possible to make a difference between "light processing" and "heavy processing" by changing the number of digits of the calculation in the face detection processing for the image data. In that case, for example, in "light processing", the processing time is shortened by discarding the digits after the decimal point and reducing the number of digits used for the calculation, and in "heavy processing", the calculation target is up to several digits after the decimal point. By doing so, it may be configured to improve the face detection accuracy.

Next, a schematic configuration example of the vehicle-mounted alarm device according to the present embodiment will be described. FIG. 5 is a functional block diagram showing a schematic configuration example of the vehicle-mounted alarm device 110 according to the present embodiment, which is composed of the image pickup device 201, the ECU 14, and the monitor device 11. As shown in FIG. 5, the vehicle-mounted alarm device 110 includes a state determination unit 111, a first processing unit 112, a second processing unit 113, a timer 114, a state flag memory 115, and an alarm output unit 116. ..

The first processing unit 112 is configured to execute the first face detection processing (hereinafter referred to as the first processing), and is the first processing for the image data input from the image pickup apparatus 201 via the state determination unit 111. Is executed, and the result is input to the state determination unit 111.

Here, the first process is, for example, a so-called "light process" in which the process time is relatively short, about several tens of ms (milliseconds) or less. Further, in the first process, for example, the face detection process is executed at the predetermined frame rate for the image data input from the image pickup apparatus 201 via the state determination unit 111 at the predetermined frame rate. The first process may be, for example, a loop process in which a process requiring a predetermined time for one execution is repeatedly executed, a periodic task scheduled to be repeatedly executed in a predetermined execution cycle, or the like.

The first processing unit 112 identifies whether the driver is in a normal state, an inattentive state, an eye closed state, or an absent / abnormal posture state as a result of the first processing, and displays information on the specified state. Input to the determination unit 111.

The second processing unit 113 is configured to execute the second face detection processing (hereinafter referred to as the second processing), and is the second processing for the image data input from the image pickup apparatus 201 via the state determination unit 111. Is executed, and the result is input to the state determination unit 111. In the present embodiment, the second process is, for example, a so-called "heavy process" in which the process time is relatively long, about 100 ms or more, and is a high-precision face detection process having higher detection accuracy than the first process.

Similar to the first processing unit 112, the second processing unit 113 also identifies whether the driver is in a normal state, an inattentive state, an eye closed state, or an absent / abnormal posture state as a result of the second processing. Then, the information regarding the specified state is input to the state determination unit 111.

The state determination unit 111 controls each unit in the vehicle-mounted alarm device 110. Further, the state determination unit 111 is triggered by the fact that a predetermined condition is satisfied, such as when the result of the first process input from the first process unit 112 indicates a careless state of the driver. Execution of the second process by the processing unit 113 is started. Further, the state determination unit 111 determines whether or not an alarm is required for the driver based on the result of the second processing input from the second processing unit 113, and if it is determined that an alarm is required, the alarm output unit 116. Is driven to issue an alarm to the driver.

The state determination unit 111, the first processing unit 112, and the second processing unit 113 have a software configuration realized in the ECU 14 by, for example, the CPU 14a of the ECU 14 reading a predetermined program from the ROM 14b or the like and executing the program. Alternatively, the hardware configuration may be realized by a dedicated chip different from the CPU 14a.

The timer 114 may be, for example, a timer provided in the ECU 14, and based on an instruction from the state determination unit 111, the operation of measuring the elapsed time, outputting the measured elapsed time, and resetting is executed.

The state flag memory 115 is, for example, a storage area reserved in the RAM 14c, and holds whether the driver is in a normal state or a careless state based on the result of face detection processing by the first processing unit 112. For example, the state flag memory 115 holds, with a one-bit or multi-bit flag, whether the driver is in a normal state or in a careless state such as an inattentive state, an eye closed state, or an absent / abnormal posture state. do. In the present embodiment, a case where the state flag holds the inattentive state, the closed eye state, and the absent / abnormal posture state separately is illustrated. In that case, as the state flag, a 2-bit flag that can distinguish and hold the four states of the normal state, the inattentive state, the closed eye state, and the absent / abnormal posture state can be used. However, when it is not possible to distinguish between the inattentive state, the closed eye state, and the absent / abnormal posture state, a 1-bit flag that can distinguish and hold the two states of the normal state and the careless state is used as the state flag. Can be transformed.

The alarm output unit 116 is composed of, for example, a display control unit 14d and a voice control unit 14a in the ECU 14, a voice output device 9 and a display device 8 in the monitor device 11, and is input from a CPU 14a constituting the state determination unit 111. Follow the instructions to issue an alarm to the driver.

Next, an outline of the alarm operation executed by the vehicle-mounted alarm device 110 in the present embodiment will be described with reference to FIG. As shown in FIG. 6, in the vehicle-mounted alarm device 110, for example, when the control system 100 starts up, the image data acquired by the image pickup device 201 is input to the state determination unit 111 at a predetermined frame rate. In normal times, the state determination unit 111 sequentially inputs the input image data to the first processing unit 112. The first processing unit 112 repeatedly executes the first processing on the image data input at a predetermined frame rate. The processing time t1 required for one first processing is, for example, 33 ms. Then, when the driver's careless state (in this example, the inattentive state) is detected as a result of the first process (timing T1), the state determination unit 111 determines the detected careless state in the state flag memory 115. The flag of (inattention state) is set, and the measurement of the elapsed time t by the timer 114 is started.

After that, when the time (duration) t continuously detected by the first process in which the careless state corresponding to the flag set in the state flag memory 115 is repeated reaches a preset predetermined time t2 ( Timing T2), the state determination unit 111 activates the second processing unit 113 to execute the second processing. The processing time t3 required for one second process is longer than the processing time t1 (for example, 33 ms) required for one first process, for example, 500 ms. Further, the predetermined time t2 is, for example, 1500 ms. The image data input to the second processing unit 113 may be, for example, image data input from the image pickup apparatus 201 to the state determination unit 111 near the timing T2.

Then, when the driver's careless state (inattention state) is detected as a result of the second processing (timing T3), the state determination unit 111 starts the first processing unit 112 again, for example, for the processing time. By executing the short first process of, it is confirmed whether or not the driver is in a careless state (inattention state) continuously until the latest. After that, when it is confirmed that the driver is in an inattentive state (inattentive state) continuously until the latest (timing T4), the state determination unit 111 drives the alarm output unit 116 to inform the driver. On the other hand, an alarm is issued by voice or video effect.

According to the above configuration, as in the above example, when the processing time t1 of the first processing is 33ms, the predetermined time t2 is 1500ms, and the processing time t3 of the second processing is 500ms, the vehicle-mounted alarm device is used. The 110 will issue a warning to the driver about 2.033 seconds after the driver's careless state is first detected by the first processing unit 112.

However, the first process executed after the second process, that is, the first process executed at the timing T3 can be omitted. In that case, the state determination unit 111 drives the alarm output unit 116 at the time of timing T3 to issue an alarm to the driver by voice, video effect, or the like. Therefore, if the predetermined time t2 is 1500 ms and the processing time t3 of the second processing is 500 ms, the in-vehicle alarm device 110 is about 2 after the driver's careless state is first detected by the first processing unit 112. After a second, a warning will be issued to the driver.

The processing time t1 (33ms) of the first processing and the processing time t3 (500ms) of the second processing described above are merely examples, and depend on the face detection processing adopted as the first processing and the second processing. It is a value that changes. Further, the predetermined time t2 (1500 ms) is a value that can be arbitrarily set.

Subsequently, the flow of the alarm operation according to the present embodiment will be described in detail with reference to the drawings. FIG. 7 is a flowchart showing a schematic example of the main flow of the alarm operation according to the present embodiment. FIG. 8 is a flowchart showing an example of the first process executed in the alarm operation according to the present embodiment. FIG. 9 is a flowchart showing an example of the second process executed in the alarm operation according to the present embodiment. In the explanation of this operation, it is assumed that image data is input to the ECU 14 from the image pickup apparatus 201 at a predetermined frame rate after the power of the control system 100 is turned on.

As shown in FIG. 7, in this alarm operation, first, the state determination unit 111 resets the state flag and the timer 114 for each state in the state flag memory 115 (step S101). In the reset state, the state flag in the state flag memory 115 indicates that the driver is in a normal state.

Next, the state determination unit 111 determines whether or not the movable portion is set in the drive range based on the position information (hereinafter referred to as shift position information) of the movable portion of the shift operation unit 7 input from the shift sensor 21. Is determined (step S102). If the drive range is not set (NO in step S102), for example, if the speed change operation unit 7 is set to the reverse range, the neutral range, or the parking range, the state determination unit 111 ends this operation. When it is determined (step S103) and the process is terminated (YES in step S103), this operation is terminated. On the other hand, if this operation is not completed (NO in step S103), the process returns to step S102.

On the other hand, when the movable unit of the speed change operation unit 7 is set to the drive range (YES in step S102), the state determination unit 111 activates the first processing unit 112 and at a predetermined frame rate from the image pickup device 201. The input image data is sequentially input to the first processing unit 112, and the execution of the first processing, which is a repetitive processing, is started (step S104). It is assumed that the activation of the first processing unit 112 includes the allocation of processing resources such as the CPU resource (CPU 14a) to the first processing unit 112, for example. Further, the result of each of the first processes to be repeatedly executed is input from the first process unit 112 to the state determination unit 111 at any time.

Next, the state determination unit 111 determines whether or not the driver's careless state (any of an inattentive state, an eye closed state, and an absent / abnormal posture state) is detected by the first process (step S105). When the careless state is not detected (NO in step S105), the state determination unit 111 returns to step S101, resets the state flag and the timer 114 in the state flag memory 115 (step S101), and thereafter. Perform the action.

On the other hand, when the careless state is detected by the first process (YES in step S105), the state determination unit 111 has already set the state of the driver detected in the first process to the state flag in the state flag memory 115. If it is not set (NO in step S106), the state of the driver detected in the first process is set in the state flag in the state flag memory 115 (step S106). S107), the measurement of the elapsed time t by the timer 114 is started (step S108), and the process returns to step S102.

When the state of the driver detected in the first process has already been set in the state flag (YES in step S106), the state determination unit 111 sets in advance the elapsed time t measured by the timer 114. It is determined whether or not the predetermined time t2 or more is reached, that is, whether or not any of the inattentive state, the closed eye state, and the absent / abnormal posture state continues for the predetermined time t2 or more (step S109), and the predetermined time t2 is determined. If has not been reached (NO in step S109), the process returns to step S102.

When the elapsed time t is equal to or longer than the predetermined time t2 (YES in step S109), the state determination unit 111 activates the second processing unit 113 and second processes the image data input from the image pickup device 201. It is input to the unit 113 and the second process is executed (step S110). When the second processing unit 113 is started, for example, the processing resources allocated to the first processing unit 112 are released and allocated to the second processing unit 113.

Next, the state determination unit 111 determines whether or not the driver's careless state is detected by the second process (step S111), and if the careless state is not detected (NO in step S111), the step. It returns to S101, resets the state flag and the timer 114 in the state flag memory 115 (step S101), and executes the subsequent operations.

On the other hand, when the driver's careless state is detected by the second process (YES in step S111), the state determination unit 111 starts the first process unit 112 again for confirmation and executes the first process. (Step S112). Subsequently, the state determination unit 111 determines whether or not the careless state is continuously detected by the first process executed in step S112 (step S113), and when the careless state is detected (step S113). YES), the alarm output unit 116 is driven to issue a warning to the driver (step S114). After that, the state determination unit 111 returns to step S110 and repeats the execution of the second process and the first process (steps S110 and S112) until the driver's careless state is no longer detected (NO in step S113).

On the other hand, when the driver's careless state is not detected by the first process of step S112 (NO in step S113), the state determination unit 111 returns to step S101, and the state flag and timer in the state flag memory 115. 114 is reset (step S101), and the subsequent operations are executed.

If the first process for confirmation is not executed after the second process is executed, steps S111 to S112 in FIG. 7 are omitted. In that case, when the driver's careless state is detected by the second process in step S113, the state determination unit 111 drives the alarm output unit 116 to issue a warning to the driver (YES in step S113). In step S114), if the driver's careless state is not detected (NO in step S113), the process returns to step S101.

Next, an example of the flow of the first process according to the present embodiment will be described. As shown in FIG. 8, in the first processing according to the present embodiment, first, the first processing unit 112 inputs the image data acquired by the image pickup apparatus 201 at a predetermined frame rate via the state determination unit 111 ( Step S121). Subsequently, the first processing unit 112 extracts the feature points of the driver's face from the input image data, and executes a template matching process of collating the template with the extracted feature points (step S122). As a result, it is detected whether the driver's state at the time of imaging is a normal state, or one of an inattentive state, an eye closed state, and an absent / abnormal posture state.

The first processing unit 112 may detect, for example, only one eye of the driver from the image data, or the line of sight of the driver specified from the image data may be significantly different from the traveling direction of the vehicle 1. For example, it is determined that the driver is in a state of looking aside. Further, in the first processing unit 112, for example, when both eyes of the driver specified from the image data (one eye when only one eye is detected) are closed, the driver is in a closed eye state (or). Judge that you are in a dozing state). Further, the first processing unit 112 is, for example, when the driver's face cannot be detected from the image data, or the position of the detected driver's face is largely deviated from the vicinity of the center of the image which is the normal position. In some cases, it is determined that the driver is absent / abnormal posture.

Next, when the state of the driver detected by the template matching process in step S122 is the inattentive state (YES in step S123), the first processing unit 112 puts the driver in the inattentive state with respect to the state determination unit 111. It outputs that there is something (step S124), and ends this operation.

Further, in the first processing unit 112, when the state of the driver detected by the template matching process in step S122 is the closed eye state (NO in step S123, YES in S125), the driver determines the state with respect to the state determination unit 111. It outputs that the eyes are closed (step S126), and ends this operation.

Further, when the driver's state detected by the template matching process in step S122 is absent or in an abnormal posture state (NO in step S123, NO in S125, YES in S127), the first processing unit 112 is a state determination unit. It is output to 111 that the driver is absent / abnormal posture state (step S128), and this operation is terminated.

On the other hand, when the state of the driver detected by the template matching process in step S122 is a normal state (NO in step S123, NO in S125, NO in S127), the first processing unit 112 refers to the state determination unit 111. It outputs that the driver is in the normal state (step S129), and ends this operation.

The operation shown in FIG. 8 may be a flow of returning to step S121 after the execution of steps S124, S126, S128 or S129. In that case, the first processing unit 112 ends the operation shown in FIG. 8, for example, by interrupt processing from the outside.

Next, an example of the flow of the second process according to the present embodiment will be described. As shown in FIG. 9, in the second processing according to the present embodiment, first, the second processing unit 113 obtains one of the image data acquired by the image pickup apparatus 201 at a predetermined frame rate via the state determination unit 111. Input (step S141). Subsequently, the second processing unit 113 executes high-precision face detection processing using machine learning such as deep learning (step S142).

As described above, the high-precision face detection process may be, for example, a face detection process using machine learning such as relatively deep deep learning in which the number of hidden layers is a dozen or more layers. As a result, it is detected whether the driver's state at the time of imaging is a normal state, or one of an inattentive state, an eye closed state, and an absent / abnormal posture state. In the second process, the criteria for determining whether the driver is in a normal state, an inattentive state, an eye closed state, or an absent / abnormal posture state are the same as the criteria used in the first process described above. May be.

Further, it is preferable that the second process, which is a high-precision face detection process, can also detect the attachment and detachment of sunglasses and a mask by the driver. For example, when sunglasses are worn, in the second treatment, the driver is in a normal state or is not in a normal state based on the orientation and inclination of the face, the positional relationship and shape of the nose and mouth, and the like. It can be determined whether it is in an attention state. Further, for example, when a mask is worn, in the second treatment, whether the driver is in a normal state or in a careless state based on the positional relationship and shape of both eyes. Can be determined. Further, when both sunglasses and a mask are worn, in the second treatment, it is determined whether the driver is in a normal state or in a careless state based on the orientation of the face or the like. Can be done. This makes it possible to enhance the robustness in face detection and further suppress the occurrence of false alarms.

Next, when the driver's state detected by the high-precision face detection process in step S142 is the inattentive state (YES in step S143), the second processing unit 113 similarly to steps S123 to S129 in FIG. It outputs to the state determination unit 111 that the driver is in the inattentive state (step S144), and outputs that the driver is in the closed eye state (NO in step S143, YES in S145) and outputs that the driver is in the closed eye state (step S146). ), If it is in the absent or abnormal posture state (NO in step S143, NO in S145, YES in S147), it outputs that it is in the absent / abnormal posture state (step S148). After that, the second processing unit 113 ends this operation. On the other hand, when the driver's state is normal (NO in step S143, NO in S145, NO in S147), the second processing unit 113 indicates that the driver is in the normal state with respect to the state determination unit 111. Output (step S149), and this operation is terminated.

As described above, according to the present embodiment, by executing the first process, which is a light process in normal times, the timing of starting the reaction to the driver's careless state is advanced, and the driver's carelessness is caused by the first process. When a state is detected, it is possible to detect the driver's careless state with high accuracy by executing the second process, which is a heavy process, so an in-vehicle alarm device with few false alarms is realized. It becomes possible to do.

In the above description, the case where the second process is executed when the elapsed time t in which the same careless state is continuously detected becomes the predetermined time t2 or more has been illustrated, but this embodiment has this embodiment. It is not limited to such a configuration. For example, when the same careless state is continuously detected a predetermined number of times, it can be modified in various ways, such as being configured to execute the second process.

Further, in the above description, when the elapsed time t of each of the inattentive state, the closed eye state, and the absent / abnormal posture state is compared with the common predetermined time t2, and the elapsed time t becomes the predetermined time t2 or more. The case where the second process is executed has been illustrated, but the present invention is not limited to this. For example, different predetermined times t2 may be set for each of the inattentive state, the closed eye state, and the absent / abnormal posture state. At that time, the predetermined time t2 for the absentee / abnormal posture state may be set to 0 seconds. In that case, when the absence / abnormal posture state is detected by the first process, the second process is immediately executed, and an alarm is issued to the driver according to the result.

(Second embodiment)
Next, the vehicle-mounted alarm device according to the second embodiment will be described in detail with reference to the drawings. In the first embodiment, the second process, which is a face detection process different from the first process, is triggered by the fact that the careless state is continuously detected for a predetermined time or longer by the first process repeatedly executed in a short cycle. Illustrated the case where is executed. However, the trigger for executing the second process is not limited to the driver being inattentive as illustrated in the first embodiment. Therefore, in the second embodiment, even when a condition different from the careless state of the driver is satisfied, the case where the second process is executed with that as a trigger will be described with an example. do.

As a condition other than the driver being inattentive for a predetermined time or longer, for example, in the automatic control of the vehicle 1, the control of changing the vehicle lane, the control of turning right or left, and the like, the driver It is possible to exemplify the occurrence of a control event that needs to ensure that the vehicle is seated in a normal state. Therefore, in the second embodiment, there is an example of a case where the second process is executed with the automatic control of the vehicle 1 as a trigger to generate a control for changing the vehicle lane and a control for turning right or left. Will be explained.

Since the vehicle and the control system mounted on the vehicle according to the present embodiment may be the same as the vehicle 1 and the control system 100 exemplified in the first embodiment, they will be referred to here in an overlapping manner. Is omitted.

FIG. 10 is a flowchart showing an example of a flow when the CPU 14a of the ECU 14 issues a second process execution request as a trigger for executing the second process in the automatic control executed by the control system 100. As shown in FIG. 10, in this operation, the CPU 14a first waits until the automatic control of the vehicle 1 is started (NO in step S201). After that, for example, when the driver operates a switch or the like provided on the operation input unit 10, the speed change operation unit 7, or the steering unit 4, the automatic control of the vehicle 1 by the ECU 14 is started (YES in step S201). , The CPU 14a determines in the automatic control whether or not a control schedule for changing the vehicle lane has occurred (step S202). Further, the CPU 14a determines whether or not a right turn or left turn control schedule has occurred in the automatic control (step S203). When neither the control schedule for changing the vehicle lane nor the control schedule for right turn or left turn has occurred (NO in step S202, NO in S203), the CPU 14a determines whether or not the automatic control has been completed (NO in step S202, NO in S203). In step S206), if it has been terminated (YES in step S206), this operation is terminated. On the other hand, when the automatic control is not completed (NO in step S206), the CPU 14a returns to step S202 and continues the subsequent operations.

On the other hand, in the automatic control, when a vehicle lane change control schedule occurs (YES in step S202), or when a right turn or left turn control schedule occurs (YES in step S203), the CPU 14a is the scheduled vehicle. In the control to change the lane or the scheduled right turn or left turn control, the vehicle waits until a predetermined time before the scheduled start time of steering of the vehicle 1 (NO in step S204), and the steering start time is higher than the scheduled start time. At the timing before a predetermined time (YES in step S204), an execution request for the second process is issued (step S205). After that, the CPU 14a proceeds to step S206.

The second process execution request issued as described above is input to the state determination unit 111. When the second process execution request is input during the execution of the alarm operation, the state determination unit 111 executes the second process by using the input of the second process execution request as a trigger. Here, FIG. 11 shows an example of the alarm operation according to the present embodiment. As shown in FIG. 11, in the alarm operation according to the present embodiment, in the same operation as the alarm operation described with reference to FIG. 7 in the first embodiment, for example, step S211 is executed before the execution of step S104. It is configured in.

In step S211th, the state determination unit 111 determines whether or not the second process execution request has been input, and if not (NO in step S211), proceeds to step S104 to execute the first process. On the other hand, when the second process execution request is input (YES in step S211), the state determination unit 111 proceeds to step S110 to execute the second process.

By configuring and operating as described above, in the second embodiment, it is ensured that the driver is seated in the normal state in the automatic control of the vehicle 1 not only in the careless state of the driver. It is possible to configure the second process to be executed by using the occurrence of a control event that needs to be performed as a trigger. As a result, high-precision face detection occurs even when control events that require the driver to be seated in a normal state, such as control to change the vehicle lane or turn right or left, occur. Since it is possible to issue an alarm to the driver based on the result of the processing, it is possible to realize an in-vehicle alarm device with few false alarms.

Since other configurations, operations, and effects may be the same as those in the above-described embodiment, detailed description thereof will be omitted here.

Further, in the above-described embodiment, the case where the first treatment is a “light treatment” and the second treatment is a “heavy treatment” is illustrated, but the configuration is not limited to this. For example, it is possible to use a method of improving the accuracy of face detection based on the results of different types of face detection processing. In that case, for example, as the first process, a face detection process using a so-called RNN (Recurrent Neural Network), which executes a face detection process using image data for several frames as time-series information, is adopted, and the second process is performed. As a result, it is possible to adopt a face detection process using a so-called CNN (Convolution Neural Network), which executes a face detection process using image data for one frame. Even in such a configuration, as in the above-described embodiment, the second process (for example, face detection process using CNN) is based on the result of the first process (for example, face detection process using RNN) executed in normal times. Is executed. Since the face detection process using RNN uses image data for several frames as time-series information, when RNN is adopted for the first process, the error of the face detection result obtained from each frame is accumulated, and the error of the face detection result is accumulated. As a result, there is a possibility that an erroneous face detection result will be continuously output. Therefore, by executing the face detection process using CNN using image data for a single frame as the second process before the warning to the driver, the driver's state is not affected by the accumulated error. Can be confirmed. As a result, it becomes possible to realize an in-vehicle alarm device with few false alarms.

Although the embodiments of the present invention have been illustrated above, the above-described embodiments and modifications are merely examples, and the scope of the invention is not intended to be limited. The above-described embodiment and modification can be implemented in various other forms, and various omissions, replacements, combinations, and changes can be made without departing from the gist of the invention. Further, the configuration and shape of each embodiment and each modification can be partially replaced.

1 ... Vehicle, 2 ... Body, 2a ... Vehicle room, 2b ... Seat, 2c ... Ceiling, 2d ... Front glass, 3 ... Wheels, 3F ... Front wheels, 3R ... Rear wheels, 4 ... Steering section, 5 ... Acceleration operation section , 6 ... Braking operation unit, 7 ... Shift operation unit, 8 ... Display device, 9 ... Audio output device, 10 ... Operation input unit, 11 ... Monitor device, 12 ... Dashboard, 13 ... Steering system, 13a ... Actuator, 13b ... Torque sensor, 14 ... ECU, 14a ... CPU, 14b ... ROM, 14c ... RAM, 14d ... Display control unit, 14e ... Voice control unit, 14f ... SSD, 15 ... Imaging unit, 16, 17 ... Distance measuring unit, 18 ... Brake system, 18a ... Actuator, 18b ... Brake sensor, 19 ... Steering angle sensor, 20 ... Accelerator sensor, 21 ... Shift sensor, 22 ... Wheel speed sensor, 23 ... Vehicle network, 25 ... Instrument panel, 25a ... Speed display Unit, 25b ... Rotation speed display unit, 110 ... Vehicle-mounted alarm device, 111 ... Status determination unit, 112 ... First processing unit, 113 ... Second processing unit, 114 ... Timer, 115 ... Status flag memory, 116 ... Alarm output unit , 201 ... Imaging device, 202 ... Handle column.

Claims (3)

A first processing unit that executes a first face detection process for each image data input at a predetermined frame rate, and a first processing unit.
A second processing unit that executes a second face detection process different from the first face detection process for image data, and a second processing unit.
An alarm output unit that issues an alarm and
When a predetermined condition is satisfied, the face detection process for the input image data is switched from the first face detection process by the first processing unit to the second face detection process by the second processing unit. As a result of the second face detection process, when the state of the person detected from the input image data is a careless state, the state determination unit that controls the alarm output unit to issue the alarm, and the state determination unit.
Equipped with
The predetermined condition is that the careless state of the person detected by the first face detection process from each of the image data input at the predetermined frame rate continues for a predetermined time or longer.
An in-vehicle alarm device whose face detection accuracy of the second face detection process is higher than that of the face detection accuracy of the first face detection process . The predetermined condition is that, in the automatic control of the vehicle, a control schedule for changing the vehicle lane in which the vehicle travels has occurred, or a control schedule for turning the vehicle to the right or left has occurred. Item 1. The in -vehicle alarm device according to Item 1. When the state of the person detected from the input image data is inadvertent as a result of the second face detection process, the state determination unit performs face detection process on the newly input image data. The second face detection process by the second processing unit is switched to the first face detection process by the first processing unit again, and as a result of the first face detection process, the newly input image data is obtained. The vehicle-mounted alarm device according to claim 1, wherein when the state of the person detected from the above is the careless state, the alarm output unit is controlled to issue the alarm.

Images