JP7799969B2 - Driver state determination method and determination system - Google Patents

Description

The present invention relates to a driver state determination method and determination system for determining the state of a vehicle driver.

Recently, the development of automated driving systems has been promoted nationwide. The applicant believes that, at present, automated driving systems can be broadly divided into two directions.

The first direction is a system in which the automobile takes the lead in transporting passengers to their destination without the need for driver operation, known as fully automated automobile driving.
The second direction is an autonomous driving system that is based on the premise that humans will be driving, with the aim of "providing an environment where driving a car is enjoyable."

In the second direction of automated driving systems, for example, if the driver develops an illness or other condition that makes normal driving difficult, the vehicle is expected to automatically take over driving on behalf of the occupants. For this reason, being able to detect driver abnormalities, particularly functional impairments or illnesses, as early and accurately as possible is extremely important from the perspective of improving the driver's survival rate and ensuring the safety of those around them.

As a method for determining the driver's state, for example, the visual target determination device in Patent Document 1 discloses a configuration that includes a gaze detection unit that detects the driver's gaze behavior, such as the direction of gaze; a vehicle information acquisition unit that acquires vehicle information related to the driving state; an image acquisition unit that acquires captured images of the vehicle's surroundings; a gaze area extraction unit that extracts a gaze area within the captured images based on the driver's gaze behavior; a candidate detection unit that detects objects with saliency in a top-down saliency map as candidates for the visual target; and a configuration that determines the visual target based on the gaze area extraction results and the candidate visual target detection results.

Japanese Patent Application Laid-Open No. 2020-071528

The top-down saliency map in Patent Document 1 is a saliency map that is uniquely determined from gaze behavior, absorbing individual differences such as the driver's viewing order, physique, and habits, as well as ambiguity in the scene from which the captured image was obtained, and fluctuations in gaze direction detected by the gaze detection unit.
Typically, the individual characteristics (individual differences) of the driver affect the driver's gaze behavior and are reflected as learning information in the state determination control of the abnormality determination system.
It is believed that the more data collected, the more reliable such learning information becomes, and the accuracy of abnormality determination also increases in proportion to the amount of collected data.

As a result of extensive research, the inventors have found that in learning information related to human factors, there are environmental conditions and vehicle conditions in which personal characteristics are likely to appear, and that in conditions in which these personal characteristics are likely to appear, the personal characteristics are emphasized, resulting in a relative decrease in the reliability of the learning information.
Therefore, even if information that overemphasizes individual characteristics is collected and applied to learning control, the accuracy of judgment cannot be improved as much as expected, despite the consumption of data collection time.
That is, it is not easy to shorten the learning period while ensuring the accuracy of state determination.

The object of the present invention is to provide a driver state determination method and determination system that can shorten the learning period while ensuring state determination accuracy.

The invention of claim 1 provides a driver state determination method for determining the state of a driver of a vehicle, comprising a peripheral information acquisition step for acquiring information on a peripheral situation of the vehicle, a vehicle state acquisition step for acquiring a physical state of the vehicle, a visual field image creation step for creating a visual field image that appears in the visual field of the driver, a reference visual field distribution generation step for generating a reference visual field distribution based on a standard visual field image, a visual field behavior distribution generation step for generating a visual field behavior distribution based on the visual field behavior of the driver, a learning information generation step for generating learning information that associates the peripheral situation of the vehicle, the physical state of the vehicle, and the visual field behavior of the driver, a driver abnormality estimation step for comparing the visual field behavior distribution with the reference visual field distribution corrected using the learning information to determine whether the driver is abnormal, and a driver abnormality estimation step for estimating whether the driver is abnormal or not based on a traffic situation estimated based on the peripheral situation of the vehicle and a vehicle state acquired from the physical state of the vehicle. The system further includes a reliability setting process for setting the reliability of the field of view image as learning data for generating the learning information, a learning restriction process for executing learning based on the field of view image when the reliability as learning data is equal to or greater than a judgment threshold and restricting the learning based on the field of view image when the reliability as learning data is less than the judgment threshold, a first reliability index calculation process for calculating a first reliability index of the field of view image as learning data based on a first disturbance factor related to the surrounding conditions of the vehicle acquired in the surrounding information acquisition process, and a second reliability index calculation process for calculating a second reliability index of the field of view image as learning data based on a second disturbance factor related to the physical state of the vehicle acquired in the vehicle state acquisition process, and the reliability setting process is characterized in that the reliability of the field of view image included in the driver's field of view as learning data is set using the first reliability index and the second reliability index .

This driver condition determination method includes a driver abnormality estimation step for comparing the gaze behavior distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal, and therefore it is possible to estimate whether the driver is in an abnormal state based on the gaze behavior.
Since the system has a reliability setting process for setting the reliability of the field of view image included in the driver's field of view and used to generate the learning information as learning data based on the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle conditions obtained from the physical conditions of the vehicle, it is possible to set the reliability of the learning data representing the driver's personal characteristics for the traffic conditions estimated based on the surrounding conditions of the vehicle and the vehicle conditions obtained from the physical conditions of the vehicle.
When the reliability of the learning data is equal to or greater than the judgment threshold, learning based on the field of view image is executed, and when the reliability of the learning data is less than the judgment threshold, a learning restriction process is included which restricts learning based on the field of view image. This makes it possible to restrict learning in situations where personal characteristics become significant, thereby ensuring the accuracy of state judgment while shortening the learning period.

According to this configuration, by calculating a first reliability index, which is the reliability of the learning data related to environmental factors, and calculating a second reliability index, which is the reliability of the learning data related to vehicle factors, the reliability of the learning data can be set with high accuracy from both the environmental factors and vehicle factors perspectives.

The driver state determination method and determination system of the present invention limit learning based on the tendencies of personal characteristics for each scene, thereby shortening the learning period while ensuring state determination accuracy.

1 is a block diagram illustrating a configuration of a driver state determination system according to a first embodiment; FIG. 4 is a diagram illustrating processing by a control unit. The gaze behavior distributions show a normal distribution corresponding to saliency, a normal distribution corresponding to saccade amplitude, and a normal distribution corresponding to saccade frequency. FIG. 10 is a diagram showing an example of a time change in saliency related to a driver's gaze point. FIG. 4 is a diagram showing an example of time-dependent change in saliency related to random points whose coordinates are randomly specified in the same environment as in FIG. 3 . 10 is a graph comparing the probability of exceeding a threshold at a random point with the probability of exceeding a threshold at a driver's gaze point. FIG. 1 is an explanatory diagram of a saliency index. FIG. 10 is a diagram showing the time series distribution of saliency indices. FIG. 10 is a diagram showing a time series distribution of risk potentials. FIG. 10 is a diagram showing a combined time series distribution of a saliency index and a risk potential. FIG. 10 is a diagram showing the probability distribution of saliency indices for each divided area of the risk potential. 10 is a graph for explaining extraction of a saccade. 10 is a graph for explaining a noise removal process of a saccade. 10 is a reliability setting table. 10 is a flowchart of a pre-learning process. 10 is a flowchart of a state determination process. 10 is a flowchart of a total evaluation value calculation process. 10 is a flowchart of a learning phase process. 10 is a flowchart of a reliability determination process.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
The following description exemplifies the application of the present invention to a driver state determination system for a vehicle, and does not limit the present invention, its applications, or its uses.
The following description also includes a description of a driver state determination method.

A first embodiment of the present invention will be described below with reference to FIGS. 1 to 19. FIG.
This driver condition determination system S is a system that estimates the state of a driver by detecting signs of an abnormal state of the driver based on the gaze behavior of the driver operating the vehicle.
The vehicle is, for example, a right-hand drive four-wheeled automatic vehicle that can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in which the vehicle travels according to the driver's operation (for example, accelerator operation, etc.). Assisted driving is driving in which the vehicle assists the driver's operation. Automatic driving is driving in which the vehicle travels without the driver's operation.

1, the driver state determination system S controls the behavior of the vehicle in manual driving and assisted driving. Specifically, the driver state determination system S controls the operation of the vehicle (particularly the driving operation) by controlling an actuator 3 provided in the vehicle.
The driver condition determination system S mainly comprises an information acquisition unit 1, a control unit 2, an actuator 3, and a notification unit 4. When an abnormality in the driver is determined, the actuator 3 causes the vehicle to retreat to a predetermined safety area, and the notification unit 4 notifies the management center of the abnormality in the driver.

First, the information acquisition unit 1 will be described.
The information acquisition unit 1 acquires various types of information used for vehicle control, mainly for driving control, various types of information related to the conditions around the vehicle, and various types of information used for determining the state of the driver.
As shown in Figure 1, the information acquisition unit 1 includes a plurality of external cameras 11, an internal camera 12, a plurality of radars 13, a position sensor 14, an external input unit 15, a vehicle condition sensor 16, a driving operation sensor 17, a biometric information sensor 18, and the like.
The external camera 11, the multiple radars 13, the position sensor 14, and the external input unit 15 correspond to the surrounding information acquisition means, the internal camera 12 and the biometric information sensor 18 correspond to the driver characteristic acquisition means, and the vehicle state sensor 16 and the driving operation sensor 17 correspond to the vehicle state acquisition means.

The external cameras 11 are provided to cover the entire area around the vehicle and have the same configuration. Each external camera 11 captures an image of a part of the environment surrounding the vehicle (external environment around the vehicle) to obtain image data showing a part of the external environment of the vehicle.
The image data acquired by the external camera 11 is transmitted to the control unit 2. The external camera 11 is a monocular camera with a wide-angle lens, and is configured using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor).

The internal camera 12 is provided inside the vehicle.
The internal camera 12 captures an image of a predetermined area including the driver's eyeballs to obtain image data including the driver's eyes. The internal camera 12 is disposed in front of the driver, and its imaging range is set so that the driver's eyeballs are within the imaging range. The vehicle has a drowsiness determination device (not shown) for determining the driver's drowsiness, and the internal camera 12 also serves as an imaging means for the drowsiness determination device. The image data obtained by the internal camera 12 is transmitted to the control unit 2.

The radars 13 are provided to cover the entire area around the vehicle and have the same configuration as each other. The radars 13 detect a part of the external environment of the vehicle (e.g., another vehicle, an obstacle such as a building) in order to determine the direction, relative speed, distance, etc. to that part.
The radar 13 detects a part of the external environment of the vehicle by transmitting radio waves toward the part of the external environment and receiving waves reflected from the part of the external environment. The detection result of the radar 13 is transmitted to the control unit 2.

The position sensor 14 detects the position (e.g., latitude and longitude) of the vehicle.
The sensor 14 receives GPS information from a global positioning system and detects the position of the vehicle based on the GPS information. The vehicle position information obtained by the position sensor 14 is transmitted to the control unit 2.

The external input unit 15 inputs information via an external network (e.g., the Internet) located outside the vehicle. The external input unit 15 receives communication information (vehicle-to-vehicle communication information) from other vehicles located around the vehicle, car navigation data from a navigation system, traffic information, high-precision map information, etc. The vehicle position information obtained by the external input unit 15 is sent to the control unit 2.

The vehicle state sensor 16 detects the vehicle state (for example, speed, acceleration, yaw rate, etc.) The vehicle state sensor 16 includes a vehicle speed sensor that detects the traveling speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, etc.
The vehicle state information obtained by the vehicle state sensor 16 is transmitted to the control unit 2 .

The driving operation sensor 17 detects driving operations applied to the vehicle and includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like.
The accelerator position sensor detects the amount of accelerator pedal operation, the steering angle sensor detects the amount of steering wheel operation, and the brake hydraulic pressure sensor detects the amount of brake pedal operation. Vehicle driving operation information obtained by the driving operation sensor 17 is sent to the control unit 2.

The biological information sensor 18 detects the driver's biological information (e.g., sweating, heart rate, etc.). The information obtained by the biological information sensor 18 is transmitted to the control unit 2.

Next, the control unit 2 will be described.
As shown in Figure 2, the control unit 2 determines the driver's abnormal state based on an evaluation value (difference) which is the distance between the gaze behavior distribution Ba, Bb, Bc based on the driver's most recent gaze behavior and the second corrected gaze distribution Ya, Yb, Yc which reflects external disturbance factors.
The second corrected gaze distributions Ya, Yb, Yc are obtained by correcting the reference gaze distributions Aa, Ab, Ac based on a standard field of view image using a first correction coefficient α (described later) to obtain first corrected gaze distributions Xa, Xb, Xc, and then correcting this first corrected gaze distribution Xa, Xb, Xc using a second correction coefficient β (learning information) (described later).

Distributions Aa, Ba, Xa, and Ya are normal distributions related to saliency (attractiveness), distributions Ab, Bb, Xb, and Yb are normal distributions related to the amplitude of saccades (saccadic eye movements), and distributions Ac, Bc, Xc, and Yc are normal distributions related to the frequency of saccades. Saliency and the frequency of saccades are indices that represent the brain's low-level processing, or the so-called latent characteristics of the driver, and saccade amplitude is an index that represents the brain's high-level processing, or the so-called explicit characteristics of the driver.
Unless otherwise specified, the reference gaze distributions Aa, Ab, and Ac will be collectively referred to as the reference gaze distribution A, the gaze behavior distributions Ba, Bb, and Bc as the gaze behavior distribution B, the first corrected gaze distribution Xa, Xb, and Xc as the first corrected gaze distribution X, and the second corrected gaze distribution Ya, Yb, and Yc as the second corrected gaze distribution Y.

As shown in FIG. 1, the control unit 2 mainly includes an image processing unit 21, an attention level detection unit 22, a field of view image creation unit 23 (field of view image creation means), a gaze behavior distribution generation unit 24 (gaze behavior distribution generation means), a correction unit 25, a learning gaze distribution generation unit 26, a learning restriction unit 27 (learning restriction means), and an abnormality estimation unit 28 (driver abnormality estimation means).

First, the image processing unit 21 will be described.
The image processing unit 21 receives and processes images captured by the multiple external cameras 11 and images captured by the internal camera 12. These captured images include images used to recognize the external environment, images used to generate a saliency map, and images used to detect the driver's line of sight.
The image processing unit 21 performs distortion correction processing to correct distortion in the image, white balance adjustment processing to adjust the white balance of the image, and the like.

Next, the caution level detection unit 22 will be described.
The caution level detection unit 22 recognizes environmental factors around the vehicle based on information output from the radar 13, the position sensor 14, and the external input unit 15. Specifically, it detects the road environment and obstacles, etc. Road environment factors mainly include the shape of intersections and T-junctions, topography such as road gradient, pavement condition, etc. Obstacle factors mainly include the presence or absence of obstacles, the type of obstacle (other vehicles, pedestrians, buildings, etc.), and the relative relationship with the obstacle (separation distance, relative speed, positional relationship, etc.).
The caution level detection unit 22 recognizes environmental factors and detects whether the level of caution of the external environment in which the vehicle is traveling is high, that is, higher than a predetermined threshold value.

Next, the field of view image creation unit 23 will be described.
The field of view image creation unit 23 creates a field of view image that appears in the field of view of the driver seated in the driver's seat, based on input from the external camera 11 and the internal camera 12.
The field of view image creation unit 23 calculates the driver's line of sight direction from the image of the driver's eyeball captured by the internal camera 12. For example, the field of view image creation unit 23 calculates the driver's line of sight direction by detecting a change in the pupil based on the state in which the driver looks into the lens of the internal camera 12.
The field of view image creating unit 23 creates a line of sight image by combining an external environment image in front of the vehicle captured by the external camera 11 with vehicle body components that are in the field of view of the driver while the vehicle is traveling.
The visual field image creating unit 23 calculates the driver's gaze point from the driver's line of sight direction and the line of sight image.

Next, the gaze behavior distribution generating unit 24 will be described.
The gaze behavior distribution generating unit 24 generates gaze behavior distributions Ba, Bb, and Bc based on the most recent gaze behavior of the driver. As shown in Fig. 3, the gaze behavior distribution B is composed of three normal distributions: a normal distribution Ba related to saliency (attractiveness), a normal distribution Bb related to the amplitude of saccades (saccadic eye movements), and a normal distribution Bc related to the frequency of saccades.
As shown in FIG. 1, the gaze behavior distribution generating unit 24 includes a saliency map generating unit 24a, an amplitude characteristic generating unit 24b, and a frequency characteristic generating unit 24c.

The saliency map creating unit 24a creates a normal distribution Ba consisting of a saliency index (AUC: Area Under Curve) and a risk potential (RP).
This saliency map creation unit 24a calculates saliency for each feature, such as color-based saliency, brightness-based saliency, and movement-based saliency, for parts other than the vehicle body components in the line-of-sight image created by the field-of-view image creation unit 23, and generates a saliency map for each feature.The final saliency map is created by adding together these generated saliency maps for each feature.

Figure 4 plots the time course of gaze saliency for a normal third-party subject. This graph plots the height of gaze saliency for each saccade, applying the gaze change to a saliency map for a given environment.
On the other hand, FIG. 5 is generated by randomly specifying coordinates (hereinafter referred to as random coordinates) from the same saliency map as the saliency map used in FIG. 4, and calculating the saliency of the random coordinates each time a saccade occurs.

Next, a receiver operating characteristic (ROC) curve is calculated between the probability of exceeding the threshold at a random point and the probability of exceeding the threshold at the gaze point of a third party.
As shown in Figure 6, the horizontal axis represents the probability of exceeding the threshold at a random point (first probability), and the vertical axis represents the probability of exceeding the threshold at a third party's gaze point (second probability), thereby representing the gaze probability relative to the random probability at the same threshold.
When the curve is convex upward like curve C1, the probability that the third party's gaze point exceeds the threshold is higher than that of a random point, and the curve is influenced by the high saliency region. When the curve is convex downward like curve C2, the probability that the third party's gaze point exceeds the threshold is lower than that of a random point, and the curve is not influenced by the high saliency region.

The saliency index (AUC) is an index whose numerical value increases as the degree of visibility of high saliency areas increases. As shown in Figure 7, the saliency index is the integral value of the lower right part of the ROC curve. By using a saliency index that is compared with random points, it is possible to reduce dependency on the driving scene, such as the number and spread of high saliency areas.

The risk potential (RP) is an artificially set field that appropriately reflects the degree of danger in the vehicle's driving environment (the driver's sense of danger). For example, an appropriate function is set that has a shape that is maximum at the center position of each surrounding vehicle relative to the surrounding vehicles and spreads out around each surrounding vehicle. Methods for determining the risk potential may also use the following distance to surrounding vehicles, TTC (Time To Collision), road alignment, distance to a line change point, etc.

Then, the time series distribution of the saliency index in a predetermined environment is calculated (see FIG. 8).
In addition, the time series distribution of the risk potential in the same environment as the time series distribution of the saliency index is obtained (see FIG. 9). As shown in FIG. 10, the time series distribution of the saliency index and the time series distribution of the risk potential for the same predetermined time (e.g., 30 seconds) are combined.
The combined time series distribution of the saliency index and the risk potential is equally divided into predetermined risk potential widths, and the probability distribution of the saliency index in each divided region (hereinafter referred to as a bin) is calculated.

As shown in Figure 11, if the number of divided regions is, for example, 10, the probability distribution of the saliency index in bin 1 to the probability distribution of the saliency index in bin 10 are all added together to calculate the final normal distribution Ba (the driver's most recent normal distribution Ba).

The amplitude characteristic generating unit 24b generates a normal distribution Bb consisting of amplitude, which is one of the saccade indices, and risk potential (RP).
A saccade is a saccadic eye movement in which a driver involuntarily and reflexly moves their line of sight, and is an eye movement in which the line of sight moves from a fixation point where the line of sight remains stationary for a predetermined period of time to the next fixation point.
12, a period sandwiched between adjacent fixation periods is a saccade period. The amplitude ds of the saccade is the distance the line of sight moves during the saccade period.

A period (e.g., 0.1 seconds) during which the gaze movement speed is less than a predetermined speed threshold (e.g., 2 deg/s) and continues for a predetermined stagnation time is extracted as a gaze period, and gaze movements in the period between adjacent gaze periods that have a movement speed equal to or greater than the speed threshold (e.g., 40 deg/s) and a movement distance equal to or greater than the distance threshold (e.g., 3 deg) are extracted as saccade candidates.

As shown in FIG. 13, a regression curve L10 is derived using multiple saccade candidates, for example, using the least squares method. Next, a first reference curve L11 is derived by shifting the regression curve L10 in the direction of decreasing movement speed, and a second reference curve L12 is derived by shifting the regression curve L10 in the direction of increasing movement speed. Then, the area between the first reference curve L11 and the second reference curve L12 is defined as a saccade range R10, and saccade candidates included in this saccade range R10 are extracted as selected saccades.

The amplitude characteristic generating unit 24b obtains a time series distribution (not shown) of the amplitude of the saccade index in a predetermined environment, as in the case of the normal distribution Ba. It also obtains a time series distribution (not shown) of the risk potential in the same environment as the time series distribution of the saccade index.
The time series distribution of the saccade index and the time series distribution of the risk potential for the same predetermined time (e.g., 30 seconds) are combined. The combined time series distribution of the saccade index and the risk potential is equally divided into predetermined risk potential widths, and the probability distribution of the saccade index for each divided region is calculated. If the number of divided regions is, for example, 10, the probability distributions of the saccade index for bin 1 to bin 10 are all combined to calculate a normal distribution Bb for saccades.

The frequency characteristic creation unit 24c creates a normal distribution Bc consisting of a frequency, which is one of the saccade indices, and a risk potential (RP). The frequency, which is one of the saccade indices, is a value obtained by dividing the number of saccades in a predetermined time (e.g., 30 seconds) by the predetermined time.
As with the normal distribution Ba, this frequency characteristic creation unit 24c calculates a normal distribution Bc for saccades using a time series distribution (not shown) relating to the frequency of saccade indicators in a specified environment and a time series distribution (not shown) of risk potentials.

Next, the correction unit 25 will be described.
The correction unit 25 calculates the first corrected gaze distribution X by correcting the reference gaze distribution A using the degree of influence of the first disturbance factor caused by the environment and vehicle characteristics.
The basic reference gaze distribution A is created in advance before the vehicle is shipped from the factory, specifically, by using the gaze behavior of a normal third party through a preliminary experiment, etc. Furthermore, a reference gaze distribution generating means for generating the reference gaze distribution A is provided in the testing site where the preliminary experiment is conducted.
The data (reference gaze distribution A, first correction coefficient α) created in advance is stored in the vehicle in advance.

Human gaze behavior has the characteristic of changing under the influence of N factors, mainly environmental factors, vehicle factors, and human factors, among external disturbance factors. The accuracy of estimating the driver's state changes due to the influence of these N factors. The first external disturbance factors are environmental factors and vehicle factors, which are less influenced by individual differences among the N factors. Examples of environmental factors are as follows:
Roads (road type, shape, number of lanes, number)
Road boundaries (road: type, relative position, TTC)
Local traffic (type: number, relative position, TTC)
Pedestrians (type: number, relative position, TTC)
Structures (blind spots, predicted collision distance)
Weather (sunny, rainy)
Lighting (day and night)
In this embodiment, the number of pedestrians, the degree of congestion of other vehicles, etc. are considered as one of the environmental factors that pose a risk.

The vehicle factors are, for example, as follows:
Vehicle behavior (vehicle speed, acceleration, deceleration, yaw rate)
Vehicle characteristics (response to operation/road input, window frame structure (shape/decoration, instrument panel depth)
Eye point (height)
HMI (HUD and other layouts)
The environmental factors and vehicle factors are set by selecting one or more factors from the above items, and the following processing is performed for the set factors.

The correction unit 25 takes into consideration the levels of the environmental factors and vehicle factors and calculates the first correction coefficients α (αa, αb, αc) according to the environmental factors and vehicle factors using a known statistical method.
After calculating the first correction coefficient α, the correction unit 25 obtains the first corrected gaze distributions Xa, Xb, and Xc by the following equation.
X = α × A ... (1)
Since the first correction coefficient α can be pre-calculated by arbitrarily changing the environmental factors and vehicle factors, in this embodiment, the reference gaze distribution A, the first correction coefficient α, and the first corrected gaze distribution X are stored in the vehicle before it is shipped from the factory.

Next, the learning gaze distribution generating unit 26 will be described.
The learning gaze distribution generation unit 26 generates the second corrected gaze distribution Y using the second correction coefficient β related to the human factors of the driver. As shown in Fig. 1, the learning gaze distribution generation unit 26 includes a learning information generation unit 26a. This learning information generation unit 26a generates the second correction coefficient β that associates the physical state of the vehicle, the characteristics of the driver, and the gaze behavior of the driver based on the field of view image.
Among the N factors, human factors have the characteristic of being highly influenced by individual differences.

Human factors are, for example:
Physical condition (illness, fatigue, alertness, drunkenness)
Psychological (pleasure/discomfort; anxiety/tension/excitement, activity, expectation)
Experience (immediate short-term, long-term)
Driving skills (mental and physical functions, metacognition)
Driving style (values, personality, preferences)
Driving awareness (safety awareness, driving motivation, self-awareness)
Intention (driving purpose, presence of other passengers)
Culture Age (young/old, values)
Gender One or more human factors are set from the above items, and the following processing is performed for the set factors.

The learning information generating unit 26a calculates the second correction coefficients β (βa, βb, βc) according to the human factors by taking into account the levels of the human factors and using a known statistical method.
After calculating the second correction coefficient β, the learning gaze distribution generating unit 26 obtains second corrected gaze distributions Ya, Yb, Yc by the following equation.
Y = β × X ... (2)

Next, the learning limiting unit 27 will be described.
The learning restriction unit 27 is configured to perform learning based on the field of view image when the reliability N of the learning data is equal to or greater than a predetermined judgment threshold, and to restrict learning based on the field of view image when the reliability N of the learning data is less than the judgment threshold.
The learning restriction unit 27 has a reliability setting unit 27a (reliability setting means).

The reliability setting unit 27a is configured to set the reliability N of the field of view image included in the driver's field of view as learning data using the first reliability index Ne and the second reliability index Nc.
The reliability setting unit 27a calculates a first reliability index Ne as learning data of the field of view image based on a first disturbance factor related to the surrounding conditions of the vehicle acquired by the surrounding information acquisition means.
Specifically, when the risk is Ne1, the road type is Ne2, the number of lanes is Ne3, and the coefficients are Ae1 to Ae3, the first reliability index Ne is calculated by the following formula.
Ne=Ae1×Ne1+Ae2×Ne2+Ae3×Ne3…(3)

The reliability setting unit 27a also calculates a second reliability index Nc as learning data of the field of view image based on a second disturbance factor related to the physical state of the vehicle acquired by the vehicle state acquisition means.
Specifically, when the vehicle speed is Nc1, the acceleration is Nc2, the yaw rate variation is Nc3, and the coefficients are Ac1 to Ac3, the second reliability index Nc is calculated by the following formula.
Nc=Ac1×Nc1+Ac2×Nc2+Ac3×Nc3…(4)

As shown in FIG. 14, the reliability setting unit 27a has a reliability setting table.
In the reliability setting table, the first reliability index Ne and the second reliability index Nc are each classified into levels (for example, large, medium, and small).
The first reliability index Ne is set so that the value of the medium level is larger than the value of the large and small level, because the personal characteristic of the medium level is smaller than the personal characteristic of the large and small level.
The second reliability index Nc is set so that the value of the medium level is larger than the value of the large and small level, because the personal characteristic of the medium level is smaller than the personal characteristic of the large and small level.

The reliability setting unit 27a sets the reliability N using the level of the first reliability index Ne, the level of the second reliability index Nc, and the reliability setting table.
When the reliability N is equal to or greater than a predetermined judgment threshold (e.g., 0.50), the learning limiting unit 27 executes learning because the reliability N of the learning data related to the human factors is high. On the other hand, when the reliability N is less than the judgment threshold, the learning limiting unit 27 limits learning because the reliability N of the learning data related to the human factors is low.

Next, the abnormality estimation unit 28 will be described.
The abnormality estimation unit 28 is configured to be able to determine whether the driver is abnormal or not. The abnormality estimation unit 28 includes a learning progress calculation unit 28a. The learning progress calculation unit 28a calculates the progress of learning about the individual characteristics of the driver, i.e., the learning progress St.
This learning progress level St is calculated using the following formula, where F1 is the number of frames of image data that are necessary and sufficient to identify the driver's personal characteristics, and F2 is the number of image frames related to the driver that were used to generate the second corrected gaze distribution Y.
St=F2/F1...(5)

The abnormality estimation unit 28 has three determination functions to determine whether the driver is abnormal: a first abnormality determination function using the deviation degree of the probability distribution, a second abnormality determination function using the saliency index (see Figure 7), and a third abnormality determination function using the degree of caution.

The first abnormality determination function determines that the driver is in an abnormal state when the evaluation value, which is the difference between the driver's most recent (e.g., 30-second period) gaze behavior distribution B and the second corrected gaze distribution Y, is equal to or greater than a determination threshold. In this embodiment, as shown in FIG. 2, the abnormality estimation unit 26 calculates the degree of deviation between the gaze behavior distributions Ba, Bb, and Bc and the second corrected gaze distributions Ya, Yb, and Yc as evaluation values using the Kullback-Leibler divergence.

The evaluation value Da of the distributions Ba and Ya, the evaluation value Db of the distributions Bb and Yb, and the evaluation value Dc of the distributions Bc and Yc are calculated, and then substituted into the following equation to calculate the total evaluation value D.
D=W×(Da+Dc)+(1-W)×Db…(6)
Incidentally, W (0<W<1) is a weighting coefficient.
The total evaluation value D is set so that when the learning progress level St is small (the learning period is short), the weights of the evaluation values Da and Dc are greater than the weight of the evaluation value Db, and when the learning progress level St is large (the learning period is long), the weights of the evaluation values Da and Dc are smaller than the weight of the evaluation value Db. When the total evaluation value D is equal to or greater than a determination threshold value (e.g., 0.6), the driver's condition is determined to be abnormal.

The weighting coefficient W can be calculated by the following equation, where P, Q, and R are coefficients.
W=1/(1+e ^-P×(St×QR) ) …(7)
The weighting coefficient W is a monotonically increasing function that is point-symmetric like a sigmoid function, which is one of the activation functions, and the coefficients P, Q, and R can be set arbitrarily.

The second abnormality determination function estimates that the driver's attention is declining when the saliency index (integral value of the lower right part of the ROC curve) is equal to or greater than a predetermined determination threshold.
The third abnormality judgment function estimates that the driver's attention has decreased if the attention level detection unit 22 detects a high level of attention and the saccade amplitude ds has decreased significantly, and estimates that the driver's attention has decreased if the attention level detection unit 22 detects a low level of attention and the frequency of saccades has decreased significantly.

An example of the driver's state determination process will be described below based on the flowcharts in Figures 15 to 19. Note that in the figures, Si (i = 1, 2, ...) indicates each step.

An example of the advance learning process is shown in the flowchart of Fig. 15. The advance learning process is basically performed before the vehicle is shipped from the factory, for example, at a testing site.
First, various information such as information from the information acquisition unit 1 and various experimental results is read (S1), and the process proceeds to S2. In S2, the standard saliency index and standard saccade index (amplitude, frequency) for a normal third person are calculated, and then the process proceeds to S3.

In S3, it is determined whether or not sufficient data regarding the standard saliency index and the standard saccade index for a normal third person has been secured.
If the result of the determination in S3 is that sufficient data is secured, the reference gaze distributions Aa, Ab, and Ac are calculated (S4), and the process proceeds to S5. If the result of the determination in S3 is that sufficient data is not secured, the process returns to S1.

In S5, the first corrected gaze distribution X is obtained, and the first correction coefficient α is calculated using equation (1), and then the process proceeds to S6.
Since the first corrected gaze distribution X corresponds to an index of a normal state, the probability distribution corresponding to the standard saliency index and the probability distribution corresponding to the standard saccade index correspond to the first corrected gaze distribution X. Therefore, the first correction coefficient α is calculated by substituting the reference gaze distribution A and the probability distribution corresponding to the standard saliency index and the probability distribution corresponding to the standard saccade index, which correspond to the first corrected gaze distribution X, into equation (1).
In S6, the reference gaze distribution A and the first correction coefficient α are stored in the control unit 2, and then the process ends.

Next, an example of a process for determining the state of the driver will be shown with reference to the flowchart of FIG.
The state determination process is performed when the driver operates the vehicle.
First, various information such as information from the information acquisition unit 1, the reference gaze distribution A, and the first correction coefficient α is read (S11), and the process proceeds to S12. In S12, the current saliency index and the current saccade index (amplitude, frequency) are calculated, and then the process proceeds to S13.

In S13, it is determined whether the second correction coefficient β is held.
If the determination result in S13 is that the second correction coefficient β is held, the second corrected gaze distribution Y can be calculated, and the process proceeds to S14. In S14, the reference gaze distribution A and the first correction coefficient α are substituted into equation (1) to calculate the first corrected gaze distribution X, and then the first corrected gaze distribution X and the second correction coefficient β are substituted into equation (2) to determine the second corrected gaze distribution Y.

In S15, the difference between gaze behavior distribution B and second corrected gaze distribution Y is calculated, and the process proceeds to S16. The difference between gaze behavior distribution B and second corrected gaze distribution Y is calculated as a deviation (separation distance) using Kullback-Leibler divergence. Each deviation is represented by an evaluation value Da for distributions Ba and Ya, an evaluation value Db for distributions Bb and Yb, and an evaluation value Dc for distributions Bc and Yc.

In S16, the final total evaluation value D is calculated using the evaluation values Da, Db, and Dc obtained in S15, and the process proceeds to S17.
In S17, it is determined whether the total evaluation value D is equal to or greater than the determination threshold value.
If the result of the judgment in S17 is that the total evaluation value D is equal to or greater than the judgment threshold, the gaze behavior distribution B representing the driver's current state is deviated from the second corrected gaze distribution Y, which corresponds to an indicator of a normal state, and therefore an abnormality judgment is made that the driver's state is abnormal (S18), and the process ends.
If the result of the judgment in S17 is that the total evaluation value D is less than the judgment threshold, the gaze behavior distribution B representing the driver's current state is close to the second corrected gaze distribution Y, which corresponds to an indicator of a normal state, so a normal judgment is made that the driver's state is normal (S19), and the process ends.

If the result of the determination in S13 is that the second correction coefficient β is not held at the time of the determination, the second corrected gaze distribution Y cannot be calculated, and the process proceeds to S20.
In S20, since the second correction coefficient β is not held, the driver's condition is estimated using biometric indicators other than saliency and saccades related to gaze behavior, such as heart rate and blood pressure, and then the process proceeds to S21.

In S21, it is determined whether the driver's condition is normal or not using a biological information index other than saliency and saccades related to gaze behavior.
If the result of the determination in S21 is that the driver is in a normal state, the process proceeds to S22 to acquire the second correction coefficient β. In S22, the gaze behavior distribution B is accumulated and learned until a highly accurate second correction coefficient β can be acquired, and then the process ends. If the result of the determination in S21 is that the driver is not in a normal state, the second correction coefficient β cannot be acquired, and the process ends.

Next, an example of the total evaluation value calculation process (S16) will be shown with reference to the flowchart of FIG.
The total evaluation value calculation process is performed when calculating the total evaluation value D from the gaze behavior distribution B and the second corrected gaze distribution Y. First, the number of image frames F1 necessary and sufficient for identifying the driver's personal characteristics and the number of image frames F2 used to generate the second corrected gaze distribution Y are substituted into equation (5) to calculate the learning progress level St (S31), and the process proceeds to S32.

In S32, it is determined whether the learning progress level St is equal to or greater than a predetermined determination threshold value.
If the result of the determination in S32 is that the learning progress level St is equal to or greater than a preset determination threshold, the image data necessary and sufficient to identify the driver's personal characteristics has been acquired, and highly accurate learning has already been performed, so the process proceeds to S33. If the result of the determination in S32 is that the learning progress level St is less than the preset determination threshold, the image data necessary and sufficient to identify the driver's personal characteristics has not been acquired, and information on human factors related to the driver is insufficient, so the process returns to S31.

In S33, the weighting coefficient W is calculated using the learning progress level St and equation (7), and the process proceeds to S34. In S34, the total evaluation value D is calculated using the weighting coefficient W, evaluation values Da, Db, and Dc, and equation (6), and the process ends.

Next, an example of the learning phase process (S22) will be shown with reference to the flowchart of FIG.
The learning phase process is performed when learning the gaze behavior distribution B in order to acquire the second correction coefficient β. First, various information such as information from the information acquisition unit 1, the reference gaze distribution A, and the first correction coefficient α is read (S41), and the process proceeds to S42. In S42, the current saliency index and the current saccade index (amplitude, frequency) are calculated, and then the process proceeds to S43.

In S43, it is determined whether the driver's condition is normal.
If the result of the determination in S43 is that the driver's condition is normal, data on gaze behavior distribution B is accumulated in order to learn gaze behavior distribution B, and the process proceeds to S44.
If the result of the determination in S43 is that the driver's condition is not normal, data regarding the gaze behavior distribution B cannot be accumulated, and the process returns to S41.
In S44, the reliability setting unit 27a sets the reliability N of the image data to be stored, and the learning limiting unit 27 limits the learning of the image data to be stored in accordance with the reliability N, and the process proceeds to S45.

In S45, it is determined whether or not a necessary and sufficient amount of data regarding the gaze behavior distribution B has been secured in calculating the second correction coefficient β.
If the result of the determination in S45 is that the amount of data related to gaze behavior distribution B has been secured, the process proceeds to S46. If the result of the determination in S45 is that the amount of data related to gaze behavior distribution B has not been secured, the process returns to S41 to secure more data.

In S46, the second correction coefficient β is calculated, and after saving the second correction coefficient β (S47), the process ends. Since the second corrected gaze distribution Y corresponds to an index of the normal state, the gaze behavior distribution B in the normal state corresponds to the most recent second corrected gaze distribution Y. Therefore, the second correction coefficient β is calculated by substituting the first corrected gaze distribution X (reference gaze distribution A, first correction coefficient α) and the gaze behavior distribution B corresponding to the most recent second corrected gaze distribution Y into equation (2).

Next, an example of the reliability determination process (S44) will be shown with reference to the flowchart of FIG.
The reliability determination process determines whether or not the stored image data can be stored as learning data for a determination index.

First, various information such as a reliability setting table is read (S51), and the process proceeds to S52.
In S52, the reliability setting unit 27a calculates the first reliability index Ne using equation (3) and calculates the second reliability index Nc using equation (4), and then proceeds to S53.
In S53, the reliability N is set using the first and second reliability indices Ne and Nc and the reliability setting table, and then the process proceeds to S54.

In S54, it is determined whether the reliability N is equal to or greater than a determination threshold.
If the result of the determination in S54 is that the reliability N is equal to or greater than the determination threshold, the situation is not complex in which personal characteristics are significantly reflected, so the image data is accumulated as learning data for the determination index (S55), and the process ends.If the result of the determination in S54 is that the reliability N is less than the determination threshold, the situation is complex in which personal characteristics are significantly reflected, so the image data is discarded without being used as learning data for the determination index (S56), and the process ends.

Next, the operation and effects of the driver condition determination system S will be described.
This driver condition determination method includes a driver abnormality estimation process S17 that compares the gaze behavior distribution B with a reference gaze distribution A corrected using a second correction coefficient β, which is learning information, to determine whether the driver is abnormal, and therefore the driver's abnormal condition can be estimated based on the gaze behavior.
Since the system has a reliability setting process S53 that sets the reliability N of the field of view image included in the driver's field of view and used as learning data for generating the second correction coefficient β based on the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle conditions acquired from the vehicle's physical conditions, it is possible to set the reliability N of the learning data that represents the driver's personal characteristics for the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle conditions acquired from the vehicle's physical conditions.
When the reliability N of the learning data is equal to or greater than the judgment threshold, learning based on the field of view image is executed in step S55, and when the reliability N of the learning data is less than the judgment threshold, learning based on the field of view image is restricted in step S56. This makes it possible to restrict learning in situations where personal characteristics become significant, thereby ensuring the accuracy of state judgment while shortening the learning period.

The system includes a first reliability index calculation step S52 for calculating a first reliability index Ne as learning data for the field of view image based on a first disturbance factor related to the vehicle's surrounding conditions acquired in the surrounding information acquisition step, and a second reliability index calculation step S52 for calculating a second reliability index Nc as learning data for the field of view image based on a second disturbance factor related to the vehicle's physical state acquired in the vehicle state acquisition step. The reliability setting step S53 sets the reliability N of the field of view image included in the driver's field of view as learning data using the first reliability index Ne and the second reliability index Nc. This allows the reliability N of the learning data to be accurately set from the perspectives of both environmental and vehicle factors by calculating the first reliability index Ne, which is the reliability N of the learning data related to environmental factors, and the second reliability index Nc, which is the reliability of the learning data related to vehicle factors.

This driver condition determination system S has an abnormality estimation unit 28 that compares the gaze behavior distribution B with a reference gaze distribution A corrected using a second correction coefficient β, which is learning information, to determine whether the driver is abnormal, and therefore can estimate whether the driver is in an abnormal state based on the gaze behavior.
The reliability setting unit 27a sets the reliability N of the field of view image included in the driver's field of view and used to generate the second correction coefficient β as learning data based on the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle state obtained from the vehicle's physical state, so it is possible to set the reliability N of the learning data representing the driver's personal characteristics for the traffic conditions estimated based on the vehicle's surrounding conditions and the vehicle state obtained from the vehicle's physical state.
The system has a learning restriction unit 27 that executes learning based on field of view images when the reliability of the learning data is equal to or greater than the judgment threshold, and restricts learning based on field of view images when the reliability N of the learning data is less than the judgment threshold. This makes it possible to restrict learning in situations where personal characteristics become significant, thereby ensuring the accuracy of state judgment while shortening the learning period.

Next, a modified example in which the above embodiment is partially modified will be described.
1) In the above embodiment, an example was described in which the reference gaze distribution A was corrected using the first correction coefficient α that is due to the environment and vehicle characteristics. However, the reference gaze distribution A may also be corrected using the first correction coefficient α that is due only to the environment, or the reference gaze distribution A may also be corrected using the first correction coefficient α that is due only to the vehicle characteristics.

2) In the above embodiment, an example was described in which the reference gaze distribution generating means was provided at a testing site where preliminary experiments were conducted, but the reference gaze distribution generating means may also be provided in the vehicle's control unit 2.

3) In the above embodiment, an example was described in which the learning progress level St was calculated using the number of image frames and equation (3), but it is sufficient to be able to detect at least the learning period, and the learning progress level St may also be calculated using the travel distance or driving time as a parameter.

4) In addition, a person skilled in the art may implement the above-described embodiments in various modified forms without departing from the spirit of the present invention, and the present invention also encompasses such modifications.

11 External camera 12 Internal camera 16 Vehicle state sensor 17 Driving operation sensor 23 Field of view image creation unit 24 Gaze behavior distribution generation unit 26 Learning gaze distribution generation unit 26a Learning information generation unit 27 Abnormality estimation unit 27a Learning progress calculation unit 28 Abnormality estimation unit S Driver state determination system

Claims (1)

A driver state determination method for determining the state of a driver of a vehicle, comprising:
a surrounding information acquisition step of acquiring information about a surrounding situation of the vehicle;
a vehicle state acquisition step of acquiring a physical state of the vehicle;
a visual field image creation step of creating a visual field image that appears in the driver's field of view;
a reference gaze distribution generating step of generating a reference gaze distribution based on a standard field of view image in advance;
a gaze behavior distribution generating step of generating a gaze behavior distribution based on the gaze behavior of the driver;
a learning information generating step of generating learning information that associates a surrounding situation of the vehicle, a physical state of the vehicle, and a gaze behavior of the driver;
a driver abnormality inference step of comparing the gaze behavior distribution with a reference gaze distribution corrected using the learning information to determine whether the driver is abnormal;
a reliability setting step of setting the reliability of a field of view image included in the driver's field of view and used to generate the learning information as learning data based on a traffic situation estimated based on a surrounding situation of the vehicle and a vehicle state acquired from a physical state of the vehicle;
a learning limiting step of executing learning based on the field of view image when the reliability as the learning data is equal to or greater than a judgment threshold, and limiting learning based on the field of view image when the reliability as the learning data is less than the judgment threshold ;
a first reliability index calculation step of calculating a first reliability index as learning data of the field of view image based on a first disturbance factor related to the surrounding conditions of the vehicle acquired in the surrounding information acquisition step;
a second reliability index calculation step of calculating a second reliability index as learning data of the field of view image based on a second disturbance factor related to the physical state of the vehicle acquired in the vehicle state acquisition step,
A driver state determination method characterized in that the reliability setting step uses the first reliability index and the second reliability index to set the reliability of the field of view image included in the driver's field of view as learning data .