JP7785194B2 - Abnormal attitude detection device, abnormal attitude detection method, and vehicle control system - Google Patents

Description

This disclosure relates to an abnormal posture detection device, an abnormal posture detection method, and a vehicle control system.

Conventionally, there is known technology for detecting abnormal occupant posture by determining whether the occupant is misaligned based on features such as facial orientation extracted from captured images of the occupant inside a vehicle and a trained model in machine learning (hereinafter referred to as the "machine learning model") (for example, Patent Document 1).

International Publication No. 2017/208529

By using a machine learning model to determine whether an occupant's posture is out of alignment, it is possible to detect abnormal occupant posture with high accuracy. On the other hand, in a scene where feature values such as facial orientation cannot be stably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values and the machine learning model. In this case, when determining whether an occupant's posture is out of alignment using the machine learning model, there is a possibility that the occupant may be determined to be in a state of posture out of alignment even though they are not.
In conventional technology, in scenes where features such as facial orientation cannot be reliably extracted from captured images, the presence or absence of an occupant's posture is determined based on low-precision features and a machine learning model, which can result in the risk of falsely detecting an abnormal occupant's posture.
However, the technology disclosed in Patent Document 1 still cannot solve the above problem because it does not take into account the possibility that when determining whether or not an occupant's posture is unstable using a machine learning model in a scene where features cannot be stably extracted from the captured image, the occupant's posture may be erroneously determined to be unstable.

This disclosure has been made to solve these problems, and aims to provide an abnormal posture detection device that can detect abnormal postures of occupants even in situations where features cannot be stably extracted from captured images.

The abnormal posture detection device according to the present disclosure includes an image acquisition unit that acquires an image capturing an area in which the face of a vehicle occupant should be located, a feature extraction unit that extracts features to be used in determining the posture of the occupant based on the image acquired by the image acquisition unit, a change amount calculation unit that calculates a change amount of the feature extracted by the feature extraction unit from the feature in a reference posture of the occupant , a first determination unit that determines whether the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change amount calculated by the change amount calculation unit with a normal posture determination condition , or by comparing the feature extracted by the feature extraction unit with a normal posture determination condition, a second determination unit that, if the first determination unit determines that the posture of the occupant is not normal, determines whether the occupant has lost their posture based on the change amount calculated by the change amount calculation unit and a machine learning model that receives the change amount as an input and outputs information indicating whether the occupant has lost their posture, and a detection unit that detects an abnormal posture of the occupant when the state in which the second determination unit has determined that the occupant has lost their posture continues for an abnormality detection time.

The abnormal posture detection device disclosed herein can detect an occupant's abnormal posture even in situations where features cannot be stably extracted from captured images.

1 is a diagram illustrating a configuration example of an abnormal posture detection device according to a first embodiment; FIG. 10 is a diagram for explaining the significance of the first determination unit determining whether the driver's posture is normal or not based on the feature amount based on the current captured image and the relative change in the feature amount in embodiment 1. 10A and 10B are diagrams for explaining a plurality of types of poor posture. 1 is a diagram illustrating a configuration example of an operation control system according to a first embodiment. 10 is a flowchart for explaining an operation in a reference posture feature value calculation process performed by the abnormal posture detection device according to the first embodiment. 10 is a flowchart for explaining operations in a change amount calculation process, a first attitude determination process, a second attitude determination process, and an abnormal attitude detection process performed by the abnormal attitude detection device according to the first embodiment. 7 is a flowchart for explaining detailed operations of a first attitude determination process by a first determination unit, which is performed in step ST14 of FIG. 6; 7 is a flowchart for explaining detailed operations of the second attitude determination process by the second determination unit, which is performed in step ST15 of FIG. 6. 7 is a flowchart for explaining detailed operations of an abnormal posture detection process by a detection unit, which is performed in step ST16 of FIG. 6; 10A and 10B are diagrams for explaining an example of a scene in which feature amounts such as facial orientation cannot be stably extracted from a captured image. 4 is a flowchart for explaining an example of an operation of a driving assistance function by the driving control system according to the first embodiment. 12A and 12B are diagrams illustrating an example of a hardware configuration of an abnormal posture detection device according to the first embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
Embodiment 1.
The abnormal posture detection device according to the first embodiment detects whether the posture of a vehicle occupant is abnormal based on a captured image in which at least the face of the vehicle occupant is captured.
In the following first embodiment, as an example, the occupant of the vehicle for which the abnormal posture detection device detects whether or not the vehicle is in an abnormal posture is the driver of the vehicle.

FIG. 1 is a diagram illustrating an example of the configuration of an abnormal posture detection device 1 according to the first embodiment.
The abnormal attitude detection device 1 according to the first embodiment is assumed to be mounted on a vehicle.
The abnormal posture detection device 1 is connected to an imaging device 2 , and the abnormal posture detection device 1 and the imaging device 2 together constitute an abnormal posture detection system 100 .
The abnormal attitude detection system 100, together with the driving control system 300, the steering mechanism 3, and the braking/driving mechanism 4, constitutes a vehicle control system SYS.A.

The imaging device 2 is mounted on a vehicle and is installed so as to be able to capture an image of at least the area where the driver's face should be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the abnormal posture detection device 1.
The abnormal posture detection device 1 detects whether or not the posture of the driver is abnormal based on the captured image captured by the imaging device 2. Details of the abnormal posture detection device 1 will be described later.

The abnormal posture detection device 1 outputs the detection result of whether the driver's posture is abnormal or not to the driving control system 300 connected to the abnormal posture detection system 100.
1, the abnormal attitude detection system 100 is connected to a driving control system 300. The driving control system 300 is linked to the steering mechanism 3 or the braking/driving mechanism 4, and provides driving assistance using the detection results of the abnormal attitude detection system 100. Details of the driving control system 300 will be described later.

As shown in Figure 1, the abnormal posture detection device 1 includes an image acquisition unit 11, a feature extraction unit 12, a reference posture feature calculation unit 13, a change amount calculation unit 14, a first judgment unit 15, a second judgment unit 16, a detection unit 17, and a memory unit 18.

The image acquisition unit 11 acquires a captured image from the imaging device 2 .
The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12 .

The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11 .
In the first embodiment, the feature amounts used to determine the driver's posture are the driver's facial direction, the driver's head position, and the face detection reliability.

The feature extraction unit 12 first detects the driver's face in the captured image acquired by the image acquisition unit 11 .
In detail, the feature extraction unit 12 detects the driver's facial feature points, which indicate the driver's facial features, using a known image recognition technique such as edge detection on the captured image. Note that the facial features include the corners and corners of the eyes, the nose, the mouth, the eyebrows, and the chin. The driver's facial feature points are represented, for example, by coordinates on the captured image. The feature extraction unit 12 may also detect the driver's facial region. The driver's facial region is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial region is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image.
Since the installation position and angle of view of the image capturing device 2 are known in advance, even if a captured image contains multiple occupants, the feature extraction unit 12 can determine which area of the captured image contains the driver's face. For example, an area in the captured image where the driver's face may be present (hereinafter referred to as the "driver detection area") is set in advance, and the feature extraction unit 12 detects the feature points and face area of the driver's face in the driver detection area using a known image recognition technique.

The feature extraction unit 12 detects the driver's facial orientation and head position based on detected information about the driver's face (hereinafter referred to as "face information") and extracts this as a feature. Face information is, for example, a captured image to which information that can identify the driver's facial feature points and facial area is added.

In the first embodiment, the facial direction and head position of the driver detected by the feature amount extracting unit 12 are, for example, the facial direction and head position in real space.
The feature extraction unit 12 detects the direction of the driver's face in real space based on the face information, more specifically, based on the driver's face in the captured image.
The feature extraction unit 12 may detect the driver's facial direction using, for example, a known facial direction detection technique for detecting the facial direction from a captured image. The driver's facial direction is expressed, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to the reference axis, with the optical axis of the image capture device 2 as the reference axis. Note that this is merely an example, and the driver's facial direction may also be expressed, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to the reference axis, with a predetermined reference axis in the longitudinal direction of the vehicle as the reference axis.

Furthermore, the feature extraction unit 12 detects the driver's head position in real space based on the facial information, more specifically, based on the driver's face in the captured image. In the first embodiment, the driver's head position in the captured image is indicated, for example, by the center of the driver's eyebrows. The feature extraction unit 12 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the driver's head position in the captured image may be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the feature extraction unit 12 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space.
The feature extraction unit 12 may detect the driver's head position by using, for example, a known coordinate transformation technique for transforming points on the captured image into points in real space. The driver's head position is represented by, for example, coordinates in real space.

Furthermore, the feature extraction unit 12 calculates the face detection reliability based on the face information and sets this as the extracted feature.
In embodiment 1, face detection reliability indicates how reliable a face detected from a captured image is, i.e., how plausible it is as a face, or more specifically, how many facial features were detected from the captured image.
The feature extraction unit 12 calculates the face detection reliability in accordance with predetermined conditions that are set in advance.
For example, the feature extraction unit 12 calculates the percentage of facial parts detected based on the captured image acquired by the image acquisition unit 11 out of all parts defined as facial parts, such as the outer corners of the eyes, the inner corners of the eyes, the nose, the mouth, the eyebrows, or the chin, as the face detection reliability.

The feature extraction unit 12 outputs information regarding the driver's facial orientation, head position, and face detection reliability extracted as features (hereinafter referred to as "feature information") to the reference posture feature calculation unit 13 and the change amount calculation unit 14, and also stores the information in chronological order in the memory unit 18.
The feature information is information that associates information on the driver's face orientation relative to a reference axis (yaw angle, pitch angle, and roll angle), coordinate information on the driver's head position (X coordinate, Y coordinate, Z coordinate), information indicating the face detection reliability, and the captured image.

The reference posture feature amount calculation unit 13 estimates the reference posture of the driver and calculates the feature amount of the reference posture of the driver (hereinafter referred to as the "reference posture feature amount").
In the first embodiment, the reference posture refers to the posture of the driver in a state where the driver is considered to be facing forward with respect to the traveling direction of the vehicle. Note that in the first embodiment, the driver facing forward with respect to the traveling direction of the vehicle is assumed to mean that the driver's face is facing forward with respect to the traveling direction of the vehicle. In the first embodiment, "forward" is not limited to strictly being forward, but includes approximately forward.
The process of estimating the driver's reference posture and calculating the feature amounts in the reference posture performed by the reference posture feature amount calculation unit 13 in the abnormal posture detection device 1 according to the first embodiment is referred to as "reference posture feature amount calculation process."

The reference posture feature calculation unit 13 determines whether the driver is considered to be facing forward in the direction of travel of the vehicle, for example, by determining whether the vehicle speed, steering wheel steering angle, and shift position satisfy predetermined conditions (hereinafter referred to as "vehicle conditions"), and estimates the driver's reference posture.
The vehicle conditions are preset, for example, such that "the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km/h), the steering wheel angle is within a predetermined angle range (e.g., ±20 degrees), and the shift position is in "D." The vehicle conditions are set to the vehicle speed, steering wheel angle, and shift position assumed in a situation in which the driver is assumed to be facing forward in the direction of travel of the vehicle.

When the vehicle speed, the steering wheel angle, and the shift position satisfy the vehicle conditions, the reference posture feature amount calculation unit 13 determines that the driver is facing forward in the direction of travel of the vehicle. In other words, when the vehicle speed, the steering wheel angle, and the shift position satisfy the vehicle conditions, the reference posture feature amount calculation unit 13 estimates that the driver's posture is the reference posture.
The reference posture feature value calculation unit 13 may acquire information on the vehicle speed from a vehicle speed sensor C62 (see FIG. 4 , which will be described later) mounted on the vehicle, information on the steering wheel steering angle from a steering torque sensor C63 (see FIG. 4 , which will be described later) mounted on the vehicle, and information on the shift position from a shift position sensor (not shown) mounted on the vehicle.

The above-described method of determining whether or not the driver is considered to be facing forward with respect to the traveling direction of the vehicle by the reference posture feature amount calculation unit 13 is merely an example. The reference posture feature amount calculation unit 13 may use other methods to determine whether or not the driver is considered to be facing forward with respect to the traveling direction of the vehicle.
For example, the reference posture feature amount calculation unit 13 may determine, based on the feature amount information output from the feature amount extraction unit 12, whether or not the driver is considered to be facing forward with respect to the traveling direction of the vehicle.
For example, if the reference axis of the driver's facial direction and head position is the optical axis of the imaging device 2, the installation position and angle of view of the imaging device 2 in the driver's vehicle are known in advance, and therefore the reference posture feature amount calculation unit 13 can determine the ranges of the driver's facial direction and head position within which the driver is considered to be facing forward in the traveling direction of the vehicle. Note that the ranges of the driver's facial direction and head position detected by the feature amount extraction unit 12 when the driver is facing forward in the traveling direction of the vehicle are assumed to be preset.

If the reference posture feature calculation unit 13 estimates that the driver's posture is the reference posture, it stores the feature information output from the feature extraction unit 12 in the memory unit 18 or in a memory area internal to the reference posture feature calculation unit 13 as candidate feature information that is a candidate for calculating the reference posture feature.

When a predetermined number of candidate feature information items are stored, the reference posture feature calculation unit 13 calculates the driver's reference posture feature information items, specifically, the face direction (yaw angle, pitch angle, and roll angle), head position (X coordinate, Y coordinate, Z coordinate), and face detection reliability, from the stored predetermined number of candidate feature information items.

For example, the reference posture feature calculation unit 13 calculates the most frequent values of the driver's face direction, the driver's head position, and the face detection reliability included in a predetermined number of candidate feature information as the driver's reference posture feature.
For example, the reference posture feature calculation unit 13 may calculate the average value of the driver's facial direction, the driver's head position, and the face detection reliability included in a predetermined number of candidate feature information as the driver's reference posture feature.

The reference posture feature amount calculation unit 13 performs a reference posture feature amount calculation process, and when it calculates the reference posture feature amount of the driver, it stores information indicating the calculated reference posture feature amount of the driver (hereinafter referred to as “reference posture feature amount information”) in the memory unit 18.
Then, the reference posture feature value calculation unit 13 sets the reference posture feature value calculation completion flag, which is provided in a location that can be referenced by the abnormal posture detection device 1, to "1" to indicate that the reference posture feature value has been calculated.
The initial value of the reference posture feature calculation completion flag is set to "0." The reference posture feature calculation completion flag is initialized when the power supply of the vehicle is turned on, when the detection unit 17 detects an abnormal posture of the driver, etc.
At this time, reference posture feature amount calculation unit 13 may delete the candidate feature amount information stored in storage unit 18 or the internal storage area of reference posture feature amount calculation unit 13 .

The change amount calculation unit 14 calculates the amount of change of the feature amount extracted by the feature amount extraction unit 12 from the reference posture feature amount.
In the abnormal posture detection device 1 according to embodiment 1, the process performed by the change amount calculation unit 14 to calculate the change amount from the reference posture feature amount of the feature amount extracted by the feature amount extraction unit 12 is referred to as the "change amount calculation process."
The "change amount calculation process", the "first posture determination process" by the first determination unit 15 described later, the "second posture determination process" by the second determination unit 16 described later, and the "abnormal posture detection process" by the detection unit 17 described later are performed after the "reference posture feature amount calculation process" by the reference posture feature amount calculation unit 13 is completed. Details of the "first posture determination process", "second posture determination process", and "abnormal posture detection process" will be described later.

The change amount calculation unit 14 compares the features extracted by the feature amount extraction unit 12, specifically the driver's facial direction, head position, and face detection reliability, with the features included in the reference posture feature amount information stored in the memory unit 18, specifically the driver's facial direction, head position, and face detection reliability, and calculates the differences as change amounts. In other words, the change amount calculation unit 14 calculates the amount of change in the driver's facial direction extracted by the feature amount extraction unit 12 from the facial direction in the driver's reference posture, the amount of change in the driver's head position extracted by the feature amount extraction unit 12 from the head position in the driver's reference posture, and the amount of change in the face detection reliability extracted by the feature amount extraction unit 12 from the face detection reliability in the driver's reference posture. The change amount calculation unit 14 calculates the amount of change for each of the yaw angle, pitch angle, and roll angle of the driver's facial direction. The change amount calculation unit 14 also calculates the amount of change for each of the X, Y, and Z coordinates of the driver's head position.

The change amount calculation unit 14 outputs information indicating the calculated change amount (hereinafter referred to as “change amount information”) to the first determination unit 15 together with the feature amount information output from the feature amount extraction unit 12 .
The change amount information includes information on the change amount of the driver's facial orientation (change amount of yaw angle, change amount of pitch angle, and change amount of roll angle), information on the change amount of the driver's head position (change amount of X coordinate, change amount of Y coordinate, and change amount of Z coordinate), and information indicating the change amount of face detection reliability.

The first determination unit 15 determines whether the driver's posture is normal by comparing the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 with the conditions for determining normal posture, by comparing the feature extracted by the feature extraction unit 12 with the conditions for determining normal posture, or by comparing the change calculated by the change calculation unit 14 with the conditions for determining normal posture.
The process of determining whether the driver's posture is normal or not, which is performed by the first determination unit 15 in the abnormal posture detection device 1 according to the first embodiment, is referred to as a "first posture determination process."

The normal posture determination conditions are conditions for determining whether the driver's posture is normal, and are set in advance by an administrator or the like and stored in a location that can be referenced by the first determination unit 15.
The normal posture determination conditions include, for example, the following conditions:

"Of the following formulas (1-1), (1-2), (2-1), (2-2), (3-1), (3-2), (4-1), (4-2), (5-1), (5-2), (6-1), and (6-2), formula (1-1) or (1-2), formula (2-1) or (2-2), formula (3-1) or (3-2), formula (4-1) or (4-2), formula (5-1) or (5-2), and formula (6-1) or (6-2) must all be satisfied.

First yaw angle threshold < Change in face direction yaw angle from face direction yaw angle in reference posture < Second yaw angle threshold (1-1)
Third yaw angle threshold < face direction yaw angle < fourth yaw angle threshold (1-2)
First pitch angle threshold value < Change in face direction pitch angle from face direction pitch angle in reference posture < Second pitch angle threshold value (2-1)
Third pitch angle threshold value < face direction pitch angle < fourth pitch angle threshold value (2-2)
First roll angle threshold < change in face direction roll angle from face direction roll angle in reference posture < second roll angle threshold (3-1)
Third roll angle threshold < face direction roll angle < fourth roll angle threshold (3-2)
First X-coordinate threshold < Amount of change in the X-coordinate of the head position from the X-coordinate of the head position in the reference posture < Second X-coordinate threshold (4-1)
Third X coordinate threshold < X coordinate of head position < fourth X coordinate threshold (4-2)
First Y-coordinate threshold value < Amount of change in Y-coordinate of head position from Y-coordinate of head position in reference posture < Second Y-coordinate threshold value (5-1)
Third Y coordinate threshold < Y coordinate of head position < fourth Y coordinate threshold (5-2)
First coordinate Z threshold value < change in Z coordinate of head position from Z coordinate of head position in reference posture < second Z coordinate threshold value (6-1)
Third coordinate Z threshold < Z coordinate of head position < fourth Z coordinate threshold (6-2)

In addition,
First yaw angle threshold < second yaw angle threshold,
third yaw angle threshold<fourth yaw angle threshold,
First pitch angle threshold value<Second pitch angle threshold value,
third pitch angle threshold < fourth pitch angle threshold,
First roll angle threshold < second roll angle threshold,
third roll angle threshold<fourth roll angle threshold,
First X-coordinate threshold value<Second X-coordinate threshold value,
Third X-coordinate threshold value<fourth X-coordinate threshold value,
First Y-coordinate threshold value<Second Y-coordinate threshold value,
third Y-coordinate threshold value<fourth Y-coordinate threshold value,
First Z coordinate threshold value < second Z coordinate threshold value,
Third Z coordinate threshold < fourth Z coordinate threshold

The first determination unit 15 determines that the driver's posture is normal if the feature extracted by the feature extractor 12 or the change calculated by the change calculator 14 satisfies the normal posture determination condition.
If the feature extracted by the feature extractor 12 and the change calculated by the change calculator 14 do not satisfy the conditions for determining normal posture, the first determination unit 15 determines that the driver's posture is not normal.

Here, we will explain the significance of the first judgment unit 15 determining whether the driver's posture is normal or not based on the features extracted by the feature extraction unit 12, in other words, the features based on the current captured image, and the change calculated by the change calculation unit 14, in other words, the relative change in the features from the reference posture features.

Figure 2 is a diagram for explaining the significance of the first determination unit 15 determining whether the driver's posture is normal or not based on the feature values based on the current captured image and the relative change in the feature values in embodiment 1.
FIG. 2 is a simplified diagram showing an example of the concept of the determination by the first determination unit 15 as to whether the driver's posture is normal or not.

As described above, in the first embodiment, the reference posture is a posture in which the driver is considered to be facing forward in the traveling direction of the vehicle.
Here, for example, as shown on the left side of FIG. 2, a posture in which the driver is driving with his elbows on the ground and the roll angle of his face is 40 degrees can also be estimated as the reference posture.
Now, for example, as shown on the right side of FIG. 2, it is assumed that the driver stops driving with his elbows on the ground and starts driving with a straight posture, that is, the roll angle of the face becomes 0 degrees.
In this case, if the first determination unit 15 determines whether the driver's posture is normal or not based solely on the change in the feature from the reference posture feature, the first determination unit 15 may determine that the driver's posture in the state shown on the right side of Figure 2 is not normal.
The first determination unit 15 determines whether the driver's posture is normal or not by taking into consideration not only the change in the feature from the reference posture feature but also the feature based on the current captured image. Therefore, even if the reference posture is estimated to be a posture such as that shown on the left side of Figure 2, the first determination unit 15 can determine that the driver's posture, which should be considered a normal posture as shown on the right side of Figure 2, is a normal posture.

In the example explanation using Figure 2, the driver's facial direction was used as an example, but the same can be said for the driver's head position.By determining whether the driver's posture is normal or not by taking into account not only the relative change in the feature from the reference posture feature but also the feature based on the current captured image, a driver's posture that should be considered normal can be determined to be normal.

The first judgment unit 15 outputs the judgment result of whether or not the driver's posture is judged to be normal (hereinafter referred to as the "normal posture judgment result") to the second judgment unit 16. At this time, the first judgment unit 15 outputs the change amount information output from the change amount calculation unit 14 to the second judgment unit 16 together with the normal posture judgment result.

If the first judgment unit 15 judges that the driver's posture is not normal, the second judgment unit 16 judges whether the driver is experiencing poor posture based on the amount of change calculated by the change amount calculation unit 14 and the machine learning model.
The machine learning model is a trained model that receives input of the change in feature amount from a reference posture feature amount and outputs information indicating whether the driver's posture is out of alignment. The change amount is, in detail, the change in the driver's facial orientation from the facial orientation in the reference posture, the change in the driver's head position from the head position in the reference posture, and the change in the face detection reliability from the face detection reliability in the reference posture.
In embodiment 1, the information indicating whether or not the driver has poor posture output by the machine learning model includes information indicating whether or not the driver has poor posture, and, if the driver has poor posture, information indicating the type of poor posture (hereinafter referred to as the "poor posture type").

The machine learning model is generated in advance and stored in a location that can be referenced by the second judgment unit 16, such as the memory unit 18. For example, an administrator or the like may generate a machine learning model by having multiple drivers test drive the vehicle, and store the model in the memory unit 18 or the like.

It should be noted that a plurality of posture error types are defined in advance as types of posture error that may occur when a driver is experiencing posture error.
Here, Figure 3 (partially modified from "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining multiple types of posture imbalance.
As shown in Figure 3, there are multiple modes of poor posture. Figure 3 shows the following modes: "head down" in which the driver leans forward and continues to have their face close to the steering wheel; "head down" in which the driver continues to have their face facing down; "backward lean" in which the driver's upper body leans backward and continues to have their face facing up; "backward lean" in which the driver's upper body leans up and continues to have their face facing up; "head tilted to the side" in which the driver's face is tilted to the left or right; "sideways lean" in which the driver's upper body leans to the left or right and their face is tilted in the same direction; and "leaning to the side" in which the driver's upper body is tilted to the left or right. In the present disclosure, the multiple pre-set posture types are "head down,""headdown,""backwardarch,""archedback,""head tilted to the side,""head tilted to the side," and "leaning to the side" shown in Fig. 3. Information about the multiple posture types is stored in a location that can be referenced by the second determination unit 16.

The second judgment unit 16 inputs the amount of change into the machine learning model to obtain information indicating whether or not the driver has poor posture, and determines whether or not the driver has poor posture, and if so, the type of poor posture.

If the first judgment unit 15 determines that the driver's posture is normal, the second judgment unit 16 does not use the machine learning model to determine whether the driver has poor posture. In other words, the second judgment unit 16 determines that the driver has not poor posture, in other words, that the driver is not experiencing poor posture.

The second determination unit 16 outputs the determination result as to whether or not the driver is experiencing poor posture (hereinafter referred to as the “poor posture determination result”) to the detection unit 17 .
The posture error determination result is information in which information indicating whether or not the driver has posture error is associated with information indicating the type of posture error if the driver has posture error.

The process of determining whether the driver is experiencing poor posture, performed by the second determination unit 16 in the abnormal posture detection device 1 of embodiment 1 as described above, is referred to as the "second posture determination process."

The detection unit 17 detects whether the driver's posture is abnormal or not based on the posture deviation determination result output from the second determination unit 16 .
In the abnormal posture detection device 1 according to the first embodiment, the process performed by the detection unit 17 to detect whether the driver's posture is abnormal or not is referred to as "abnormal posture detection process."

The detection unit 17 detects that the driver's posture is abnormal if the state in which the second judgment unit 16 judges that the driver's posture is poor continues for a predetermined time (hereinafter referred to as the ``abnormality detection time'').
The detection unit 17 detects that the driver's posture is not abnormal when the second determination unit 16 determines that the driver is not suffering from poor posture, or when the state in which the second determination unit 16 determines that the driver is suffering from poor posture does not continue for the abnormality detection time.

In detail, the detection unit 17 first determines whether the second determination unit 16 has determined that the driver has poor posture. From the poor posture determination result output from the second determination unit 16, the detection unit 17 can determine whether the second determination unit 16 has determined that the driver has poor posture.

If the second judgment unit 16 judges that the driver has no posture imbalance, that is, if a posture imbalance judgment result indicating that the driver has not experienced posture imbalance is output, the detection unit 17 detects that the driver's posture is not abnormal.

If the second judgment unit 16 judges that the driver has poor posture, that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is experiencing poor posture, the detection unit 17 counts up a counter for counting the abnormality detection time (hereinafter referred to as the "abnormal posture detection counter").
On the other hand, if the second judgment unit 16 does not judge that the driver has poor posture, that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is not experiencing poor posture, the detection unit 17 resets the abnormal posture detection counter.

Next, the second determination unit 16 determines whether or not the abnormal posture detection counter has reached a preset threshold value (hereinafter referred to as the "abnormality detection threshold value").
When the abnormal posture detection counter reaches the abnormality detection threshold, the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver is experiencing poor posture has continued for the abnormality detection time, and detects that the driver's posture is abnormal. The detection unit 17 outputs information indicating that the driver's posture is abnormal (hereinafter referred to as "abnormal posture detection information") to the driving control system 300. The abnormal posture detection information includes information indicating that the driver's posture is abnormal and information indicating the posture deviation type. The detection unit 17 can identify the posture deviation type based on the posture deviation determination result output from the second determination unit 16.

The detection unit 17 may output the abnormal posture detection information to an output device (not shown) as information for analysis in the event of any unforeseen circumstances.

If the abnormal posture detection counter has not reached the abnormality detection threshold, the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver was experiencing poor posture has not continued for the abnormality detection time, and does not detect that the driver's posture is abnormal. In other words, the detection unit 17 detects that the driver's posture is not abnormal. The detection unit 17 may output information indicating that the driver's abnormal posture was not detected as abnormal posture detection information to the driving control system 300 or to an output device.

The storage unit 18 stores various information such as feature amount information, reference posture feature amount information, and machine learning models.
1, the storage unit 18 is provided in the abnormal posture detection device 1, but this is merely an example. The storage unit 18 may be provided outside the abnormal posture detection device 1 in a location that can be referenced by the abnormal posture detection device 1.

FIG. 4 is a diagram showing an example of the configuration of an operation control system 300 according to the first embodiment.
In the following description, a vehicle equipped with the abnormal attitude detection system 100 will be referred to as a first vehicle, and a vehicle other than the first vehicle will be referred to as a second vehicle. Note that there may be cases where the first vehicle and the second vehicle are not distinguished from each other and the first vehicle and the second vehicle are each referred to as a vehicle. In the following, an example will be described in which the abnormal attitude detection system 100 is used as part of a driving control system 300, but the abnormal attitude detection system 100 may be configured to be able to communicate with the driving control system 300 wirelessly or via a wire, as shown in FIG. 1 .

The driving control system 300 includes a driving control device C3 that controls the driving of the first vehicle by outputting a control signal based on abnormal attitude detection information from the abnormal attitude detection system 100 to control the steering mechanism 3 and braking/driving mechanism 4 of the first vehicle. The driving control system 300 also includes a map information storage device C4 that stores map information used for automatic driving of the vehicle, a surrounding condition monitoring device C5 that monitors the surrounding conditions of the vehicle, a vehicle condition acquisition device C6 that acquires information indicating the condition of the first vehicle, and an alarm control device C7 that controls warnings or notifications to the driver.

In addition, the abnormal posture detection system 100, driving control device C3, map information storage device C4, surrounding conditions monitoring device C5, vehicle status acquisition device C6, and alarm control device C7 provided in the driving control system 300 are each connected to communication bus C8, and data can be sent and received via communication bus C8.

The steering mechanism 3 is a mechanism provided on the first vehicle for determining the direction of travel of the first vehicle, and includes, for example, a steering column, steering shaft, rack, pinion, and steering actuator 31. The braking/driving mechanism 4 is a mechanism for controlling the traveling speed of the first vehicle and switching between forward and reverse travel, and includes, for example, an accelerator, brake, shift, and braking/driving actuator 41. The steering actuator 31 that controls the steering mechanism 3 is composed of, for example, an EPS (Electric Power Steering) motor, and the braking/driving actuator 41 that controls the braking/driving mechanism 4 is composed of, for example, an electronically controlled throttle and brake actuator.

The map information storage device C4 is a storage medium that stores map information, including road connectivity, the number of lanes, intersection location information, and railroad crossing location information. Here, the map information storage device C4 is assumed to be installed in the first vehicle, but the map information storage device C4 may also be configured, for example, as a server that transmits map data to the driving control device C3 via communication. When the map information storage device C4 is configured as the above-mentioned server, for example, a storage medium may be provided in the driving control system 300, and map information obtained from the map information storage device C4 via the on-board communication device C52 described below may be stored in the storage medium.

The surrounding conditions monitoring device C5 monitors the conditions around the first vehicle and is equipped with a GPS (Global Positioning System) receiver C51, an on-board communication device C52, an external sensor C53, and a navigation system C54.

The GPS receiver C51 receives signals transmitted from GPS positioning satellites and detects the current position of the first vehicle.

The outside vehicle sensor C53 is composed of, for example, at least one of a camera that captures images outside the vehicle, millimeter-wave radar, LiDAR, and an ultrasonic sensor, and detects the position or distance from the first vehicle of a second vehicle, pedestrian, obstacle, etc. that are present in the vicinity of the first vehicle.

The navigation system C54 calculates a route from the current position of the first vehicle to the destination and provides guidance along the calculated route. The navigation system C54 has a display unit, for example, an LCD display, and is housed in the instrument panel. The navigation system C54 also has an operation unit such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. The operation unit of the navigation system C54 may also be configured with a microphone or the like to be able to accept operations via voice uttered by the occupant.

The following describes the on-board communication device C52. The on-board communication device C52 is, for example, a wireless communication device connected to an antenna for wireless communication. The on-board communication device C52 communicates with a communication device in a second vehicle or a communication device installed on the road, etc., to obtain information on the location of the second vehicle or pedestrians, traffic information, etc. Here, traffic information includes, for example, congestion information, traffic regulation information, construction zone information, etc.

Furthermore, the onboard communication device C52 can perform vehicle-to-vehicle communication via wireless communication with the onboard communication device C52 of a second vehicle located around the first vehicle. In addition, the onboard communication device C52 may perform mobile communication with a base station external to the first vehicle. Furthermore, the onboard communication device C52 is configured to be able to transmit information about the first vehicle output onto the communication bus C8 to the second vehicle and a call center, etc. Note that the onboard communication device C52 can output information received from the second vehicle and information received from a call center, etc., to the communication bus C8. The onboard communication device C52 may also be configured to be able to obtain information from a server, etc. located outside the vehicle, such as the map information storage device C4 configured as a server that transmits map data to the driving control device C3.

The vehicle state acquisition device C6 acquires information indicating the state of the first vehicle and is equipped with a steering angle sensor C61, a vehicle speed sensor C62, a steering torque sensor C63, an accelerator position sensor C64, and a brake position sensor C65.

The steering angle sensor C61 is provided, for example, on the EPS motor or steering wheel and detects the steering angle of the first vehicle. The vehicle speed sensor C62 is provided, for example, on a wheel and detects the traveling speed of the first vehicle.

The steering torque sensor C63 is provided, for example, on the steering wheel and detects the magnitude of the steering wheel operating force applied by the driver. The accelerator position sensor C64 detects the amount of accelerator pedal depression by the driver. The brake position sensor C65 detects the amount of brake pedal depression by the driver.

The warning control device C7 acquires abnormality detection information from the abnormal posture detection system 100. If the warning control device C7 acquires abnormality detection information indicating that the driver's posture is abnormal, it outputs a warning to the driver via the warning unit C71. The warning unit C71 is, for example, an audio device mounted on the first vehicle, and the warning is a voice or notification sound output from the audio device. The warning unit C71 may also be configured as a display unit of the navigation system C54, and the warning may be a warning message displayed on the display unit of the navigation system C54. As described above, the warning may be one that can be recognized by the driver of the first vehicle. The driving control device C3 may also be controlled to output a control signal to cause the warning control device C7 to issue a warning.

Furthermore, the alarm control device C7 stops the alarm if the driver responds to the alarm after issuing the alarm. Here, a case where the driver responds to the warning means a case where the driver is recognized to have made a conscious action in response to the warning, such as when the abnormal posture detection system 100 outputs information indicating that the driver's posture is not abnormal, or when an operating unit provided in the navigation system C54 or a switch provided on the steering wheel receives an operation by the driver. The alarm control device C7 may also output a signal to the abnormal posture detection system 100 indicating that the driver has responded to the warning.

Furthermore, when abnormal posture detection information indicating that the driver's posture is abnormal is output from the abnormal posture detection system 100, the alarm control device C7 activates an alarm unit C72, such as turn signals, hazard lights, and headlights, which are provided on the vehicle exterior so as to be visible from the second vehicle, to notify people outside the first vehicle, such as occupants of the second vehicle, that the driver of the first vehicle is in an abnormal posture. For example, if the driver does not respond to the alarm for a predetermined period of time after the alarm is issued, the alarm control device C7 activates the alarm unit C72 to notify people outside the first vehicle that the driver of the first vehicle is in an abnormal posture.

In addition, the warning control device C7 may notify the occupants of the second vehicle that the driver of the first vehicle is in an abnormal posture through vehicle-to-vehicle communication via the above-mentioned in-vehicle communication device C52, for example by displaying an alert message on the display unit of the navigation system C54 of the second vehicle.

Furthermore, when driving assistance is performed by the driving control device C3, the notification control unit may use the vehicle state acquired by the vehicle state acquisition unit or the signal output from the driving control device C3 to notify people outside the first vehicle, such as occupants of the second vehicle, of the direction of travel of the first vehicle or whether or not it is changing lanes, using the notification unit C72, which is composed of the above-mentioned lighting fixtures, etc., provided on the vehicle exterior, etc.

The driving control device C3 of the driving control system 300 controls the steering actuator 31 or braking/driving actuator 41 installed in the first vehicle to assist the driver in driving. The driving control device C3 performs automatic driving control of the first vehicle, for example, by outputting signals to the steering actuator 31 or braking/driving actuator 41. The driving control device C3 performs braking/driving control of the first vehicle, such as by controlling the braking/driving actuator 41, which is composed of an electronically controlled throttle and brake actuator, to operate the brakes to slow down or stop the first vehicle. The driving control device C3 performs steering control of the first vehicle, such as by controlling the steering actuator 31, which is composed of an EPS motor, to maintain the lane in which the first vehicle is traveling.

Furthermore, the driving control device C3 has multiple driving assistance functions that assist or act on behalf of the driver in driving operations by controlling the driving force, braking force, steering force, etc. of the first vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function will be referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function will be referred to as LKA (Lane Keeping Assist).

When the driving control device C3 executes ACC, it controls the driving speed of the first vehicle by adjusting the driving force and braking force based on monitoring information of the vehicle ahead obtained from the surrounding conditions monitoring device C5. If no vehicle ahead is detected, ACC causes the first vehicle to travel at a constant speed at a target speed preset by the driver or other person. On the other hand, if a vehicle ahead is detected, ACC causes the first vehicle to follow the vehicle ahead while maintaining a safe distance from the vehicle ahead.

Furthermore, when the driving control device C3 executes LKA, it controls the steering force and steering force based on shape information of the lane markings in the direction of travel obtained from the surrounding conditions monitoring device C5. LKA applies a steering force to the steering wheel in a direction that prevents the first vehicle from approaching the lane markings, causing the first vehicle to travel along the lane. Note that road information output by the surrounding conditions monitoring device C5 may also be used for vehicle control by ACC and LKA.

Furthermore, when the driving control device C3 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the first vehicle. When the automatic evacuation control is initiated, the driving control device C3 causes the surrounding condition monitoring device C5 to search for an evacuation location where the first vehicle should be stopped. The driving control device C3 then moves the first vehicle to the evacuation location set by the surrounding condition monitoring device C5 and stops the first vehicle at this evacuation location. Note that, on expressways, the above-mentioned evacuation location may be the shoulder outside the lane in which the vehicle is traveling, and on ordinary roads, it may be a location that avoids locations where there is a high possibility of the second vehicle, trains, pedestrians, and other moving objects, such as intersections, railroad crossings, and sidewalks, as well as locations outside the lane in which the vehicle is traveling.

Here, if the driver's posture is abnormal, it is estimated that the driver is in an abnormal state. If the driver is in an abnormal state, there is a high possibility that any operation by the driver is an incorrect operation. For example, if the driver's physical condition suddenly changes and they lean their head over the steering wheel, there is a high possibility that they will operate the steering wheel, but this operation will be an incorrect operation. Also, if the driver's physical condition suddenly changes and they lean back, there is a possibility that the accelerator pedal will be accidentally pressed.

Therefore, if the driving control device C3 executes the driving assistance function due to the driver's abnormal posture, it may be possible to disable any erroneous operation by the driver while enabling driving operations aimed at preventing accidents (hereinafter referred to as override operations), such as intentional driving operations by the driver or driving operations by a passenger in place of the driver. The vehicle state acquisition device C6 may be provided with an override operation detection unit C66 that detects override operations.

The following explains the override operation. Regarding vehicle acceleration, if the driver is in an abnormal state, the driver's posture may become unstable and the driver may accidentally press the accelerator pedal. Therefore, when the driving assistance function is being executed by the driving control device C3, accelerator operation may be disabled. In other words, accelerator operation does not have to be included in the override operation.

On the other hand, when it comes to vehicle deceleration, even if the driver is in an abnormal state, a drowsy driver may stop the first vehicle in an attempt to avoid colliding with an obstacle. Therefore, the override operation detection unit C66 obtains the amount of brake pedal depression from the brake position sensor C65, and if the braking force obtained from the brake pedal depression is greater than the braking force in the driving assistance function of the driving control device C3, detects the brake pedal operation as an override operation.

Furthermore, when changing the vehicle's course, a passenger may operate the steering wheel in place of the driver even if the driver is in an abnormal state. However, if the driver is in an abnormal state, the driver's posture may become unstable and the driver may fall onto the steering wheel, so the override operation detection unit C66 does not need to detect this erroneous operation as an override operation. Therefore, the override operation detection unit C66 detects, as an override operation, steering wheel operation performed when the abnormal posture detection system 100 provides a detection result indicating that the driver's posture is not unstable.

The driving control system 300 may substitute the control of the steering actuator 31 or braking/driving actuator 41 with the driving operation of the passenger or driver when the override operation detection unit C66 detects an override operation while the driving assistance function is being executed. In other words, while the driving assistance function is being executed, the driving control of the vehicle may be performed based on the override operation detected by the override operation detection unit C66.

The operation of the abnormal posture detection device 1 related to embodiment 1 is described.

FIG. 5 is a flowchart for explaining the operation of the reference posture feature value calculation process performed by the abnormal posture detection device 1 according to the first embodiment.
The operation of the reference posture feature calculation process by the abnormal posture detection device 1 shown in the flowchart of Figure 5 is repeatedly performed, for example, in accordance with instructions from a control unit (not shown), while the reference posture feature calculation completion flag is "0" until the reference posture feature calculation completion flag becomes "1".
For example, when the ignition of the vehicle is turned on, the control unit refers to the reference posture feature calculation completion flag. When the control unit confirms that the reference posture feature calculation completion flag is "0", the control unit instructs the image acquisition unit 11, the feature extraction unit 12, and the reference posture feature calculation unit 13 of the abnormal posture detection device 1 to perform a reference posture feature calculation process.

The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1).
The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12 .

The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2).
The feature amount extraction unit 12 outputs the feature amount information to the reference posture feature amount calculation unit 13 and stores it in chronological order in the storage unit 18 .

The reference posture feature amount calculation unit 13 estimates the reference posture of the driver and performs a reference posture feature amount calculation process to calculate the reference posture feature amount (step ST3).
In the reference posture feature amount calculation process in step ST3, the reference posture feature amount calculation unit 13 first estimates whether the driver's posture is the reference posture.
When the reference posture feature calculation unit 13 estimates that the driver's posture is the reference posture, it stores the feature information output from the feature extraction unit 12 in the memory unit 18 or in a memory area inside the reference posture feature calculation unit 13 as candidate feature information that is a candidate for calculating the reference posture feature.
If the predetermined number of candidate feature information pieces have not been stored, reference posture feature calculation unit 13 does not set the reference posture feature calculation completion flag to “1” and ends the processing of step ST3. Then, the operation of abnormal posture detection device 1 returns to the processing of step ST1.
In step ST3, when a predetermined number of candidate feature information items are stored, the reference posture feature calculation unit 13 calculates the driver's reference posture feature information and face detection reliability from the stored predetermined number of candidate feature information items.
Reference posture feature quantity calculation unit 13 stores the reference posture feature quantity information in storage unit 18 and sets the reference posture feature quantity calculation completion flag to “1.” At this time, reference posture feature quantity calculation unit 13 may delete the reference posture feature quantity information stored in storage unit 18 or an internal storage area of reference posture feature quantity calculation unit 13.

FIG. 6 is a flowchart for explaining the operations of the change amount calculation process, the first posture determination process, the second posture determination process, and the abnormal posture detection process performed by the abnormal posture detection device 1 according to the first embodiment.
The operations of the change amount calculation process, first attitude determination process, second attitude determination process, and abnormal attitude detection process by the abnormal attitude detection device 1 shown in the flowchart of Figure 6 are repeatedly performed after the reference attitude feature calculation process described using the flowchart of Figure 5 is completed, for example, until the vehicle ignition is turned off.
When the reference posture feature calculation process is completed and the reference posture feature calculation completion flag is set to "1", the control unit confirms this and instructs the image acquisition unit 11, feature extraction unit 12, change amount calculation unit 14, first judgment unit 15, second judgment unit 16, and detection unit 17 of the abnormal posture detection device 1 to perform change amount calculation process, first posture judgment process, second posture judgment process, or abnormal posture detection process.

The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST11).
The image acquisition unit 11 outputs the acquired captured image to the feature extraction unit 12 .

The feature extraction unit 12 extracts feature amounts used to determine the driver's posture based on the captured image acquired by the image acquisition unit 11 in step ST11 (step ST12).
The feature extraction unit 12 outputs the feature information to the change amount calculation unit 14 and stores it in chronological order in the storage unit 18 .

The change amount calculation unit 14 performs a change amount calculation process to calculate the amount of change from the reference posture feature amount of the feature amount extracted by the feature amount extraction unit 12 in step ST12 (step ST13).
The change amount calculation unit 14 outputs the change amount information to the first determination unit 15 together with the feature amount information output from the feature amount extraction unit 12 .

The first judgment unit 15 performs a first posture judgment process to determine whether the driver's posture is normal or not by comparing the feature extracted by the feature extraction unit 12 in step ST12 and the change calculated by the change calculation unit 14 in step ST13 with the conditions for normal posture judgment, or by comparing the feature extracted by the feature extraction unit 12 with the conditions for normal posture judgment, or by comparing the change calculated by the change calculation unit 14 with the conditions for normal posture judgment (step ST14).

Here, Figure 7 is a flowchart for explaining the detailed operation of the first attitude determination process by the first determination unit 15 performed in step ST14 of Figure 6.

The first judgment unit 15 compares the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 with the conditions for determining normal posture, compares the feature extracted by the feature extraction unit 12 with the conditions for determining normal posture, or compares the change calculated by the change calculation unit 14 with the conditions for determining normal posture (step ST110), and judges whether the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 satisfy the conditions for determining normal posture, whether the feature extracted by the feature extraction unit 12 satisfies the conditions for determining normal posture, or whether the change calculated by the change calculation unit 14 satisfies the conditions for determining normal posture (step ST120).

If the feature extracted by the feature extraction unit 12 or the change calculated by the change calculation unit 14 satisfies the normal posture determination condition, if the feature extracted by the feature extraction unit 12 satisfies the normal posture determination condition, or if the change calculated by the change calculation unit 14 satisfies the normal posture determination condition ("YES" in step ST120), the first determination unit 15 determines that the driver's posture is normal (step ST130). The first determination unit 15 outputs a normal posture determination result indicating that the driver's posture is normal to the second determination unit 16. At this time, the first determination unit 15 outputs the change amount information output from the change calculation unit 14 together with the normal posture determination result to the second determination unit 16.

If the feature extracted by the feature extraction unit 12 and the change calculated by the change calculation unit 14 do not satisfy the conditions for determining normal posture ("NO" in step ST120), the first determination unit 15 does not determine that the driver's posture is normal. In other words, the first determination unit 15 determines that the driver's posture is not normal. The first determination unit 15 outputs a normal posture determination result to the second determination unit 16 indicating that the driver's posture has not been determined to be normal, in other words, that the driver's posture has been determined to be not normal. At this time, the first determination unit 15 outputs the change amount information output from the change calculation unit 14 together with the normal posture determination result to the second determination unit 16.

Returning to the description of the flowchart in FIG.
The second determination unit 16 performs a second posture determination process to determine whether the driver is experiencing poor posture based on the normal posture determination result output from the first determination unit 15 in step ST14 (step ST15).

Here, Figure 8 is a flowchart for explaining the detailed operation of the second posture determination process by the second determination unit 16 performed in step ST15 of Figure 6.

If the first judgment unit 15 determines in step ST14 of Figure 6 that the driver's posture is normal (if "YES" in step ST210), the second judgment unit 16 determines that the driver's posture is not poor, in other words, that the driver is not experiencing poor posture, and outputs a posture poor judgment result indicating that the driver is not experiencing poor posture to the detection unit 17, thereby terminating the processing shown in the flowchart of Figure 8.

If the first judgment unit 15 judges in step ST14 of FIG. 6 that the driver's posture is not normal ("NO" in step ST210), the second judgment unit 16 judges whether the driver is experiencing poor posture based on the amount of change calculated by the change amount calculation unit 14 in step ST13 of FIG. 6 and the machine learning model (step ST220). The second judgment unit 16 outputs a posture imbalance judgment result indicating that the driver is experiencing poor posture to the detection unit 17.

Returning to the description of the flowchart in FIG.
The detection unit 17 performs an abnormal posture detection process to detect whether the driver's posture is abnormal or not based on the posture collapse determination result output from the second determination unit 16 in step ST15 (step ST16).

Here, Figure 9 is a flowchart to explain the detailed operation of the abnormal posture detection process by the detection unit 17 performed in step ST16 of Figure 6.

The detection unit 17 determines whether the second judgment unit 16 has determined that the driver has poor posture at step ST15 in Figure 6 (step ST310).

If the second judgment unit 16 judges that the driver has poor posture (if the answer is "YES" in step ST310), that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is experiencing poor posture, the detection unit 17 counts up the abnormal posture detection counter (step ST320).

On the other hand, if the second judgment unit 16 does not determine that the driver has poor posture (if the answer is "NO" in step ST310), that is, if the second judgment unit 16 outputs a posture deviation judgment result indicating that the driver is not experiencing poor posture, the detection unit 17 resets the abnormal posture detection counter (step ST330).

The second judgment unit 16 determines whether the abnormal posture detection counter has reached the abnormality detection threshold (step ST340).

If the abnormal posture detection counter reaches the abnormality detection threshold ("YES" in step ST340), the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver was experiencing poor posture has continued for the abnormality detection time, and detects the driver's abnormal posture (step ST350). The detection unit 17 outputs abnormal posture detection information indicating that the driver's abnormal posture has been detected to the driving control system 300. The detection unit 17 may also output the abnormal posture detection information to an output device as information for analysis in the event of any unforeseen circumstances.

If the abnormal posture detection counter has not reached the abnormality detection threshold ("NO" in step ST340), the detection unit 17 determines that the state in which the second determination unit 16 determined that the driver was experiencing poor posture has not continued for the abnormality detection time, and does not detect an abnormal posture of the driver. In other words, the detection unit 17 detects that the driver's posture is not abnormal. The detection unit 17 may output information that an abnormal posture of the driver has not been detected to the driving control system 300 or to an output device.

In this way, the abnormal posture detection device 1 according to embodiment 1 calculates the amount of change from a reference posture feature of a feature extracted based on an image capturing the area where the driver's face should be. The abnormal posture detection device 1 first compares the extracted feature and the calculated amount of change with the normal posture determination conditions, and determines whether the driver's posture is normal by comparing the extracted feature with the normal posture determination conditions or by comparing the calculated amount of change with the normal posture determination conditions. If the abnormal posture detection device 1 determines that the driver's posture is not normal, it then determines whether the driver is exhibiting poor posture based on the calculated amount of change and the machine learning model. The abnormal posture detection device 1 then detects the driver's abnormal posture if the state in which it has determined that the driver is exhibiting poor posture continues for the abnormality detection time.

By using a machine learning model to determine whether an occupant's posture is out of alignment, it is possible to detect abnormal occupant posture with high accuracy. On the other hand, in a scene where feature values such as facial orientation cannot be stably extracted from a captured image, the presence or absence of an occupant's posture is determined based on low-accuracy feature values and the machine learning model. In this case, when determining whether an occupant's posture is out of alignment using the machine learning model, there is a possibility that the occupant may be determined to be in a state of posture out of alignment even though they are not.
Here, FIGS. 10A and 10B are diagrams for explaining an example of a scene in which feature amounts such as facial orientation cannot be stably extracted from a captured image.
FIG. 10A shows an example of a case where the driver's face is no longer captured in the captured image because the driver has assumed a "prone" posture.
FIG. 10B shows an example of a case where the driver touches his/her face with his/her hand, causing a part of the driver's face to be obscured in the captured image.
10A and 10B, the captured image is indicated by Im, and the driver is indicated by Dr. In the first embodiment, the upper left corner of the captured image is the origin, the rightward direction of the captured image is the positive direction of the x-axis, and the downward direction of the captured image is the positive direction of the y-axis.

For example, if the driver falls into a prone position, which is a result of poor posture, the driver's face will no longer be captured in the captured image, and features such as the driver's facial direction will no longer be extracted (see Figure 10A).

For example, if the driver is touching his or her face with a hand, part of the driver's face will be occluded in the captured image, making it difficult to stably extract features such as the driver's facial direction from the captured image (see Figure 10B). In the example of Figure 10B, the driver's eyes are occluded, making it difficult to stably extract the driver's facial direction.

It is assumed that the amount of change from the reference posture feature amount of the feature amount is large in both the scene shown in Fig. 10A and the scene shown in Fig. 10B. As a result, for example, if the amount of change is input to a machine learning model that determines whether the driver has poor posture, it may be determined that the driver has poor posture in both scenes. In fact, it should not be determined that the driver has poor posture in the scene shown in Fig. 10B.
When determining whether a driver has poor posture using a machine learning model, in scenes where features cannot be extracted stably, an unintended change from the reference posture feature may occur, increasing the possibility of misjudging whether the driver has poor posture.

In contrast, as described above, the abnormal posture detection device 1 according to embodiment 1 first determines whether the driver's posture is normal by comparing the extracted feature amount and the calculated change amount with the normal posture determination conditions, or by comparing the extracted feature amount with the normal posture determination conditions, or by comparing the calculated change amount with the normal posture determination conditions. If the abnormal posture detection device 1 then determines that the driver's posture is not normal, it determines whether the driver is exhibiting poor posture based on the calculated change amount and the machine learning model, and detects the driver's abnormal posture if the state in which it has been determined that the driver is exhibiting poor posture continues for the abnormality detection time.

Therefore, the abnormal posture detection device 1 can detect the driver's abnormal posture even in cases where features cannot be stably extracted from the captured image.

FIG. 11 is a flowchart for explaining an example of the operation of the driving assistance function by the driving control system 300 according to the first embodiment.
In the following description, an example will be given in which the driving control system 300 performs an emergency evacuation function for the vehicle as a driving assistance function when the abnormal posture detection system 100 detects that the driver's posture is abnormal. Note that the operation of the driving control system 300 is started, for example, after the ignition of the vehicle is turned on.

First, the driving control device C3 of the driving control system 300 acquires abnormal posture detection information from the abnormal posture detection system 100 (step ST401) and determines whether or not an abnormal posture of the driver has been detected (step ST402). If the abnormal posture detection information does not indicate that an abnormal posture of the driver has been detected ("NO" in step ST402), the operation of the driving control system 300 proceeds to the processing of ST401.

On the other hand, if the abnormal posture detection information indicates that an abnormal posture of the driver has been detected, the warning control device C7 activates the warning unit C71 to start warning the driver (step ST403). Here, the warning to the driver by the warning unit C71 may be, for example, the display of a warning message on the display unit of the navigation system C54 installed in the first vehicle, or the output of audio from an audio device installed in the first vehicle.

Next, the warning control device C7 determines whether or not the driver has responded to the warning (step ST404). Here, the determination of whether or not the driver has responded to the warning is made based on, for example, whether or not abnormal posture detection information indicating that the driver's abnormal posture has been detected is obtained from the abnormal posture detection system 100 before a predetermined time has elapsed since the warning was issued by the warning unit C71.

In other words, if abnormal posture detection information indicating that an abnormal posture of the driver has been detected is not received from the abnormal posture detection system 100 before a predetermined time has elapsed since the warning unit C71 was activated, the warning control device C7 determines that a response has been received from the driver to the warning. On the other hand, if a predetermined time has elapsed since the warning unit C71 was activated while abnormal posture detection information indicating that an abnormal posture of the driver has been detected remains received from the abnormal posture detection system 100, the warning control device C7 determines that a response has not been received from the driver to the warning. Note that the determination of whether a response has been received from the driver to the warning may be made, for example, based on whether the driver operates the operating unit to cancel the warning before a predetermined time has elapsed since the warning unit C71 was activated. The predetermined time is, for example, 3 seconds.

If the driver responds to the warning ("YES" in step ST404), the warning control device C7 stops the warning from the warning unit C71 (step ST405), and the operation of the driving control system 300 proceeds to the processing of step ST401. On the other hand, if the driver does not respond to the warning ("NO" in step ST404), the warning control device C7 activates the notification unit C72 and begins notifying people outside the first vehicle (step ST406). Here, notifying the second vehicle means notifying people outside the first vehicle, such as occupants of the second vehicle, that the driver of the first vehicle is not in a condition to continue driving, or notifying that the first vehicle will initiate a driving assistance function, such as an emergency evacuation function.

Then, the driving control device C3 starts the emergency evacuation function. First, the driving control device C3 acquires information about the vehicle state from the vehicle state acquisition device C6 (step ST407). Note that the vehicle state is information indicating the state of the first vehicle and is used for vehicle control in driving assistance functions such as the emergency evacuation function.

Next, the driving control device C3 acquires information about the surrounding conditions of the first vehicle from the surrounding conditions monitoring device C5 (step ST408) and sets a location to which the first vehicle should be evacuated (step ST409). Hereinafter, the location to which the first vehicle should be evacuated is referred to as the evacuation location. Note that the evacuation location may be within a range of 150 m from the current location of the first vehicle, or a location that takes the first vehicle 60 seconds or less to move from its current location. This prevents the first vehicle from entering the intersection more than necessary as it moves to the evacuation location.

For example, the driving control device C3 searches for a position where the first vehicle can be stopped safely using the surrounding conditions monitoring device C5, and sets the safest position from among the candidate safe stopping positions as the evacuation position. Here, the evacuation position is a position where there is no possibility of collision with a moving object such as a second vehicle or a pedestrian, or an obstacle, and where there is no possibility of collision with the aforementioned moving object or obstacle on the route from the current position of the first vehicle to the evacuation position. Examples of evacuation positions include the edge of the road such as a road shoulder, or within a lane other than an intersection or railroad crossing. Furthermore, if the evacuation position is the edge of the road such as a road shoulder, space may be secured to allow passengers to escape from the first vehicle.

After setting the evacuation position, the driving control device C3 performs vehicle control to stop the first vehicle at the evacuation position. After setting the evacuation position, the driving control device C3 may display the route from the current position of the first vehicle to the set evacuation position on the display unit of the navigation system C54.

First, the driving control device C3 determines whether a lane change is required before moving the first vehicle to the evacuation position (step ST410). If a lane change is not required before moving the first vehicle to the evacuation position ("NO" in step ST410), i.e., if the evacuation position is set within the lane in which the first vehicle is currently traveling, the driving control device C3 controls the steering actuator 31 and braking/driving actuator 41 to cause the first vehicle to travel within the lane in which the first vehicle is currently traveling (step ST411). Then, the driving control device C3 controls the steering actuator 31 and braking/driving actuator 41 to decelerate the first vehicle and then stop it at the evacuation position (step ST412).

In the process of step ST411, the driving control device C3 may drive the first vehicle at a speed at which an emergency stop can be made, such as 10 km/h. Furthermore, if it is not necessary to continue driving the first vehicle, such as when the evacuation position is set near the first vehicle, the process of step ST411 may be omitted. Furthermore, in the processes of steps ST411 and ST412, when the driving control device C3 decelerates the first vehicle, the deceleration may be, for example, 3 m/ ^s² or less to prevent passengers from falling over.

Then, after the driving control device C3 stops the first vehicle at the evacuation position, a report is sent via the onboard communication device C52 to a call center or the like outside the vehicle (step ST413). The report sent via the onboard communication device C52 includes, for example, a message indicating that the driver of the first vehicle is in an abnormal position and may be unable to continue driving, the current position of the first vehicle acquired by the surrounding conditions monitoring device C5, and the evacuation position set by the driving control device C3. Note that even when a report is sent via the onboard communication device C52, the driving control device C3 may continue to keep the first vehicle stopped, and the alarm control device C7 may continue to alert people outside the first vehicle.

The following describes the case where the driving control device C3 determines in the processing of step ST410 that a lane change is required before moving the first vehicle to the evacuation position ("YES" in step ST410). Note that lane change includes cases where the first vehicle is moved to a lane adjacent to the lane in which the first vehicle is traveling, such as a lane change, and cases where the first vehicle is deviated from the lane in which the first vehicle is traveling, such as pulling the first vehicle to the edge of the road.

If the driving control device C3 determines that a lane change is necessary, it controls the steering actuator 31 and braking/driving actuator 41 to change the lane of the first vehicle (step ST414). Here, the lateral movement speed of the vehicle during lane change may be approximately 0.3 m/s so that a second vehicle or pedestrian, etc., present at the lane change destination can recognize the lane change of the first vehicle and avoid a collision. Furthermore, when a lane change is to be made, the warning control device C7 may notify a person outside the first vehicle that the first vehicle will begin changing lane via the notification unit C72 at least a predetermined time before the lane change begins.

Next, if no response is received from the driver in response to the warning issued by the alarm unit, the notification unit C72 notifies the second vehicle that the driver of the first vehicle is in an abnormal position and that the driver may be unable to continue driving. Note that the notification to the second vehicle by the notification unit C72 is not limited to a notification that the driver of the first vehicle is in an abnormal position, but may also be a notification that the vehicle's emergency evacuation function will be initiated or a notification that the vehicle's emergency evacuation function is being executed, which is performed in subsequent processing.

Then, similar to the processing of step ST412, the driving control device C3 controls the steering actuator 31 and braking/driving actuator 41 to decelerate the first vehicle and then stop it at the evacuation position (step ST415). After the driving control device C3 stops the first vehicle at the evacuation position, the operation of the driving control system 300 proceeds to the processing of step ST413, and a report is sent to a call center or the like outside the vehicle via the on-board communication device C52.

Furthermore, if an override operation is performed while the driving assistance function is being executed, the driving control system 300 may substitute the control of the steering actuator 31 or braking/driving actuator 41 for driving operations by the passenger or driver. That is, for example, if the override operation detection unit C66 detects an override operation during the processing of steps ST406 to ST415 in FIG. 11, the vehicle may be driven based on the detected override operation.

In the above example, when the abnormal posture detection system 100 detects that the driver's posture is abnormal, the driving control system 300 performs an emergency evacuation function for the vehicle as a driving assistance function. However, the driving control system 300 may also perform ACC or LKA as a driving assistance function. Furthermore, driving assistance functions may be combined, such as by having the driving control system 300 perform ACC and LKA and then execute the emergency evacuation function once safety in the surrounding area is ensured. Driving assistance functions can be combined as appropriate.

In the above-described first embodiment, the abnormal posture detection device 1 includes the reference posture feature amount calculation unit 13 and has the function of estimating the reference posture of the driver and calculating the reference posture feature amount for the estimated reference posture, but this is merely an example. For example, the function of estimating the reference posture of the driver and calculating the reference posture feature amount for the reference posture may be possessed by a device (not shown) connected to the abnormal posture detection device 1 outside the abnormal posture detection device 1.
In this case, abnormal posture detection device 1 does not necessarily have to include reference posture feature value calculation unit 13. Furthermore, with regard to the operation of abnormal posture detection device 1, the operation of the reference posture feature value calculation process as described using the flowchart shown in FIG.
Alternatively, for example, an administrator or the like may set a reference posture based on the facial orientation, head position, and face detection reliability of a typical driver in advance and store the reference posture in a location accessible to the abnormal posture detection device 1, such as the storage unit 18. In this case, the abnormal posture detection device 1 does not necessarily need to include the reference posture feature amount calculation unit 13, and the operation of the reference posture feature amount calculation process as described using the flowchart shown in FIG. 5 can be omitted. However, a configuration having a function for estimating the reference posture in the reference posture feature amount calculation unit 13 or an external device allows the abnormal posture detection device 1 to estimate a reference posture and calculate reference posture feature amounts that are more tailored to the individual than a configuration not having a function for estimating the reference posture in the reference posture feature amount calculation unit 13 or an external device. As a result, the abnormal posture detection device 1 can more accurately detect whether the driver is in an abnormal posture.

In the first embodiment, the amount of change used as an input to the machine learning model includes the amount of change in face detection reliability, but this is merely an example. The amount of change used as an input to the machine learning model may not include the amount of change in face detection reliability.
However, if face detection reliability is included in the change amount used as input to the machine learning model, the abnormal posture detection device 1 can more accurately determine whether or not posture is abnormal, compared to when face detection reliability is not included.
Nowadays, imaging devices 2 mounted on vehicles have become smaller. Miniaturized imaging devices 2 often employ narrow-angle lenses. When the imaging device 2 is a so-called narrow-angle camera, the imaging range is narrow. That is, there is a high possibility that the driver's facial features will be out of frame. In other words, the ease of detecting the driver's facial features decreases. In this case, even if the driver's posture is not abnormal, the plausibility of the detected driver's face decreases. By using the face detection reliability as an input to the machine learning model, it is possible to prevent overdetection of posture imbalances and to expect to obtain more accurate inference results regarding information indicating whether or not the driver's posture is imbalance, obtained from the machine learning model.

In addition, in the above-described first embodiment, the vehicle occupant for whom the abnormal posture detection device 1 detects whether the vehicle is in an abnormal posture is the vehicle driver, but this is merely an example. The abnormal posture detection device 1 can also detect whether any vehicle occupant other than the driver is in an abnormal posture.

In addition, in the above-described embodiment 1, the abnormal posture detection device 1 is an on-board device mounted on a vehicle, and the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first judgment unit 15, the second judgment unit 16, and the detection unit 17 are provided in the on-board device.
Without being limited to this, some of the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 may be mounted on the in-vehicle device of the vehicle, and the rest may be provided on a server connected to the in-vehicle device via a network, so that a system may be configured with the in-vehicle device and the server.
In addition, the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, and the detection unit 17 may all be provided in the server.

12A and 12B are diagrams illustrating an example of the hardware configuration of the abnormal posture detection device 1 according to the first embodiment.
In the first embodiment, the functions of the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, the detection unit 17, and a control unit (not shown) are realized by the processing circuit 101. That is, the abnormal posture detection device 1 includes the processing circuit 101 for performing control to detect whether the posture of a vehicle occupant is an abnormal posture based on a captured image.
The processing circuitry 101 may be dedicated hardware as shown in FIG. 12A, or may be a processor 104 that executes a program stored in memory 105 as shown in FIG. 12B.

When the processing circuit 101 is dedicated hardware, the processing circuit 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

When the processing circuit is a processor 104, the functions of the image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. The processor 104 reads and executes the program stored in memory 105 to perform the functions of the image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown). In other words, the abnormal posture detection device 1 includes memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST3 of FIG. 5 and steps ST11 to ST16 of FIG. 6 described above. In addition, it can also be said that the program stored in memory 105 causes a computer to execute the processing procedures or methods of the image acquisition unit 11, the feature extraction unit 12, the reference posture feature calculation unit 13, the change amount calculation unit 14, the first determination unit 15, the second determination unit 16, the detection unit 17, and a control unit (not shown). Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

Note that the functions of image acquisition unit 11, feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of image acquisition unit 11 may be implemented by processing circuit 101 as dedicated hardware, and the functions of feature extraction unit 12, reference posture feature calculation unit 13, change amount calculation unit 14, first determination unit 15, second determination unit 16, detection unit 17, and a control unit (not shown) may be implemented by processor 104 reading and executing programs stored in memory 105.
The storage unit 18 is configured by, for example, a memory 105 .
The abnormal posture detection device 1 also includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the imaging device 2 or various devices included in the operation control system 300.

As described above, according to the first embodiment, the abnormal posture detection device 1 includes an image acquisition unit 11 that acquires an image of the area where the face of a vehicle occupant should be located, a feature extraction unit 12 that extracts features to be used in determining the posture of the occupant based on the image acquired by the image acquisition unit 11, a change amount calculation unit 14 that calculates the amount of change in the feature extracted by the feature extraction unit 12 from the feature in the occupant's reference posture, and a normal posture determination condition by comparing the feature extracted by the feature extraction unit 12 and the change amount calculated by the change amount calculation unit 14 with the normal posture determination condition. The abnormal posture detection device 1 is configured to include a first determination unit 15 that determines whether the posture of the occupant is normal by comparing the amount of change calculated by the change amount calculation unit 14 with a normal posture determination condition, a second determination unit 16 that, when the first determination unit 15 determines that the posture of the occupant is not normal, determines whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit 14 and a machine learning model that inputs the amount of change and outputs information indicating whether the occupant has lost their posture, and a detection unit 17 that detects an abnormal posture of the occupant when the state in which the second determination unit 16 has determined that the occupant has lost their posture continues for an abnormality detection time. Therefore, the abnormal posture detection device 1 can detect an abnormal posture of the occupant even when a scene occurs in which feature amounts cannot be stably extracted from a captured image.

Note that this disclosure allows for modifications to any of the components of the embodiments or the omission of any of the components of the embodiments.

The abnormal posture detection device disclosed herein can detect an occupant's abnormal posture even in situations where features cannot be stably extracted from captured images.

1 Abnormal posture detection device, 11 Image acquisition unit, 12 Feature extraction unit, 13 Reference posture feature calculation unit, 14 Change amount calculation unit, 15 First determination unit, 16 Second determination unit, 17 Detection unit, 18 Memory unit, 100 Abnormal posture detection system, 2 Imaging device, 300 Driving control system, 3 Steering mechanism, 31 Steering actuator, 4 Braking/driving mechanism, 41 Braking/driving actuator, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory, C3 Driving control device, C4 Map information storage device, C5 Surrounding situation monitoring device, C51 GPS receiver, C52 In-vehicle communication device, C53 External sensor, C54 Navigation system, C6 Vehicle state acquisition device, C61 Steering angle sensor, C62 Vehicle speed sensor, C63 Steering torque sensor, C64 Accelerator position sensor, C65 brake position sensor, C66 override operation detection unit, C7 alarm control device, C71 warning unit, C72 notification unit.

Claims (6)

an image acquisition unit that acquires a captured image capturing an area where the face of a vehicle occupant should be present;
a feature extraction unit that extracts feature amounts used to determine the posture of the occupant based on the captured image acquired by the image acquisition unit;
a change amount calculation unit that calculates a change amount of the feature amount extracted by the feature amount extraction unit from the feature amount in a reference posture of the occupant;
a first determination unit that determines whether the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change calculated by the change calculation unit with a normal posture determination condition , or by comparing the feature extracted by the feature extraction unit with the normal posture determination condition;
a second determination unit that, when the first determination unit determines that the posture of the occupant is not normal, determines whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit and a machine learning model that receives the amount of change as an input and outputs information indicating whether the occupant has lost their posture;
and a detection unit that detects an abnormal posture of the occupant when the state in which the second determination unit has determined that the occupant is losing their posture continues for an abnormality detection time. The abnormal posture detection device according to claim 1 , further comprising: a reference posture feature amount calculation unit that estimates the reference posture of the occupant and calculates the feature amount in the reference posture. 3. The abnormal posture detection device according to claim 1, wherein the feature amount includes a facial orientation of the occupant and a head position of the occupant. 3. The abnormal posture detection device according to claim 1, wherein the feature amounts include a facial orientation of the occupant, a head position of the occupant, and a face detection reliability. an image acquisition unit acquiring a captured image capturing an area where the face of a vehicle occupant should be present;
a feature extraction unit extracting feature amounts used to determine the posture of the occupant based on the captured image acquired by the image acquisition unit;
a change amount calculation unit calculating a change amount of the feature amount extracted by the feature amount extraction unit from the feature amount in a reference posture of the occupant;
a step in which a first determination unit determines whether the posture of the occupant is normal by comparing the feature extracted by the feature extraction unit and the change calculated by the change calculation unit with a normal posture determination condition , or by comparing the feature extracted by the feature extraction unit with the normal posture determination condition;
a second determination unit, when the first determination unit determines that the posture of the occupant is not normal, determining whether the occupant has lost their posture based on the amount of change calculated by the change amount calculation unit and a machine learning model that receives the amount of change as an input and outputs information indicating whether the occupant has lost their posture;
and a step in which a detection unit detects an abnormal posture of the occupant when the state in which the second determination unit has determined that the occupant is losing their posture continues for an abnormality detection time. A vehicle control system comprising the abnormal attitude detection device according to claim 1 and a driving control device that controls driving of the vehicle,
the abnormal posture detection device outputs a detection result of whether the occupant is in the abnormal posture to the operation control device;
The vehicle control system is characterized in that the driving control device outputs a control signal to issue a warning to the occupants or perform evacuation processing of the vehicle to an alarm control device, a steering mechanism, or a braking/driving mechanism mounted on the vehicle based on the detection result.