JP7630063B2 - Abnormal attitude determination device, abnormal attitude determination method, and vehicle control system - Google Patents

Description

The present disclosure relates to an abnormal attitude determination device, an abnormal attitude determination method, and a vehicle control system.

Conventionally, there is known a technology for detecting an abnormal state of an occupant by judging the occupant's posture collapse based on an image of the occupant in a vehicle. One of the posture collapse patterns is "head down" in which the occupant falls forward and continues to maintain a posture with the face close to the steering wheel. For example, Patent Document 1 discloses an image processing device that identifies a nose feature point and a left shoulder feature point of a driver captured in an image, and estimates that the driver is head down if the amount of decrease in the distance between the nose feature point and the left shoulder feature point in the vertical direction of the moving body exceeds a threshold value.
Incidentally, in recent years, imaging devices that are mounted on vehicles and capture images of the interior of the vehicle have become increasingly smaller, which may result in a narrower imaging range of the imaging devices.

JP 2021-149442 A

For example, when checking a sign ahead, a vehicle occupant may lean forward, or assume a so-called "leaning forward" posture. This "leaning forward" is not caused by the occupant being in an abnormal state. In other words, it does not fall under the category of posture loss that is used to detect an abnormal state of the occupant.
Here, whether the occupant is "leaning forward", which is a posture error, or "leaning forward", which is not a posture error, the occupant leans forward, so the occupant's face may not be captured in the captured image. In particular, in images captured by imaging devices with narrower imaging ranges in recent years, there is a high possibility that the occupant's face will not be captured in the captured image if the occupant is "leaning forward" or "leaning forward".
With conventional technology, when the occupant's face is no longer captured in the captured image, it is not possible to distinguish whether the occupant is in a "prone" or "leaning forward" position, which could result in the occupant being mistakenly detected as being in a "prone" position when in fact the occupant is in a "leaning forward" position.
Incidentally, the technology of the image processing device disclosed in Patent Document 1 does not take into consideration the case where the face of the occupant is no longer captured in the captured image, and therefore the above problem remains unresolved.

This disclosure has been made to solve such problems, and aims to provide an abnormal posture determination device that makes it possible to prevent the occupant's "leaning forward" posture from being mistakenly detected as a "prone" posture.

The abnormal posture determination device according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image captured in an area in which the face of the occupant should be, a posture determination feature detection unit that detects the facial direction of the occupant, the position of the occupant's head, and the positions of the occupant's facial features based on the face of the occupant in the image detected by the face detection unit, and a posture determination feature detection unit that performs a posture deviation type determination as to whether the occupant's posture falls into one of a plurality of posture deviation types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which the occupant is assumed not to be in a posture deviation, and if the face detection unit cannot detect the occupant's face, the occupant's posture is determined to fall into one of a plurality of posture deviation types. The vehicle is equipped with a posture type determination unit which determines that the occupant's posture corresponds to prone position among the posture collapse types, a forward lean determination unit which, if the posture type determination unit determines that the occupant's posture corresponds to prone position, makes a forward lean determination as to whether the occupant's posture corresponds to forward lean, which does not correspond to any of the multiple posture collapse types, based on the positions of the occupant's facial features based on the detected captured image, which is an captured image traced back from when the occupant's posture was determined to correspond to prone position and when the face detection unit was able to detect the occupant's face, and an abnormal posture determination unit which determines whether the occupant's posture is an abnormal posture based on the posture collapse type determination result by the posture type determination unit and the forward lean determination result by the forward lean determination unit.

The abnormal posture determination device disclosed herein can prevent an occupant's "leaning forward" posture from being mistakenly detected as a "prone" posture.

1 is a diagram illustrating an example of the configuration of an abnormal posture determination device according to a first embodiment; 11A and 11B are diagrams for explaining a plurality of types of poor posture. Figures 3A and 3B are diagrams illustrating an example of the transition in the position of the driver's face on the captured image when the driver assumes a "prone" position and the driver's face is no longer captured in the captured image, and when the driver assumes a "leaning forward" position and the driver's face is no longer captured in the captured image. 11 is a diagram for explaining a specific method of forward lean determination performed by a forward lean determination unit in embodiment 1. FIG. FIG. 1 is a diagram illustrating a configuration example of an operation control system according to a first embodiment. 4 is a flowchart for explaining an operation of the abnormal posture determination device according to the first embodiment. 4 is a flowchart showing an example of the operation of a driving assistance function by the driving control system according to the first embodiment. 8A and 8B are diagrams illustrating an example of a hardware configuration of the abnormal posture determination device according to the first embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
Embodiment 1.
The abnormal posture determination device according to the first embodiment determines whether or not the posture of a vehicle occupant is an abnormal posture, based on a captured image in which at least the face of the vehicle occupant is captured.
In the following first embodiment, as an example, the vehicle occupant who is the subject of the abnormal attitude determination device for determining whether or not the vehicle is in an abnormal attitude is the driver of the vehicle.

FIG. 1 is a diagram showing an example of the configuration of an abnormal posture determination device 1 according to the first embodiment.
The abnormal posture determination device 1 according to the first embodiment is assumed to be mounted on a vehicle.
The abnormal posture determination device 1 is connected to an image capture device 2 and an output device 3 , and the abnormal posture determination device 1 , the image capture device 2 and the output device 3 constitute an abnormal posture determination system 100 .
The abnormal attitude determination system 100, together with the driving control system 400, the steering mechanism 5, and the braking/driving mechanism 6, constitutes a vehicle control system SYS.A.

The imaging device 2 is mounted on the vehicle and is installed so as to be able to capture at least an area in which the driver's face should be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) that is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the abnormal posture determination device 1.
The abnormal posture determination device 1 determines whether or not the posture of the driver is abnormal based on the captured image captured by the imaging device 2. The abnormal posture determination device 1 will be described in detail later.
When the abnormal posture determination device 1 determines that the posture of the driver is abnormal, it causes the output device 3 to output a warning that the posture of the driver is abnormal.

The output device 3 is mounted on the vehicle and outputs a warning that the driver's posture is abnormal.
The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be, for example, provided in an audio device provided in a vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the abnormal posture determination device 1, the output device 3 outputs a warning sound or a voice message notifying that the driver's posture is abnormal.
The output device 3 may be, for example, a display device such as a display. For example, when warning information is output from the abnormal posture determination device 1, the output device 3 displays a message notifying the driver that the posture of the driver is abnormal.
The output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight provided on the exterior of the vehicle so as to be visible from other vehicles. For example, the abnormal attitude determination device 1 may output warning information to the output device 3 to notify a person outside the vehicle, such as a passenger of the other vehicle, that the driver of the vehicle is in an abnormal state.
The output device 3 may be mounted on, for example, another vehicle. The abnormal posture determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle, and can output a voice or display a message to inform the passengers of the other vehicle that the driver of the vehicle is in an abnormal state.

In addition, the abnormal posture determination device 1 outputs the determination result as to whether or not the driver's posture is abnormal to a driving control system 400 connected to the abnormal posture determination system 100.
1, the abnormal attitude determination system 100 is connected to a driving control system 400. The driving control system 400 cooperates with the steering mechanism 5 or the braking/driving mechanism 6, and performs driving assistance using the determination result of the abnormal attitude determination system 100. Details of the driving control system 400 will be described later.

As shown in FIG. 1, the abnormal posture determination device 1 includes an image acquisition unit 11, a face detection unit 12, a posture determination feature detection unit 13, a reference posture estimation unit 14, a determination unit 15, and an output control unit 16.
The posture determining feature detection unit 13 includes a face direction detection unit 131 , a head position detection unit 132 , and a features position detection unit 133 .
The determination unit 15 includes a posture type determination unit 151 , a lean-forward determination unit 152 , and an abnormal posture determination unit 153 .

The image acquisition unit 11 acquires a captured image from the imaging device 2 .
The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12 .

The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 .
In detail, the face detection unit 12 detects the driver's facial feature points indicating the driver's facial parts by using a known image recognition technique such as edge detection for the captured image. The facial parts are the corners and corners of the eyes, the nose, the mouth, the eyebrows, the chin, etc. The driver's facial feature points are represented, for example, by coordinates on the captured image. The face detection unit 12 may also detect the driver's facial area. The driver's facial area is, for example, the smallest rectangle that surrounds the outline of the driver's face. The driver's facial area is, for example, represented by the coordinates of the four corners of the smallest rectangle on the captured image.
Since the installation position and the angle of view of the imaging device 2 are known in advance, even if a plurality of passengers are captured in the captured image, the face detection unit 12 can determine which area of the captured image is the face of the driver. For example, an area in the captured image where the driver's face may be present (hereinafter referred to as the "driver detection area") is set in advance, and the face detection unit 12 detects the characteristic points and face area of the driver's face in the driver detection area using a known image recognition technique.
The face detection unit 12 outputs information about the detected face of the driver (hereinafter referred to as “face information”) to the posture determination feature detection unit 13 .
The face information is, for example, a captured image to which information capable of identifying the driver's facial feature points and facial area has been added.

The posture determination feature detection unit 13 detects features for determining the posture of an occupant, in this case the driver. The features for determining the driver's posture are the driver's facial direction, the driver's head position, and the positions of the driver's facial features. In the first embodiment, the driver's facial direction and head position detected by the posture determination feature detection unit 13 are, for example, the facial direction and head position in real space. Also, in the first embodiment, the positions of the driver's facial features detected by the posture determination feature detection unit 13 are, for example, the positions of the facial features on a captured image. In the first embodiment, as an example, the driver's facial parts for determining the driver's posture are the driver's eyes. In the following description, the driver's facial parts are assumed to be the driver's eyes.

The facial direction detection unit 131 of the posture determination feature detection unit 13 detects the facial direction of the driver in real space based on the facial information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12.
The face direction detection unit 131 may detect the face direction of the driver using, for example, a known face direction detection technique for detecting the face direction from a captured image. The face direction of the driver is represented, for example, by an angle (yaw angle, pitch angle, or roll angle) relative to a predetermined reference axis in the front-rear direction of the vehicle (hereinafter referred to as the "reference axis").
The face direction detection unit 131 outputs information relating to the detected face direction of the driver (hereinafter referred to as “face direction information”) to the reference posture estimation unit 14 and the determination unit 15 .
The face direction information includes information on the yaw angle, pitch angle, or roll angle with respect to a reference axis.

The head position detection unit 132 of the posture determination feature detection unit 13 detects the head position of the driver in real space based on the face information output from the face detection unit 12, specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In the first embodiment, the head position of the driver in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is only an example, and the position of the driver's head in the captured image may be indicated, for example, by the center of the driver's face area or the center of a straight line connecting the driver's eye corners. In this case, the head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's face area in the captured image or the center of a straight line connecting the driver's eye corners as the driver's head position in real space.
The head position detection unit 132 may detect the head position of the driver by using a known coordinate conversion technique for converting points on a captured image into points in real space. The driver's head position is represented by coordinates in real space, for example.
The head position detection unit 132 outputs information relating to the detected head position of the driver (hereinafter referred to as “head position information”) to the reference posture estimation unit 14 and the determination unit 15 .
The head position information includes coordinate information of the driver's head position.

The part position detection unit 133 of the posture determination feature detection unit 13 detects the position of the driver's eyes on the captured image based on the face information output from the face detection unit 12, specifically, based on the face of the driver in the captured image detected by the face detection unit 12. The face detection unit 12 detects the position of the driver's right eye and the position of the left eye on the captured image. In the first embodiment, the position of the driver's eyes is the center of a straight line connecting the inner corner and the outer corner of the driver's eye on the captured image. The part position detection unit 133 may detect the position of the driver's eyes using, for example, a known image recognition technology. The position of the driver's eyes is represented, for example, by coordinates on the captured image.
The part position detection unit 133 outputs information relating to the detected position of the driver's eyes (hereinafter referred to as “part position information”) to the reference posture estimation unit 14 and the determination unit 15 .
The part position information includes coordinate information of the position of the driver's right eye and coordinate information of the position of the driver's left eye.

The reference posture estimation unit 14 estimates the driver's reference posture based on the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 during a predetermined period (hereinafter referred to as the "reference setting period").
The reference setting period is, for example, a period from when the ignition of the vehicle is turned on until a preset time (for example, 10 seconds) has elapsed. Note that this is merely an example, and the reference setting period may be, for example, a predetermined time (for example, 3 seconds) when the vehicle speed is equal to or greater than a preset speed (for example, 25 km/h) and the steering angle is within a preset range (for example, ±20 degrees).
The reference posture estimation unit 14 estimates, for example, the average values of the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 during the reference setting period as the driver's facial direction, head position, and eye position in the driver's reference posture.
The reference posture estimation unit 14 outputs information relating to the estimated reference posture of the driver (hereinafter referred to as “reference posture information”) to the determination unit 15 .
The reference posture information includes information on the driver's facial orientation, head position, and eye position in the reference posture. The driver's facial orientation in the reference posture is represented by a yaw angle, pitch angle, or roll angle with respect to a reference axis. The driver's head position in the reference posture is represented, for example, by coordinates in real space. The driver's eye position in the reference posture is represented, for example, by coordinates on a captured image. The reference posture information includes information on the driver's facial orientation, such as a yaw angle, pitch angle, or roll angle with respect to a reference axis. The reference posture information also includes information on the driver's head position, such as coordinate information on the driver's head position. The reference posture information also includes information on the driver's eye position, such as coordinate information on the driver's right eye position and coordinate information on the driver's left eye position.

The judgment unit 15 judges whether the driver's posture is abnormal or not based on the driver's facial direction, head position, and eye position detected by the posture judgment feature detection unit 13.

The posture type determination unit 151 of the determination unit 15 determines whether or not the driver's posture corresponds to any one of a plurality of predetermined posture error types (hereinafter referred to as "posture error types") based on the face direction, head position, and eye position of the driver detected by the posture determination feature detection unit 13. In the first embodiment, the determination by the posture type determination unit 151 of whether the driver's posture corresponds to any one of a plurality of posture error types is also referred to as "posture error type determination."
The plurality of posture deviation types are set in advance as types of abnormal postures that are caused by the occupant being in an abnormal state.

FIG. 2 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" of the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining multiple types of posture imbalance.
As shown in Fig. 2, there are several types of posture failure. Fig. 2 shows "face down" in which the driver falls forward and continues to face near the steering wheel, "head down" in which the driver continues to face downward, "backward lean" in which the driver's upper body is tilted backward and the face continues to face upward, "backward lean" in which the driver's upper body is arched up and the face continues to face upward, "neck only sideways" in which the driver's face continues to be tilted to the left or right, "sideways" in which the driver's upper body is tilted to the left or right and the face continues to be tilted in the same direction, and "leaning to the side" in which the driver's upper body is tilted to the left or right. In the present disclosure, the multiple pre-defined types of posture problems are "face down", "head down", "backward arch", "arched back", "head only sideways", "sideways", and "leaning" shown in Fig. 2. Information on the multiple types of posture problems is stored in a location that can be referenced by the posture type determination unit 151.

The posture type determination unit 151 performs posture correction type determination as to whether the driver falls into any of a plurality of posture correction types by comparing the driver's facial direction detected by the facial direction detection unit 131 and the driver's head position detected by the head position detection unit 132 with the driver's facial direction in a reference posture (hereinafter referred to as the "reference facial direction") and the driver's head position (hereinafter referred to as the "reference head position").
In detail, the posture type determination unit 151 determines which of a plurality of posture failure types the driver belongs to, depending on how much the driver's face direction has changed compared to a reference face direction, or how much the driver's head position has changed compared to a reference head position. Note that the conditions for determining which posture failure type the driver belongs to when the driver's face direction has changed compared to the reference face direction, or when the driver's head position has changed compared to the reference head position, are preset and stored in the posture type determination unit 151.
The posture type determining section 151 may determine the posture disorder type by using a learning device such as a Support Vector Machine (SVM).

Here, if the face detection unit 12 fails to detect the driver's face, the posture type determination unit 151 determines that the driver's posture corresponds to "prone" among the multiple posture deviation types. The face detection unit 12 fails to detect the driver's face means that the posture determination feature detection unit 13 fails to detect the driver's facial direction, head position, and part positions. The posture type determination unit 151 can grasp, via the posture determination feature detection unit 13, that the face detection unit 12 fails to detect the driver's face.
When the driver falls forward, it becomes increasingly difficult to capture the driver's face in the captured image, and eventually the entire driver's face may not be captured.
In particular, if the imaging device 2 is a narrow-angle camera with a narrow imaging range, there is a high possibility that the driver's face will not be captured if the driver falls forward.
Therefore, when the face detection unit 12 fails to detect the face of the driver, the posture type determination unit 151 assumes that this is due to the driver having fallen forward, and determines that the driver's posture corresponds to "prone". Note that the determination of the driver's posture as "prone" by the posture type determination unit 151 due to the driver's face not being detected is a provisional, temporary determination, and the abnormal posture determination unit 153 ultimately determines whether the driver's posture is "prone". Details of the abnormal posture determination unit 153 will be described later.

The posture type determination unit 151 outputs the posture error type determination result to the forward lean determination unit 152. The posture error type determination result includes information indicating which of a plurality of posture error types the driver's posture falls under, and, if it is determined that the driver's posture falls under one of a plurality of posture error types, information capable of identifying the posture error type.

When the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the leaning forward determination unit 152 of the determination unit 15 determines whether the driver's posture corresponds to "leaning forward". In the first embodiment, the determination by the leaning forward determination unit 152 of whether the driver's posture corresponds to "leaning forward" is also referred to as "leaning forward determination". The "leaning forward" posture is a posture in which the body is inclined as if it is about to fall forward. For example, the driver may assume the "leaning forward" posture when checking a sign in front of the vehicle.
As shown in Fig. 2, "leaning forward" is a type of posture that does not fall into any of the multiple posture collapse types. In other words, "leaning forward" cannot be considered to be an abnormal posture caused by the occupant being in an abnormal state.

As described above, when the driver's face is no longer captured in the captured image and the face detection unit 12 cannot detect the driver's face, the posture type determination unit 151 provisionally determines the posture type to be "prone".
However, even when the driver assumes a "leaning forward" posture, the driver leans his/her body forward in the same manner as when the driver assumes a "prone" posture. Therefore, if the abnormal posture determination device 1 uniformly determines that the driver's posture is "prone" because the driver's face is no longer captured in the captured image, there is a possibility that the abnormal posture of the driver will be erroneously detected. Therefore, when the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the leaning forward determination unit 152 determines whether the determination is incorrect and the driver's posture is not "leaning forward".

Here, Figures 3A and 3B are figures for explaining an example of the transition in the position of the driver's face on the captured image when the driver assumes a "prone" position and the driver's face is no longer captured on the captured image, and when the driver assumes a "leaning forward" position and the driver's face is no longer captured on the captured image.
Fig. 3A shows an example of the transition of the position of the driver's face on the captured image when the driver's face is no longer captured on the captured image because the driver has taken a "prone" posture, and Fig. 3B shows an example of the transition of the position of the driver's face on the captured image when the driver has taken a "leaning forward" posture. In Fig. 3A and Fig. 3B, the captured image is indicated by Im, the driver is indicated by Dr, and the seat is indicated by S.
It is assumed that the imaging device 2 that captured the images shown in FIGS. 3A and 3B is a narrow-angle camera.
In the first embodiment, the upper left corner of the captured image is the origin, the rightward direction of the captured image is the positive direction of the x-axis, and the downward direction of the captured image is the direction of the y-axis.

As shown in Figure 3A, when the driver assumes a "prone" position, the driver's face moves downward and to the left in the captured image (see the middle diagram in Figure 3A), and then goes out of frame and is no longer captured in the captured image (see the right diagram in Figure 3A).
On the other hand, as shown in Figure 3B, when the driver assumes a "leaning forward" posture, the driver's face moves to the upper left in the captured image (see the middle diagram in Figure 3B), and then goes out of frame and is no longer captured in the captured image (see the right diagram in Figure 3B).
In this way, the driver's face is not captured in the captured image in either case, but the movement of the position of the driver's face in the captured image before the driver's face is not captured in the captured image differs between the case where the driver takes a "prone" posture and the case where the driver takes a "leaning forward" posture. The leaning forward determination unit 152 uses this to determine whether the driver's posture corresponds to "leaning forward".

When the posture type determination unit 151 determines that the driver's posture corresponds to "leaning forward," the leaning forward determination unit 152 performs a leaning forward determination as to whether the driver's posture corresponds to "leaning forward" or not, based on the position of the driver's eyes based on the captured image (hereinafter referred to as the "detected captured image") at the time when the face detection unit 12 was able to detect the driver's face going back from the time when the posture type determination unit 151 determined that the occupant's posture corresponds to "leaning forward."

For example, the posture determination feature detection unit 13 associates face information, face direction information, head position information, and part position information, assigns detection times, and stores the information in chronological order in a storage unit (not shown). The storage unit is provided in a location that can be referenced by the abnormal posture determination device 1.
When the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the lean-forward determination unit 152 refers to the storage unit and detects a detected captured image acquired prior to the determination. The lean-forward determination unit 152 can identify the detected captured image, for example, from face information stored in the storage unit. For example, if the face detection unit 12 can detect the driver's face based on the captured image acquired by the image acquisition unit 11 immediately before the captured image when the posture type determination unit 151 determines that the driver's posture corresponds to "prone", the lean-forward determination unit 152 detects the captured image acquired by the image acquisition unit 11 immediately before as the detected captured image.
The lean-forward determination unit 152 then compares the driver's eye height on the detected captured image with the eye height on the captured image of the driver in the reference posture (hereinafter referred to as the "reference eye height"), and determines that the driver's posture is "leaning forward" if the driver's eyes on the detected captured image are higher than a preset threshold (hereinafter referred to as the "height determination threshold"). The lean-forward determination unit 152 can specify the reference eye height based on the reference posture information output from the reference posture estimation unit 14. The eye height of the driver on the captured image is represented by the y coordinate of the position of the driver's eyes on the captured image.

The lean-forward determination unit 152 determines whether or not the posture of the occupant corresponds to “leaning forward” based on, for example, the following formula (1).

(Reference eye height)−(Eye height on detected captured image)>(Height determination threshold) (1)

If the formula (1) is satisfied, the lean-forward determination unit 152 determines that the occupant's posture is "leaning forward."
If the formula (1) is not satisfied, the lean-forward determination unit 152 determines that the occupant's posture is not "leaning forward," in other words, that the occupant's posture is "prone."

Note that a plurality of patterns of height determination threshold values are set in advance in accordance with combinations of the installation position of the imaging device 2 and the position of the vehicle's steering wheel, and are stored in a location that can be referenced by the forward lean determination unit 152 .
For example, before a product is shipped, an administrator or the like experimentally sits a subject in the driver's seat with the steering wheel in various positions and verifies what the eye height is in the captured image when the subject is in the reference posture and just before the subject's face is no longer captured when "leaning forward."
Then, the administrator or the like sets a height determination threshold for each handlebar position from the eye height in the captured image in the reference posture and the eye height in the captured image immediately before the subject's face is no longer captured when "leaning forward". The administrator or the like sets the height determination threshold, for example, to the difference between the eye height in the reference posture and the eye height immediately before the subject's face is no longer captured when "leaning forward".

FIG. 4 is a diagram for explaining a specific method of forward lean determination by the forward lean determination unit 152 in the first embodiment.
In FIG. 4, the left side shows an example of a captured image of the driver in the reference posture, and the right side shows an example of a captured image of the driver just before the driver's face is no longer captured when the driver leans forward.
As described above, when the driver takes a "leaning forward" posture, the driver's face moves to the upper left on the captured image. The eye height of the driver in the captured image taken just before the driver's face is no longer captured when "leaning forward" (shown as 402 in FIG. 4) is higher than the eye height on the captured image in the reference posture (shown as 401 in FIG. 4). In other words, when the driver takes a "leaning forward" posture, the above formula (1) is satisfied.

The leaning forward determination unit 152 determines whether or not the driver's posture corresponds to "leaning forward" based on the above formula (1) for each of the driver's right eye and left eye.
When it is determined that the driver's posture corresponds to "leaning forward" for at least one of the driver's right eye or left eye, the leaning forward determination unit 152 determines that the driver's posture corresponds to "leaning forward".

It should be noted that the above-described method of forward lean determination by the forward lean determination unit 152 is merely one example, and the forward lean determination unit 152 may perform the forward lean determination using other methods.
For example, when the facial features are the eyes, the leaning forward judgment unit 152 may judge that the driver's posture is "leaning forward" when the height of the driver's eyes on the detected captured image detected by the posture judgment feature detection unit 13 is higher than the top end of the steering wheel on the detected captured image, i.e., when the driver's eyes are located higher than the top end of the steering wheel on the detected captured image.
When leaning forward, the eye position is assumed to be lower than the handlebar position, and when leaning forward, the eye position is assumed to be higher than the handlebar position.

In addition, for example, the leaning forward determination unit 152 may compare the driver's eye height with the height of the top of the steering wheel in real space, in other words, in three-dimensional space, and determine that the driver's posture is leaning forward if the driver's eye height in the three-dimensional space is higher than the top of the steering wheel in the three-dimensional space, i.e., if the driver's eyes are located higher than the top of the steering wheel in the three-dimensional space.

The lean-forward determination unit 152 can calculate the height of the driver's eyes in three-dimensional space by converting the position of the driver's eyes on the detected captured image from a two-dimensional position to a three-dimensional position.
Furthermore, the forward lean determination unit 152 can calculate the three-dimensional position of the steering wheel and the height of the top end of the steering wheel in three-dimensional space based on operation information of the tilt steering and telescopic steering mounted on the vehicle.
The coordinate system of the three-dimensional position of the driver's eyes and the three-dimensional position of the top end of the steering wheel is defined as a vehicle coordinate system in which the position of the imaging device 2 is the origin, the right direction with respect to the vehicle's traveling direction is the X-axis, the upward direction of the vehicle is the Y-axis, and the direction opposite to the vehicle's traveling direction is the Z-axis.

This section describes an example method for calculating the driver's eye height in three-dimensional space and the height of the top end of the steering wheel in three-dimensional space using the forward lean determination unit 152.

First, an example of a method for calculating the driver's eye height in three-dimensional space by the lean-forward determination unit 152 will be described.
The lean-forward determination unit 152 calculates the distance from the position of the imaging device 2 to the driver's eyes (this distance is referred to as a first distance).
When the driver approaches the imaging device 2, the driver's face is captured larger in the detection captured image. Also, when the driver approaches the imaging device 2, the distance between the driver's eyes in the detection captured image increases. The lean-forward determination unit 152 calculates the first distance by using, for example, statistical information on the size of the driver's face in the detection captured image or the distance between the driver's eyes in the detection captured image.
Furthermore, the lean-forward determination unit 152 calculates an angle (defined as a first angle) of a line connecting the position of the imaging device 2 and the driver's eyes with respect to the Z-axis direction based on the distance on the image from the center of the detected captured image to the driver's eyes. At this time, the lean-forward determination unit 152 takes into consideration the mounting angle of the imaging device 2. The mounting angle of the imaging device 2 is known in advance.
Based on the calculated first distance and first angle, the forward lean determination unit 152 can calculate the height of the driver's eyes in a three-dimensional space using a trigonometric function. Also, the forward lean determination unit 152 can calculate the three-dimensional position of the driver's eyes.

Next, an example of a method for calculating the height of the upper end of the handlebar in three-dimensional space by the lean-forward determination unit 152 will be described.
The forward lean determination unit 152 calculates the angle of the steering wheel (second angle) based on the tilt steering operation information. The forward lean determination unit 152 also calculates the depth of the steering wheel position (second distance) based on the telescopic steering operation information.
Since the size of the steering wheel and its mounting position on the vehicle are known in advance, the forward lean determination unit 152 can calculate the three-dimensional position of the top end of the steering wheel based on the second angle and the second distance. The forward lean determination unit 152 can also calculate the height of the top end of the steering wheel in three-dimensional space.

The lean-forward determination unit 152 outputs the lean-forward determination result to the abnormal posture determination unit 153. The lean-forward determination result includes information indicating whether or not the posture of the driver corresponds to "leaning forward."
The lean forward determination unit 152 outputs the posture error type determination result outputted from the posture type determination unit 151 together with the lean forward determination result to the abnormal posture determination unit 153 .

The abnormal posture determination unit 153 determines whether or not the driver's posture is abnormal, based on the posture deviation type determination result by the posture type determination unit 151 and the forward lean determination result by the forward lean determination unit 152 .
In detail, even if the posture type determination unit 151 determines that the driver's posture is “prone”, if the forward lean determination unit 152 determines that the driver's posture is “leaning forward”, the abnormal posture determination unit 153 determines that the driver's posture is “leaning forward”, i.e., the driver is not in an abnormal posture.
When the posture type determination unit 151 determines that the driver's posture is "head down" and the lean forward determination unit 152 determines that the driver's posture is not "head down", the abnormal posture determination unit 153 determines the duration of the state in which the driver's posture is "head down". For example, the abnormal posture determination unit 153 stores the acquired posture failure type determination result in a storage unit (not shown) in association with information on the date and time when the posture failure type determination result was acquired. The abnormal posture determination unit 153 may determine the duration of the state in which the driver's posture is "head down" from the stored posture failure type determination result. When the duration of the state in which the driver's posture is "head down" reaches a preset threshold (hereinafter referred to as "abnormal posture determination threshold"), the abnormal posture determination unit 153 determines that the driver's posture is "head down", that is, that the driver has an abnormal posture.
Furthermore, when the posture type determination unit 151 determines that the driver's posture corresponds to a posture error type other than "prone", the abnormal posture determination unit 153 determines the duration of the state in which the driver's posture corresponds to a posture error type other than "prone". The abnormal posture determination unit 153 may determine the duration of the state in which the driver's posture corresponds to a posture error type other than "prone" from the stored posture error type determination result. When the duration of the state corresponding to a posture error type other than "prone" reaches the abnormal posture determination threshold, the abnormal posture determination unit 153 determines that the driver's posture is of the abnormal posture type determined by the posture type determination unit 151, that is, that the driver has an abnormal posture.
When the posture type determination unit 151 determines that the driver's posture does not correspond to any of the plurality of posture abnormality types, the abnormal posture determination unit 153 determines that the driver is not in an abnormal posture.
The abnormal posture determination unit 153 outputs the determination result as to whether or not the driver is in an abnormal posture (hereinafter referred to as the “abnormal posture determination result”) to the output control unit 16 .

When the abnormal posture determination unit 153 determines that the driver is in an abnormal posture, the output control unit 16 outputs warning information for warning the driver to the output device 3.
In addition, the output control unit 16 outputs the abnormal attitude determination result output from the abnormal attitude determination unit 153 to the operation control system 400.

FIG. 5 is a diagram showing an example of the configuration of an operation control system 400 according to the first embodiment.
In the following description, a vehicle equipped with the abnormal attitude determination system 100 is referred to as a first vehicle, and a vehicle other than the first vehicle is referred to as a second vehicle. Note that there are cases where the first vehicle and the second vehicle are not distinguished from each other, and the first vehicle and the second vehicle are each referred to as a vehicle. In the following, an example in which the abnormal attitude determination system 100 is used as a part of the driving control system 400 will be described, but the abnormal attitude determination system 100 may be configured to be able to communicate with the driving control system 400 wirelessly or via a wire, as shown in FIG. 1.

The driving control system 400 includes a driving control device C3 that controls the driving of the first vehicle by outputting a control signal based on the abnormal attitude determination result by the abnormal attitude determination system 100 to control the steering mechanism 5 and the braking/driving mechanism 6 of the first vehicle. The driving control system 400 also includes a map information storage device C4 that stores map information used for automatic driving of the vehicle, a surrounding situation monitoring device C5 that monitors the surrounding situation of the vehicle, and a vehicle status acquisition device C6 that acquires information indicating the status of the first vehicle.

In addition, the abnormal posture determination system 100, driving control device C3, map information storage device C4, surrounding conditions monitoring device C5, and vehicle state acquisition device C6, which are provided in the driving control system 400, are each connected to a communication bus C7, and data can be sent and received via the communication bus C7.

The steering mechanism 5 is a mechanism provided on the first vehicle for determining the direction of travel of the first vehicle, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 51. The braking/driving mechanism 6 is a mechanism for controlling the traveling speed of the first vehicle and switching between forward and reverse, and includes, for example, an accelerator, a brake, a shift, and a braking/driving actuator 61. The steering actuator 51 that controls the steering mechanism 5 is composed of, for example, an EPS (Electric Power Steering) motor, and the braking/driving actuator 61 that controls the braking/driving mechanism 6 is composed of, for example, an electronically controlled throttle, a brake actuator, etc.

The map information storage device C4 is a storage medium that stores map information including road connection relationships, the number of lanes, intersection location information, and railroad crossing location information. Here, the map information storage device C4 is assumed to be mounted on the first vehicle, but the map information storage device C4 may be configured, for example, as a server that transmits map data to the driving control device C3 via communication. When the map information storage device C4 is configured as the above-mentioned server, for example, a storage medium may be provided in the driving control system 400, and map information acquired from the map information storage device C4 via the vehicle-mounted communication device C52 described below may be stored in the storage medium.

The surrounding conditions monitoring device C5 monitors the conditions around the first vehicle and includes a GPS (Global Positioning System) receiver C51, an on-board communication device C52, an external sensor C53, and a navigation system C54.

The GPS receiver C51 receives signals transmitted from GPS positioning satellites and detects the current position of the first vehicle.

The outside vehicle sensor C53 is composed of, for example, at least one of a camera that captures images outside the vehicle, a millimeter wave radar, a LiDAR, and an ultrasonic sensor, and detects the position of a second vehicle, pedestrian, obstacle, etc. that is present in the vicinity of the first vehicle or the distance from the first vehicle.

The navigation system C54 calculates a route from the current position of the first vehicle to the destination and provides guidance along the calculated route. The navigation system C54 has a display unit, for example, a liquid crystal display, and is housed in an instrument panel. The navigation system C54 also has an operation unit, such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. The operation unit of the navigation system C54 may be configured with a microphone or the like to be able to accept operations via voice uttered by the occupant.

The vehicle-mounted communication device C52 will be described. The vehicle-mounted communication device C52 is, for example, a wireless communication device connected to an antenna for wireless communication. The vehicle-mounted communication device C52 acquires information on the position of the second vehicle or pedestrians, traffic information, etc. by communicating with a communication device of a second vehicle or a communication device installed on the road. Here, traffic information is, for example, congestion information, traffic regulation information, construction section information, etc.

In addition, the vehicle-mounted communication device C52 can perform vehicle-to-vehicle communication by wireless communication with the vehicle-mounted communication device C52 of a second vehicle present around the first vehicle. In addition, the vehicle-mounted communication device C52 may perform mobile communication with a base station outside the first vehicle. Furthermore, the vehicle-mounted communication device C52 is configured to be able to transmit information about the first vehicle output onto the communication bus C7 to the second vehicle and a call center, etc. In addition, the vehicle-mounted communication device C52 can output information received from the second vehicle and information received from the call center, etc., to the communication bus C7. In addition, the vehicle-mounted communication device C52 may be configured to be able to acquire information from a server, etc., present outside the vehicle, such as the map information storage device C4 configured as a server that transmits map data to the driving control device C3.

The vehicle state acquisition device C6 acquires information indicating the state of the first vehicle and is equipped with a steering angle sensor C61, a vehicle speed sensor C62, a steering torque sensor C63, an accelerator position sensor C64, and a brake position sensor C65.

The steering angle sensor C61 is provided, for example, on an EPS motor or a steering wheel, and detects the steering angle of the first vehicle. The vehicle speed sensor C62 is provided, for example, on a wheel, and detects the traveling speed of the first vehicle.

The steering torque sensor C63 is provided, for example, on the steering wheel and detects the magnitude of the steering wheel operating force applied by the driver. The accelerator position sensor C64 detects the amount of depression of the accelerator pedal by the driver. The brake position sensor C65 detects the amount of depression of the brake pedal by the driver.

The driving control device C3 of the driving control system 400 controls the steering actuator 51 or braking/driving actuator 61 mounted on the first vehicle to assist the driver in driving. The driving control device C3 performs automatic driving control of the first vehicle, for example, by outputting a signal to the steering actuator 51 or braking/driving actuator 61. The driving control device C3 performs braking/driving control of the first vehicle, such as by controlling the braking/driving actuator 61, which is composed of an electronically controlled throttle and a brake actuator, to operate the brakes to decelerate or stop the first vehicle. The driving control device C3 performs steering control of the first vehicle, such as by controlling the steering actuator 51, which is composed of an EPS motor, to maintain the lane in which the first vehicle is traveling.

Furthermore, the driving control device C3 has multiple driving assistance functions that assist or substitute for the driver's driving operations by controlling the driving force, braking force, steering force, etc. of the first vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function is referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function is referred to as LKA (Lane Keeping Assist).

When the driving control device C3 executes ACC, it controls the driving speed of the first vehicle by adjusting the driving force and braking force based on monitoring information of the vehicle ahead obtained from the surrounding conditions monitoring device C5. If a vehicle ahead is not detected, the ACC causes the first vehicle to travel at a constant speed at a target speed preset by the driver or the like. On the other hand, if a vehicle ahead is detected, the ACC causes the first vehicle to follow the vehicle ahead while maintaining a distance from the vehicle ahead.

In addition, when the driving control device C3 executes LKA, it controls the steering force and steering force based on shape information of the lane markings in the travel direction obtained from the surrounding conditions monitoring device C5. LKA applies a steering force to the steering wheel in a direction that prevents the first vehicle from approaching the lane markings, causing the first vehicle to travel along the lane. Note that road information output by the surrounding conditions monitoring device C5 may be used for vehicle control by ACC and LKA.

When the driving control device C3 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the first vehicle. When the automatic evacuation control is started, the driving control device C3 causes the surrounding situation monitoring device C5 to search for an evacuation location to stop the first vehicle. Then, the driving control device C3 moves the first vehicle to the evacuation location set by the search by the surrounding situation monitoring device C5, and stops the first vehicle at this evacuation location. Note that the above-mentioned evacuation location may be a shoulder outside the lane in which the vehicle is traveling on an expressway, and may be a location that avoids positions where moving bodies such as a second vehicle, a train, and a pedestrian are likely to be present, such as intersections, railroad crossings, and sidewalks, in addition to the shoulder outside the lane in which the vehicle is traveling on a general road.

Here, if the driver is in an abnormal state, there is a high possibility that the operation by the driver is an incorrect operation. For example, if the driver's physical condition suddenly changes and he or she leans forward against the steering wheel, the steering wheel will be operated, but this operation is likely to be an incorrect operation. Also, if the driver's physical condition suddenly changes and he or she leans back, the accelerator pedal may be pressed by mistake.

Therefore, when the driving control device C3 executes the driving assistance function due to the driver being in an abnormal state, it may be possible to disable erroneous operations by the driver while enabling driving operations aimed at preventing accidents, such as the driver's intentional driving operations or driving operations by a passenger in place of the driver (hereinafter referred to as override operations). The vehicle state acquisition device C6 may be provided with an override operation detection unit C66 that detects override operations.

The override operation will be explained below. Regarding vehicle acceleration, if the driver is in an abnormal state, the driver may lose posture and accidentally press the accelerator pedal. Therefore, when the driving assistance function by the driving control device C3 is being executed, the accelerator operation may be disabled. In other words, the accelerator operation does not have to be included in the override operation.

On the other hand, regarding the deceleration of the vehicle, even if the driver is in an abnormal state, the driver, who is in a daze, may stop the first vehicle in an attempt to avoid a collision with an obstacle. Therefore, the override operation detection unit C66 obtains the amount of depression of the brake pedal from the brake position sensor C65, and if the braking force obtained from the depression of the brake pedal is greater than the braking force in the driving assistance function of the driving control device C3, detects the operation of the brake pedal as an override operation.

Furthermore, when changing the vehicle's course, even if the driver is in an abnormal state, a passenger may operate the steering wheel in place of the driver. However, if the driver is in an abnormal state, the driver's posture may become distorted and the driver may fall onto the steering wheel, so the override operation detection unit C66 does not need to detect this erroneous operation as an override operation. Therefore, the override operation detection unit C66 detects, as an override operation, the operation of the steering wheel performed when a detection result indicating that the driver's posture has not become distorted is obtained from the abnormal posture determination system 100.

When the override operation detection unit C66 detects an override operation while the driving support function is being executed, the driving control system 400 may substitute the control of the steering actuator 51 or the braking/driving actuator 61 for the driving operation by the passenger or the driver. In other words, while the driving support function is being executed, the driving control of the vehicle may be performed by the override operation detected by the override operation detection unit C66.

The operation of the abnormal posture determination device 1 according to the first embodiment will be described.
FIG. 6 is a flowchart for explaining the operation of the abnormal posture determination device 1 according to the first embodiment.
The operation of the abnormal posture determination device 1 starts, for example, after the ignition of the vehicle is turned on.

The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1).
The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12 .

The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2).
The face detection unit 12 outputs the face information to the posture determination feature detection unit 13 .

The facial direction detection unit 131 of the posture determination feature detection unit 13 detects the facial direction of the driver in real space based on the facial information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3).
The face direction detection unit 131 outputs the face direction information to the reference posture estimation unit 14 and the determination unit 15 .

The head position detection unit 132 of the posture determination feature detection unit 13 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4).
The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the determination unit 15 .

The part position detection unit 133 of the posture determination feature detection unit 13 detects the position of the driver's eyes in the captured image based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST5).
The part position detection unit 133 outputs the part position information to the reference posture estimation unit 14 and the determination unit 15 .

If the reference posture estimation unit 14 has not completed estimation of the driver's reference posture (if "NO" in step ST6), the reference posture of the driver is estimated based on the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 during the reference setting period (step ST7).
The reference posture estimation unit 14 may determine whether or not estimation of the driver's reference posture has been completed based on, for example, a reference posture estimation completion flag. For example, when the reference posture estimation unit 14 estimates the driver's reference posture, it sets the reference posture estimation completion flag to "1". The initial value of the reference posture estimation completion flag is "0", and the reference posture estimation completion flag is initialized, for example, when the ignition of the abnormal posture determination device 1 is turned off. The reference posture estimation completion flag is stored in a location that can be referenced by the abnormal posture determination device 1.

If the reference posture estimation unit 14 has completed estimation of the driver's reference posture (if "YES" in step ST6), the posture type determination unit 151 of the determination unit 15 performs a posture correction type determination as to whether the driver's posture corresponds to any of a plurality of predetermined posture correction types based on the driver's facial direction, head position, and eye position detected by the posture determination feature detection unit 13 (the driver's facial direction detected by the facial direction detection unit 131 in step ST3, the driver's head position detected by the head position detection unit 132 in step ST4, and the driver's eye position detected by the parts position detection unit 133 in step ST5) (step ST8).
The posture type determination unit 151 outputs the posture error type determination result to the lean forward determination unit 152 .

If the posture type determination unit 151 determines in step ST8 that the driver's posture corresponds to "prone", the forward lean determination unit 152 of the determination unit 15 performs a forward lean determination to determine whether the driver's posture corresponds to "leaning forward" (step ST9).
The forward lean determination unit 152 outputs the forward lean determination result to the abnormal posture determination unit 153 .
The forward lean determination unit 152 outputs the posture type determination result output from the posture type determination unit 151 in step ST8 to the abnormal posture determination unit 153 together with the forward lean determination result.

The abnormal posture determination unit 153 determines whether the driver's posture is abnormal or not based on the posture deviation type determination result by the posture type determination unit 151 in step ST8 and the forward lean determination result by the forward lean determination unit 152 in step ST9 (step ST10).

If the abnormal posture determination unit 153 determines in step ST10 that the driver is in an abnormal posture, the output control unit 16 outputs warning information for warning the driver to the output device 3 (step ST11).
In addition, the output control unit 16 outputs the abnormal attitude determination result output from the abnormal attitude determination unit 153 to the operation control system 400.

FIG. 7 is a flowchart showing an example of the operation of the driving assistance function by the driving control system 400 according to the first embodiment.
In the following description, an example will be given in which the driving control system 400 performs an emergency evacuation function for the vehicle as a driving assistance function when the abnormal posture determination system 100 determines that the driver is in a bad posture, in other words, that the driver is in an abnormal state. Note that the operation of the driving control system 400 is started, for example, after the ignition of the vehicle is turned on.

The driving control device C3 of the driving control system 400 obtains the abnormal posture judgment result of the driver of the first vehicle from the abnormal posture judgment system 100 (ST201).

Then, the driving control device C3 starts the emergency evacuation function. First, the driving control device C3 acquires the vehicle state from the vehicle state acquisition device C6 (ST202). The vehicle state is information indicating the state of the first vehicle, and is used for vehicle control in driving assistance functions such as the emergency evacuation function.

Next, the driving control device C3 acquires the surrounding conditions of the first vehicle from the surrounding conditions monitoring device C5 (ST203) and sets a position to which the first vehicle is to be evacuated (ST204). Hereinafter, the position to which the first vehicle is to be evacuated is referred to as the evacuation position. The evacuation position may be within a range of 150 m from the current position of the first vehicle, or a position where the time required for the first vehicle to move from the current position is within 60 seconds. In this way, it is possible to prevent an unnecessary increase in the number of times the first vehicle enters the intersection as it moves to the evacuation position.

For example, the driving control device C3 searches for a position where the first vehicle can be stopped safely from the surrounding situation monitoring device C5, and sets the safest position from among the candidate positions where the first vehicle can be stopped safely as the evacuation position. Here, the evacuation position is a position where there is no possibility of collision with a moving object such as a second vehicle or a pedestrian, or an obstacle, and where there is no possibility of collision with the above-mentioned moving object or obstacle on the route from the current position of the first vehicle to the evacuation position. Examples of evacuation positions are the edge of the road such as a road shoulder, inside a lane other than an intersection or a railroad crossing, etc. Furthermore, if the evacuation position is the edge of the road such as a road shoulder, space may be secured to allow passengers to escape from the first vehicle.

After setting the evacuation position, the driving control device C3 performs vehicle control to stop the first vehicle at the evacuation position. After setting the evacuation position, the driving control device C3 may also display the route from the current position of the first vehicle to the set evacuation position on the display unit of the navigation system C54.

The driving control device C3 determines whether a lane change is required before moving the first vehicle to the evacuation position (ST205). If a lane change is not required before moving the first vehicle to the evacuation position ("NO" in ST205), that is, if the evacuation position is set within the lane in which the first vehicle is currently traveling, the driving control device C3 controls the steering actuator 51 and the braking/driving actuator 61 to cause the first vehicle to travel within the lane in which the first vehicle is currently traveling (ST206). Then, the driving control device C3 controls the steering actuator 51 and the braking/driving actuator 61 to decelerate the first vehicle and then stop it at the evacuation position (ST207).

In the process of ST207, the driving control device C3 may drive the first vehicle at a speed at which an emergency stop can be performed, such as 10 km/h. In addition, if it is not necessary to continue driving the first vehicle, such as when the evacuation position is set near the first vehicle, the process of ST206 may be omitted. Furthermore, in the processes of ST206 and ST207, when the driving control device C3 decelerates the first vehicle, the deceleration may be, for example, 3 m/s2 ^or less to prevent the passenger from falling over.

Then, the driving control device C3 stops the first vehicle at the evacuation position, and then reports the same to a call center or the like outside the vehicle via the in-vehicle communication device C52 (ST210). The report via the in-vehicle communication device C52 includes, for example, a message indicating that the driver of the first vehicle is in an abnormal state, the current position of the first vehicle acquired by the surrounding conditions monitoring device C5, and the evacuation position set by the driving control device C3.

The following describes the case where the driving control device C3 determines in the processing of ST205 that a lane change is required before the first vehicle is moved to the evacuation position ("YES" in ST205). Note that lane change includes a case where the first vehicle is moved to a lane adjacent to the lane in which the first vehicle is traveling, such as a lane change, and a case where the first vehicle is deviated from the lane in which the first vehicle is traveling so as to pull the first vehicle over to the side of the road.

When the driving control device C3 determines that a lane change is required, it controls the steering actuator 51 and the braking/driving actuator 61 to change the lane of the first vehicle (ST208). Here, the lateral movement speed of the vehicle during lane change may be about 0.3 m/s so that a second vehicle or pedestrian, etc., present at the lane change destination can recognize the lane change of the first vehicle and avoid a collision.

Then, similar to the process of ST207, the driving control device C3 controls the steering actuator 51 and the braking/driving actuator 61 to decelerate the first vehicle and then stop it at the evacuation position (ST209). After the driving control device C3 stops the first vehicle at the evacuation position, the operation of the driving control system 400 proceeds to the process of ST210, and a report is sent to a call center or the like outside the vehicle via the in-vehicle communication device C52.

Furthermore, when an override operation is performed while the driving assistance function is being executed, the driving control system 400 may substitute the control of the steering actuator 51 or the braking/driving actuator 61 for driving operation by the passenger or the driver. That is, for example, when the override operation detection unit C66 detects an override operation during the processing of ST202 to ST210 in FIG. 7, the vehicle may be driven according to the detected override operation.

In the above example, when the abnormal posture determination system 100 determines that the driver is in a bad posture, in other words, that the driver is in an abnormal state, the driving control system 400 performs an emergency evacuation function for the vehicle as a driving assistance function. However, the driving control system 400 may execute ACC or LKA as a driving assistance function. In addition, the driving assistance functions may be combined, such as by executing ACC and LKA and then executing the emergency evacuation function when the safety of the surroundings is ensured. In this way, when the driver is in a bad posture, in other words, when it is determined that the driver is in an abnormal state, the driving assistance functions can be appropriately combined.

In the above embodiment 1, the facial features of the driver have been described as the driver's eyes, but this is merely one example. For example, the facial features of the driver may be the nose or eyebrows. For example, if the facial feature of the driver is the nose, the lean-forward determination unit 152 compares the height of the nose on the detected captured image of the driver with the height of the nose on the captured image of the driver in the reference posture, and if the nose on the detected captured image of the driver is positioned higher by a preset threshold or more, determines that the driver's posture is leaning forward.

In the above-described first embodiment, the abnormal posture determination device 1 assumes that once the reference posture estimation unit 14 estimates the driver's reference posture, the reference posture is not revised until, for example, the vehicle ignition is turned off. However, this is merely an example, and for example, the reference posture estimation unit 14 may constantly monitor the reference posture, and if the reference posture has changed within a range that can be said to be no deviation from the driver's posture, the reference posture may be updated.

In the above-described first embodiment, the reference posture estimating unit 14 estimates the reference posture of the driver, but this is merely an example.
For example, an administrator or the like may set a reference posture in advance based on the facial direction, head position, and position of facial features such as the eyes of a typical driver, and store the reference posture in a location that can be referenced by the determination unit 15.
In this case, the abnormal posture determination device 1 can be configured not to include the reference posture estimation unit 14. However, if the reference posture estimation unit 14 estimates the reference posture, the abnormal posture determination device 1 can set a reference posture that is more suited to an individual than if the abnormal posture determination device 1 is configured not to include the reference posture estimation unit 14. As a result, the abnormal posture determination device 1 can more accurately determine whether or not the driver is in an abnormal posture.

In the above embodiment 1, the vehicle occupant for which the abnormal posture determination device 1 determines whether the vehicle is in an abnormal posture is the driver of the vehicle, but this is merely one example. The abnormal posture determination device 1 can determine whether any vehicle occupant other than the driver is in an abnormal posture.

In addition, in the above embodiment 1, the abnormal posture judgment device 1 is an in-vehicle device mounted on a vehicle, and the image acquisition unit 11, face detection unit 12, posture judgment feature detection unit 13, reference posture estimation unit 14, judgment unit 15, and output control unit 16 are provided in the in-vehicle device.
Without being limited to this, some of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may be mounted on the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, so that a system is configured with the in-vehicle device and a server.
Furthermore, the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may all be provided in a server.

8A and 8B are diagrams illustrating an example of a hardware configuration of the abnormal posture determination device 1 according to the first embodiment.
In the first embodiment, the functions of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 are realized by a processing circuit 1001. That is, the abnormal posture determination device 1 includes the processing circuit 1001 for performing control to determine whether the posture of a vehicle occupant is an abnormal posture by distinguishing between "prone" and "leaning forward" based on a captured image.
The processing circuitry 1001 may be dedicated hardware as shown in FIG. 8A, or may be a processor 1004 executing a program stored in a memory 1005 as shown in FIG. 8B.

When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1005. The processor 1004 executes the functions of the image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 by reading and executing the program stored in the memory 1005. In other words, the abnormal posture determination device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST11 in FIG. 6 described above. In addition, the program stored in memory 1005 can also be said to cause the computer to execute the processing procedures or methods of image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16. Here, the memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

The functions of image acquisition unit 11, face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of image acquisition unit 11 and output control unit 16 can be realized by processing circuit 1001 as dedicated hardware, and the functions of face detection unit 12, posture determination feature detection unit 13, reference posture estimation unit 14, and determination unit 15 can be realized by processor 1004 reading and executing programs stored in memory 1005.
The storage unit (not shown) is composed of, for example, a memory 1005 .
The abnormal posture determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as an imaging device 2 or an output device 3.

As described above, according to the first embodiment, the abnormal posture determination device 1 includes a face detection unit 12 that detects the face of the vehicle occupant in an image captured in which the area in which the face of the occupant should be present is captured, a posture determination feature detection unit 13 that detects the facial direction of the occupant, the position of the occupant's head, and the position of the facial features of the occupant in the captured image based on the face of the occupant in the captured image detected by the face detection unit 12, and a posture determination feature detection unit 13 that compares the facial direction and head position of the occupant detected by the posture determination feature detection unit 13 with the facial direction and head position of the occupant in a reference posture in which the occupant is assumed not to be in a posture disorder to determine whether the occupant's posture falls into one of a plurality of posture disorder types. If the face detection unit 12 cannot detect the occupant's face, the abnormal posture determination device 1 determines whether the occupant's posture falls into one of a plurality of posture disorder types. The abnormal posture determination device 1 is configured to include a posture type determination unit 151 that determines that the posture of the occupant corresponds to the prone posture among posture deviation types, a forward lean determination unit 152 that, when the posture type determination unit 151 determines that the posture of the occupant corresponds to the prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to a forward lean that does not correspond to any of the multiple posture deviation types based on the positions of the facial features of the occupant on the detected captured image, which is an image captured when the face detection unit 12 was able to detect the face of the occupant going back from when the posture of the occupant was determined to correspond to the prone posture, and an abnormal posture determination unit 153 that determines whether or not the posture of the occupant is an abnormal posture based on the posture deviation type determination result by the posture type determination unit 151 and the forward lean determination result by the forward lean determination unit 152. Therefore, the abnormal posture determination device 1 can prevent the occupant's "forward lean" posture from being erroneously detected as the "prone" posture.

Various aspects of this disclosure are summarized below as appendices.

(Appendix 1)
a face detection unit that detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured;
a posture determination feature detection unit that detects a facial direction of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit;
a posture type determination unit which performs a posture error type determination as to whether the posture of the occupant corresponds to one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types;
a forward lean determination unit that, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to forward leaning that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant;
and an abnormal posture determination unit that determines whether the posture of the occupant is an abnormal posture based on a posture collapse type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit.
(Appendix 2)
The abnormal posture determination device according to claim 1, characterized in that even if the posture type determination unit determines that the posture of the occupant is prone, if the forward lean determination unit determines that the posture of the occupant is leaning forward, the abnormal posture determination unit determines that the posture of the occupant is not the abnormal posture.
(Appendix 3)
An abnormal posture determination device as described in Appendix 1 or Appendix 2, comprising a reference posture estimation unit that estimates the reference posture of the occupant based on the facial direction, head position, and facial feature positions of the occupant detected by the posture determination feature detection unit during a reference setting period.
(Appendix 4)
The abnormal posture determination device according to any one of Supplementary Note 1 to Supplementary Note 3, characterized in that the forward lean determination unit compares a height of a facial feature of the occupant in the detected captured image with a height of a facial feature of the occupant in the captured image in the reference posture, and determines that the posture of the occupant is forward leaning if the facial feature of the occupant in the detected captured image is located at a position higher than or equal to a height determination threshold.
(Appendix 5)
the facial features of the occupant are the eyes of the occupant,
The abnormal posture determination device according to any one of claims 1 to 3, wherein the forward lean determination unit determines that the posture of the occupant is leaning forward when the eye height of the occupant is higher than the top end of the handlebars.
(Appendix 6)
The abnormal posture determination device described in Appendix 5, characterized in that the forward leaning determination unit compares the height of the occupant's eyes with the height of the top end of the handlebars on the detection captured image, and determines that the occupant's posture is leaning forward if the occupant's eyes are at a higher position than the top end of the handlebars on the detection captured image.
(Appendix 7)
The abnormal posture judgment device described in Appendix 5, characterized in that the forward lean judgment unit compares the height of the occupant's eyes with the height of the top end of the steering wheel in three-dimensional space, and judges that the occupant's posture is leaning forward if the occupant's eyes are at a higher position than the top end of the steering wheel in the three-dimensional space.
(Appendix 8)
The abnormal state determination device according to claim 7, wherein the forward lean determination unit calculates the height of the occupant's eyes in the three-dimensional space based on the detected captured image and a position of an imaging device at which the captured image is taken, and calculates the height of an upper end of the steering wheel in the three-dimensional space based on tilt steering and telescopic steering operation information.
(Appendix 9)
An abnormal posture determination device as described in any one of Supplementary Note 1 to Supplementary Note 8, further comprising: an output control unit that outputs warning information to warn the occupant when the abnormal posture determination unit determines that the occupant is in the abnormal posture.
(Appendix 10)
The abnormal posture determination device according to any one of claims 1 to 9, wherein the facial features of the occupant are the eyes of the occupant.
(Appendix 11)
A face detection unit detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured;
a posture determination feature detection unit detecting a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit;
a posture type determination unit performs a posture error type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types;
a forward lean determination unit, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to a forward lean that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to the time when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant;
an abnormal posture determination unit determining whether or not the posture of the occupant is an abnormal posture based on a posture imbalance type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit.
(Appendix 12)
A vehicle control system including the abnormal attitude determination device according to any one of claims 1 to 10 and a driving control device that controls driving of the vehicle,
the abnormal posture determination device outputs a determination result as to whether or not the occupant is in the abnormal posture to the operation control device;
The vehicle control system is characterized in that the driving control device outputs a control signal for outputting a warning to the occupants or performing evacuation processing of the vehicle based on the judgment result to an alarm control device, a steering mechanism, or a braking/driving mechanism mounted on the vehicle.

In addition, this disclosure allows for the modification of any of the components of the embodiments or the omission of any of the components of the embodiments.

The occupant status detection device disclosed herein can be installed in a vehicle and used as a device to detect whether or not the vehicle's occupants are able to drive.

1 Abnormal posture determination device, 2 Imaging device, 3 Output device, 5 Steering mechanism, 6 Braking/driving mechanism, 11 Image acquisition unit, 12 Face detection unit, 13 Posture determination feature detection unit, 131 Face direction detection unit, 132 Head position detection unit, 133 Part position detection unit, 14 Reference posture estimation unit, 15 Determination unit, 151 Posture type determination unit, 152 Forward lean determination unit, 153 Abnormal posture determination unit, 16 Output control unit, 100 Abnormal posture determination system, 400 Driving control system, 51 Steering actuator, 61 Braking/driving actuator, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory, C3 Driving control device, C4 Map information storage device, C5 Surrounding situation monitoring device, C51 GPS receiver, C52 Vehicle-mounted communication device, C53 External sensor, C54 navigation system, C6 vehicle state acquisition device, C61 steering angle sensor, C62 vehicle speed sensor, C63 steering torque sensor, C64 accelerator position sensor, C65 brake position sensor, C66 override operation detection unit.

Claims (12)

a face detection unit that detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured;
a posture determination feature detection unit that detects a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit;
a posture type determination unit which performs a posture error type determination as to whether the posture of the occupant corresponds to one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types;
a forward lean determination unit that, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to forward leaning that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant;
and an abnormal posture determination unit that determines whether the posture of the occupant is an abnormal posture based on a posture collapse type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit. The abnormal posture determination device according to claim 1, characterized in that even if the posture type determination unit determines that the posture of the occupant is prone, if the forward lean determination unit determines that the posture of the occupant is leaning forward, the abnormal posture determination unit determines that the posture of the occupant is not the abnormal posture. The abnormal posture determination device of claim 1 , further comprising a reference posture estimation unit that estimates the reference posture of the occupant based on the facial direction, head position, and facial feature positions of the occupant detected by the posture determination feature detection unit during a reference setting period. The abnormal posture determination device according to any one of claims 1 to 3, characterized in that the forward leaning determination unit compares the height of the facial features of the occupant in the detected captured image with the height of the facial features of the occupant in the captured image in the reference posture, and determines that the posture of the occupant is forward leaning if the facial features of the occupant in the detected captured image are located at a position higher than or equal to a height determination threshold. the facial features of the occupant are the eyes of the occupant,
The abnormal posture determination device according to any one of claims 1 to 3, characterized in that the forward lean determination unit determines that the posture of the occupant is leaning forward when the height of the occupant's eyes is higher than the top end of the handlebars. The abnormal posture determination device according to claim 5, characterized in that the forward lean determination unit compares the height of the occupant's eyes with the height of the top end of the handlebars on the detection captured image, and determines that the posture of the occupant is leaning forward if the occupant's eyes are at a higher position than the top end of the handlebars on the detection captured image. The abnormal posture determination device according to claim 5, characterized in that the forward lean determination unit compares the height of the occupant's eyes with the height of the top end of the steering wheel in a three-dimensional space, and determines that the occupant's posture is leaning forward if the occupant's eyes are located higher than the top end of the steering wheel in the three-dimensional space. 8. The abnormal posture determination device according to claim 7, wherein the forward lean determination unit calculates the height of the occupant's eyes in the three-dimensional space based on the detected captured image and a position of an imaging device at which the captured image is taken, and calculates the height of an upper end of the steering wheel in the three-dimensional space based on tilt steering and telescopic steering operation information. The abnormal attitude determination device according to claim 1 , further comprising: an output control unit that outputs warning information for warning the occupant when the abnormal attitude determination unit determines that the occupant is in the abnormal attitude. 2. The abnormal posture determination device according to claim 1, wherein the facial features of the occupant are the eyes of the occupant. A face detection unit detects a face of a vehicle occupant in a captured image in which a range in which the face of the vehicle occupant should be present is captured;
a posture determination feature detection unit detecting a facial orientation of the occupant, a head position of the occupant, and positions of facial features of the occupant based on the face of the occupant in the captured image detected by the face detection unit;
a posture type determination unit performs a posture error type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture error types by comparing the facial direction and head position of the occupant detected by the posture determination feature detection unit with the facial direction and head position of the occupant in a reference posture in which it is assumed that the occupant is not experiencing posture error, and if the face detection unit cannot detect the face of the occupant, determines that the posture of the occupant corresponds to the prone position among the plurality of posture error types;
a forward lean determination unit, when the posture type determination unit determines that the posture of the occupant corresponds to prone posture, performs a forward lean determination as to whether or not the posture of the occupant corresponds to a forward lean that does not correspond to any of the plurality of posture collapse types, based on positions of facial features of the occupant based on a detected captured image, which is an captured image retroactive to the time when the posture of the occupant was determined to correspond to prone posture and when the face detection unit was able to detect the face of the occupant;
an abnormal posture determination unit determining whether or not the posture of the occupant is an abnormal posture based on a posture imbalance type determination result by the posture type determination unit and a forward lean determination result by the forward lean determination unit. A vehicle control system comprising the abnormal attitude determination device according to claim 1 and a driving control device that controls driving of the vehicle,
the abnormal posture determination device outputs a determination result as to whether or not the occupant is in the abnormal posture to the operation control device;
The vehicle control system is characterized in that the driving control device outputs a control signal for outputting a warning to the occupants or performing evacuation processing of the vehicle based on the judgment result to an alarm control device, a steering mechanism, or a braking/driving mechanism mounted on the vehicle.

Images