JP7742973B2 - Occupant state determination device, occupant state determination system, occupant state determination method, and occupant state determination program - Google Patents

Description

This disclosure relates to an occupant status determination device, an occupant status determination system, an occupant status determination method, and an occupant status determination program.

Technology has been developed that captures images of the interior of a vehicle and uses information extracted from the occupants in the captured images to detect if the occupants are in an abnormal state.

For example, in the abnormality determination device of Patent Document 1, if a person disappears from a preset operable range within which the vehicle's steering wheel can be operated and then reappears within the operable range, if the appearance position, where the person appeared within the operable range, and the disappearance position, where the person disappeared from the operable range, are not the same, the device determines that the person who appeared and the person who disappeared are not the same person, and that the driver is not in an abnormal state. Furthermore, if the appearance position and the disappearance position are the same, the device determines that the person who appeared and the person who disappeared are the same person, and that the driver is not in an abnormal state.

JP 2020-009179 A

However, there are various possible ways in which a person may disappear or reappear within the operable range. Therefore, there are also abnormal driver conditions that cannot be detected by determining the appearance and disappearance positions. For example, if the driver becomes abnormal and disappears from the operable range, and a passenger other than the driver reappears within the operable range from the same position as the driver's disappearance in an attempt to assist the driver, the abnormality determination device of Patent Document 1 will determine that the driver is not in an abnormal state. Therefore, the abnormality determination device of Patent Document 1, which determines the driver's abnormal state based on a comparison of the appearance and disappearance positions as described above, has the problem of reduced accuracy in detecting the passenger's condition due to insufficient determination of the driver's abnormal state.

This disclosure has been made to solve such problems and aims to provide an occupant condition determination device that improves the accuracy of detecting occupant conditions.

The occupant state determination device according to the present disclosure includes a face detection unit that detects the face of an occupant in a plurality of captured images, each of which includes both a first range, which is a range in which the face of the driver of a vehicle occupant may be present, and a second range, which is a range in which a face of a person other than the driver may be present and has a range overlapping with the first range; a first face detection determination unit that performs a first face detection determination as to whether the face of the occupant has been detected in the first range in the plurality of captured images; a second face detection determination unit that, when a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit, performs a second face detection determination as to whether the face of the occupant has been detected in the first range using a second captured image of the plurality of captured images that was captured before the first captured image; and a second face detection determination unit that performs a second face detection determination as to whether the face of the occupant has been detected in the first range when a determination result indicating that the face of the occupant has not been detected is obtained from the second face detection determination unit. an occupant determination unit that, when a determination result indicating that the first face is a face other than the driver is obtained from the occupant determination unit, determines that the first face is a face other than the driver's if a first face, which is the face of an occupant detected in the first captured image, is located within a second range of the first captured image, and that the first face is the face of the driver if the first face is not located within the second range of the first captured image; an abnormal attitude determination unit that, when a determination result indicating that the first face is a face other than the driver is obtained from the occupant determination unit, determines whether the driver is in an abnormal attitude using a third captured image among the multiple captured images that was captured before the first captured image and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit; and an abnormal condition determination unit that determines whether the driver is in an abnormal condition using the determination result of the abnormal attitude determination unit.

The occupant condition determination device according to an embodiment of the present disclosure can improve the accuracy of detecting occupant conditions.

1 is a block diagram showing a configuration of an occupant state determination device according to a first embodiment; 1 is a diagram showing the interior of a vehicle, a first range, and a second range according to a first embodiment; 10A and 10B are diagrams for explaining a plurality of types of poor posture. 4 is a flowchart showing an example of the operation of the occupant state determination device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the occupant state determination device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the occupant state determination device according to the first embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
Embodiment 1.
The occupant state determination device 1 according to this embodiment determines whether the posture of a vehicle occupant is abnormal based on a captured image of the vehicle occupant.
In the following embodiment, as an example, the vehicle occupant for whom the occupant state determination device 1 determines whether or not the vehicle is in an abnormal state is the driver of the vehicle.

FIG. 1 is a block diagram showing the configuration of an occupant condition determination device 1 according to this embodiment.
FIG. 2 is a diagram showing the interior of the vehicle, a first range (to be described later), and a second range (to be described later).
In this embodiment, it is assumed that the occupant state determination device 1 is mounted on a vehicle. However, the occupant state determination device 1 is not limited to being permanently installed in a vehicle, and may be implemented in a portable device that can be carried into a vehicle, such as a portable navigation device. Furthermore, a part or all of the occupant state determination device 1 may be implemented on a server that can communicate with the vehicle.

The occupant state determination device 1 is connected to an imaging device 2 and an output device 3, and the occupant state determination device 1, the imaging device 2, and the output device 3 together constitute an occupant state determination system 100.

The imaging device 2 is mounted on the vehicle and is installed so that it can capture an imaging range that includes both a first range 6, which is the range in which the driver's face may be present, and a second range 7, which is the range in which faces other than the driver's may be present. Here, the first range 6 is the range in which the face of the driver sitting in the driver's seat to drive the vehicle is located. If the front left seat installed inside the vehicle is the driver's seat, the first range 6 is the range indicated by the dotted line in Figure 2 and is set to include the driver's seat headrest 61 and a portion of the driver's seat backrest 62. Note that this is merely an example; the first range 6 may be set as appropriate by an administrator of the occupant state determination device 1, or may be pre-stored in the occupant state determination device 1. Details of the second range 7 will be described later.

The imaging device 2 is assumed to be installed, for example, near the center of the instrument panel in the vehicle width direction, or on the center console. For example, the imaging device 2 may be a device shared with a so-called DMS (Driver Monitoring System), which is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 captures images of the interior of the vehicle at intervals of, for example, 30 to 60 fps (frames per second) and outputs the captured images to the image acquisition unit 11 of the occupant state determination device 1. When the imaging device 2 captures images of the interior of the vehicle at intervals of 30 fps, it captures 30 frames per second.

The occupant state determination device 1 determines whether or not the driver is in an abnormal state using the captured image captured by the imaging device 2. For example, the occupant state determination device 1 determines whether or not the driver is in an abnormal state for each frame. Details of the occupant state determination device 1 will be described later.
When the occupant state determination device 1 determines that the driver is in an abnormal state, it causes the output device 3 to output a warning that the driver is in an abnormal state.

The output device 3 is mounted on the vehicle and outputs a warning to the driver that an abnormal state is occurring.
The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be provided, for example, in an audio device provided in a vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the occupant state determination device 1, the output device 3 outputs a warning sound or a voice message notifying the driver that an abnormal state is present. The output device 3 may also be, for example, a display device. For example, when warning information is output from the occupant state determination device 1, the output device 3 displays a message notifying the driver that an abnormal state is present.

The output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight that is provided on the exterior of the vehicle so as to be visible from other vehicles. For example, the occupant state determination device 1 may output warning information to the output device 3 to notify people outside the vehicle, such as occupants of other vehicles, that the driver of the vehicle is in an abnormal state.
The output device 3 may be mounted on, for example, another vehicle. The occupant state determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle and output a voice or a display to notify the occupant of the other vehicle that the driver of the vehicle is in an abnormal state.

As shown in FIG. 1, the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature amount detection unit 13, a reference feature amount estimation unit 14, and a determination unit 1515.
The feature amount detection unit 13 includes a face direction detection unit 31, a head position detection unit 32, a facial feature position detection unit 33, and a detection reliability calculation unit .
The determination unit 15 includes a first face detection determination unit 51 , a second face detection determination unit 52 , an occupant determination unit 53 , an abnormal posture determination unit 54 , and an abnormal state determination unit 55 .

The image acquisition unit 11 acquires the captured image from the imaging device 2. The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

The face detection unit 12 detects the face of the driver in the first range 6 of the captured image acquired by the image acquisition unit 11 .
Specifically, the face detection unit 12 detects facial feature points of the driver, which indicate facial features of the driver, using known image recognition techniques such as edge detection on the captured image. Note that facial features include the corners and corners of the eyes, nose, mouth, eyebrows, chin, etc. The facial feature points of the driver are represented by, for example, coordinates on the captured image.
The face detection unit 12 also detects the driver's face area. The driver's face area is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's face area is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image.
The face detection unit 12 outputs information about the detected facial feature points and facial area of the driver (hereinafter referred to as "face information") to the feature amount detection unit 13. The face information is, for example, a captured image to which information that can identify the facial feature points and facial area of the driver is added.

The feature detection unit 13 detects the features of the driver's face using the driver's face detected by the face detection unit 12. The features are the driver's facial direction, the driver's head position, and the driver's facial features. In this embodiment, the driver's facial direction and head position detected by the feature detection unit 13 are, for example, the facial direction and head position in real space. In this embodiment, the positions of the driver's facial features detected by the feature detection unit 13 are, for example, the positions of the facial features in the captured image. In this embodiment, as an example, the driver's facial features are the driver's eyes. In the following description, the driver's facial features are assumed to be the driver's eyes.

The face direction detection unit 31 detects the driver's face direction in real space based on the face information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face direction detection unit 31 may detect the driver's face direction, for example, using a known face direction detection technique that detects face direction from a captured image. The driver's face direction is represented, for example, by at least one of a yaw angle, which is a rotation angle about the vehicle's vertical axis, a pitch angle, which is a rotation angle about the vehicle's horizontal axis, and a roll angle, which is a rotation angle about the vehicle's longitudinal axis, which serve as a predetermined reference. The face direction detection unit 31 outputs information regarding the driver's face direction (hereinafter referred to as "face direction information") to the reference feature estimation unit 14 and the determination unit 15. The face direction information includes at least one of yaw angle, pitch angle, and roll angle information.

The head position detection unit 32 detects the driver's head position in real space based on the face information output from the face detection unit 12, specifically based on the driver's face in the captured image detected by the face detection unit 12. In this embodiment, the driver's head position in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 32 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the driver's head position in the captured image may also be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the head position detection unit 32 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space.

The head position detection unit 32 may detect the driver's head position, for example, using a known coordinate transformation technique that converts points on a captured image into points in real space. The driver's head position is expressed, for example, by coordinates in real space. The head position detection unit 32 outputs information related to the detected driver's head position (hereinafter referred to as "head position information") to the reference feature estimation unit 14 and the determination unit 15. The head position information includes coordinate information of the driver's head position.

The feature position detection unit 33 detects the position of the driver's eyes in the captured image based on the face information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face detection unit 12 detects the positions of the driver's right eye and left eye in the captured image. In this embodiment, the position of the driver's eyes is defined as the center of a line connecting the inner corner and outer corner of the driver's eye in the captured image. The feature position detection unit 33 may detect the position of the driver's eyes using, for example, known image recognition technology. The position of the driver's eyes is represented, for example, by coordinates on the captured image. The feature position detection unit 33 outputs information regarding the detected position of the driver's eyes (hereinafter referred to as "feature position information") to the determination unit 15. The feature position information includes coordinate information for the position of the driver's right eye and coordinate information for the position of the driver's left eye.

The detection reliability calculation unit 34 calculates a face detection reliability indicating the reliability that at least one of the detected face direction and head position of the driver is the predetermined face direction and head position of the driver. The detection reliability calculation unit 34 also calculates a face part detection reliability indicating the reliability that the detected face parts are the predetermined face parts.
The detection reliability calculation unit 34 then outputs the calculated face detection reliability and facial part detection reliability to the determination unit 15. The face detection reliability and facial part detection reliability can be calculated, for example, using the methods described in the following documents. In the following document, the face detection reliability is calculated from the degree of match between at least one of a face direction and a head position learned in advance from a large number of face images and at least one of face direction and head position parts extracted from an image of the driver. Furthermore, the facial part detection reliability is calculated from the degree of match between face parts learned in advance from a large number of face images and face parts extracted from an image of the driver.
“Rapid Object Detection Boosted Cascade of Simple Features”, 2001, Paul Viola, etal.

In the following description, the features detected by the feature detection unit 13 refer to the driver's facial orientation, the driver's head position, and the driver's facial features. Furthermore, of the features detected by the feature detection unit 13, at least one of the driver's facial orientation and head position is defined as a first feature, and at least one of the face detection reliability and facial feature detection reliability derived from the features is defined as a second feature.

The reference feature estimation unit 14 estimates the driver's reference feature as a reference feature using at least one of facial direction information and head position information. Specifically, the reference feature estimation unit 14 estimates at least one of the driver's reference facial direction and head position as a reference feature using at least one of the driver's facial direction and head position detected over a predetermined period (hereinafter referred to as the "reference setting period"). The reference setting period is, for example, the period from when the vehicle ignition is turned on until a predetermined time (e.g., 10 seconds) has elapsed. Note that this is merely an example, and the reference setting period may be, for example, a predetermined time (e.g., 3 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km/h) and the steering angle is within a predetermined range (e.g., ±20 degrees).

In addition, at least one of the driver's face direction and head position is used to estimate the reference feature, but the reference feature may also be estimated using at least one of the driver's face direction and head position and the face detection reliability.
For example, the reference feature estimation unit 14 estimates the average value of at least one of the driver's face direction and head position during the reference setting period as at least one of the driver's face direction and head position in the driver's reference feature.
The reference feature estimation unit 14 outputs information on the estimated reference feature of the driver (hereinafter referred to as "reference feature information") to the determination unit 15. The reference feature information includes information on at least one of the facial orientation and head position in the reference feature of the driver. The facial orientation in the reference feature of the driver is expressed by at least one of the yaw angle, pitch angle, and roll angle. The facial orientation and head position in the reference feature of the driver is expressed, for example, by coordinates in real space. The reference feature information also includes coordinate information of the driver's head position as information on the driver's head position.

In the following description, the most recent captured image among the captured images acquired by the image acquisition unit 11 will be referred to as the first captured image. The first captured image may also be referred to as the current frame.

The first face detection determination unit 51 uses the second feature amount calculated in the captured image by the feature amount detection unit 13 to perform a first face detection determination as to whether or not the face of the occupant has been detected within the first range.
In the following description, the process of the occupant state determination device when the captured image used for the first face detection determination is the first captured image will be described.
For example, when the first face detection judgment unit 51 performs a first face detection judgment in the first captured image, the first face detection judgment unit 51 performs a first face detection judgment as to whether or not the occupant's face has been detected in the first range 6 using the first face detection feature, which is the second feature calculated by the feature detection unit 13 in the first captured image.

First, we will explain the first face detection judgment when using the first face detection reliability, which is the first face detection feature. The first face detection judgment unit 51 judges whether the first face detection reliability is equal to or greater than a fourth threshold. If the first face detection reliability is equal to or greater than the fourth threshold, the first face detection judgment unit 51 determines that the occupant's face has been detected, and if the first face detection reliability is less than the fourth threshold, it determines that the occupant's face has not been detected.

When the first face detection reliability is equal to or greater than the fourth threshold, the first face detection determination unit 51 outputs a first determination result, which is a determination result including information indicating that the occupant's face was detected and information indicating the first captured image, to the second face detection determination unit 52, the occupant determination unit 53, and the abnormal posture determination unit 54. When the first face detection reliability is less than the fourth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the first captured image to the abnormal posture determination unit 54.

Next, we will explain the first face detection judgment when using the first facial part detection reliability, which is the first face detection feature. The first face detection judgment unit 51 judges whether the first facial part detection reliability is equal to or greater than a fifth threshold. If the first facial part detection reliability is equal to or greater than the fifth threshold, the first face detection judgment unit 51 determines that the occupant's face has been detected, and if the first facial part detection reliability is less than the fifth threshold, it determines that the occupant's face has not been detected.

When the first facial part detection reliability is equal to or greater than the fifth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was detected and information indicating the first captured image to the second face detection determination unit 52. When the first facial part detection reliability is less than the fifth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the first captured image to the abnormal posture determination unit 54.

While an example has been shown in which the first face detection determination is made using either the first face detection reliability or the first facial part detection reliability, the present invention is not limited to this, and the first face detection determination may be made using both the first face detection reliability and the first facial part detection reliability. In this case, the first face detection determination unit 51 determines that the occupant's face has been detected if the first face detection reliability is equal to or greater than the fourth threshold and the first facial part detection reliability is equal to or greater than the fifth threshold, and determines that the occupant's face has not been detected if the first face detection reliability is less than the fourth threshold or the first facial part detection reliability is less than the fifth threshold.

When the second face detection determination unit 52 receives a determination result from the first face detection determination unit 51 indicating that the occupant's face was detected in the first captured image, it performs a second face detection determination to determine whether the occupant's face was detected in the first range 6 using a second captured image captured before the first captured image. Specifically, the second face detection determination unit 52 performs the second face detection determination using second face detection features, which are second features calculated in the second captured image by the feature detection unit 13. Specifically, the second captured image is the captured image one frame before the first captured image. The second captured image may also be referred to as the immediately preceding frame.

First, we will explain the second face detection judgment when using the second face detection reliability, which is the second face detection feature. The second face detection judgment unit 52 judges whether the second face detection reliability is equal to or greater than a fourth threshold. If the second face detection reliability is equal to or greater than the fourth threshold, the second face detection judgment unit 52 determines that the occupant's face has been detected, and if the second face detection reliability is less than the fourth threshold, the second face detection judgment unit 52 determines that the occupant's face has not been detected.

When the second face detection reliability is equal to or greater than the fourth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was detected and information indicating the second captured image to the abnormal posture determination unit 54. When the second face detection reliability is less than the fourth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the second captured image to the occupant determination unit 53.

Next, we will explain the second face detection judgment when using the second facial part detection reliability, which is the second face detection feature. The second face detection judgment unit 52 judges whether the second facial part detection reliability is equal to or greater than a fifth threshold. If the second facial part detection reliability is equal to or greater than the fifth threshold, the second face detection judgment unit 52 determines that the occupant's face has been detected, and if the second facial part detection reliability is less than the fifth threshold, the second face detection judgment unit 52 determines that the occupant's face has not been detected.

When the second facial part detection reliability is equal to or greater than the fifth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was detected and information indicating the second captured image to the abnormal posture determination unit 54. When the second facial part detection reliability is less than the fifth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the second captured image to the occupant determination unit 53.

While an example has been shown in which the second face detection determination is made using either the second face detection reliability or the second facial part detection reliability, the present invention is not limited to this, and the second face detection determination may be made using both the second face detection reliability and the second facial part detection reliability. In this case, the second face detection determination unit 52 determines that the occupant's face has been detected if the second face detection reliability is equal to or greater than the fourth threshold and the second facial part detection reliability is equal to or greater than the fifth threshold, and determines that the occupant's face has not been detected if the second face detection reliability is less than the fourth threshold or the second facial part detection reliability is less than the fifth threshold.

When the occupant determination unit 53 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face could not be detected, it determines that the first face, which is the occupant's face detected in the first captured image, is a face other than that of the driver if the first face is located within a second range of the first captured image, and determines that the first face is the driver's face if the first face is not located within the second range of the first captured image.

Here, we will explain the second range 7 when the face other than the driver's is the face of another occupant other than the driver. The second range 7 is the range where the face of the other occupant is likely to be located. More specifically, the second range 7 includes the range where the faces of other occupants seated in seats other than the driver's seat of the vehicle are located, as well as the range where the faces of other occupants attempting to assist the driver in an abnormal condition are located. As shown in FIG. 2 , the second range 7 is the range indicated by the dashed-dotted line and is set to include a portion of the driver's seat backrest 62, headrests of seats other than the driver's seat, such as the passenger seat headrest 71 and the rear seat headrest 73, and portions of the backrests of seats other than the driver's seat, such as the passenger seat backrest 72 and the rear seat backrest 74. Note that this is merely an example; the second range 7 may be set as appropriate by an administrator of the occupant state determination device 1, or may be pre-stored in the occupant state determination device 1.

In this embodiment, an example has been given in which the faces other than the driver are the faces of other passengers, but this is not limited to this, and faces other than the driver also include patterns with shapes or structural features similar to those of a human face, specifically faces printed on masks or clothing, landscape patterns, etc.

Here, we will explain why the occupant determination unit 53 performs occupant determination. First, when the occupant determination unit 53 receives a determination result from the second face detection determination unit 52 indicating that the occupant's face could not be detected (hereinafter referred to as "Case A"), this refers to a case where the occupant's face was detected in the first range 6 set in the first captured image but not in the first range 6 set in the second captured image. As previously mentioned, the second captured image was captured before the first captured image. In other words, Case A refers to a case where the occupant's face could no longer be detected by the occupant state determination device 1, but the occupant's face can be detected again (hereinafter referred to as "face lost return"). For example, face lost return includes a state where the driver disappears from the first range 6 to pick up an object near the driver's seat and then reappears in the first range 6, or a state where the driver becomes abnormal and disappears from the first range 6, and another occupant reappears in the first range 6 to assist the driver. In other words, when face loss recovery occurs, there is a possibility that the face of an occupant detected in the first range 6 set in the first captured image is a face other than that of the driver. Therefore, if the occupant determination unit 53 does not perform occupant determination, an abnormal posture determination, which will be described later, is performed using information about faces other than that of the driver. In other words, if information about faces other than that of the driver, which are not the subject of the occupant state determination device 1's determination of whether or not an abnormal state exists, is used in the abnormal posture determination, the accuracy of the abnormal state determination, which will be described later, performed using the results of the abnormal posture determination, is reduced. As a result, there is a possibility that the accuracy of detecting the occupant state in the occupant state determination device 1 is reduced. For the above reasons, the occupant determination unit 53 performs occupant determination.

Specifically, the occupant determination unit 53 determines whether or not the face of the occupant is located at least within the second range 7, thereby performing the occupant determination.
In this embodiment, five types of occupant determination will be described, which are performed by the occupant determination unit 53. The occupant determination unit 53 performs one of the five occupant determinations.

In the following description, the first feature detected by the feature detection unit 13 in the first captured image will be referred to as the first judgment feature, and the first feature detected by the feature detection unit 13 in the third captured image, which is an image captured before the first captured image and for which the first face detection judgment unit 51 has output a judgment result indicating that the occupant's face was detected, will be referred to as the third judgment feature.

Here, we will explain the third captured image. The third captured image is determined using the judgment results output by the first face detection judgment unit 51 up until the occupant judgment using the first captured image. Specifically, the third captured image is an image captured one or more frames before the first captured image, and is an captured image in which the first face detection judgment unit 51 judges that the occupant's face has been detected.

For example, consider the case where the first face detection determination unit 51 determines that an occupant's face was detected in the first captured image, determines that an occupant's face was not detected in the captured image one to three frames before the first captured image, and determines that an occupant's face was detected in the first captured image four frames before the first captured image (hereinafter referred to as "Case B"). In Case B, the first face detection determination unit 51 outputs a determination result corresponding to the captured image four frames before the first captured image to the occupant determination unit 53, and then outputs a determination result corresponding to the first captured image. Then, in Case A, the occupant determination unit 53 performs occupant determination using the captured image four frames before the first captured image as the third captured image.

In other words, Case B refers to a situation in which the occupant's face was detected in the image captured four frames prior to the first captured image, but was no longer detectable in the images captured one to three frames prior to the first captured image, and was then re-detectable in the first captured image. Here, a situation in which the occupant state assessment device 1 was able to detect the occupant's face but is now unable to detect it is referred to as a "face lost" occurrence, a situation in which the occupant's face is no longer detectable, and, as mentioned above, a situation in which the occupant's face becomes undetectable but is then re-detectable is referred to as a "face lost recovery." The first captured image is the image captured after face lost recovery, the second captured image is the image captured during face lost, and the third captured image is the image captured before face lost occurred. In the following description, the third captured image may also be referred to as the frame before face lost occurred.

First, the occupant determination performed using the first head position, which is the first determination feature, will be described.
The occupant determination unit 53 determines whether the first head position falls within the second range 7 .
If the first head position falls within the second range 7, the occupant determination unit 53 determines that the first face is the face of a person other than the driver, and if the first head position does not fall within the second range 7, it determines that the first face is the face of the driver.

If the first head position falls within the second range 7, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54.
If the first head position does not fall within the second range 7, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

Next, the occupant determination performed using the first head position and first face orientation, which are the first determination feature amounts, will be described.
The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the first face direction is less than a first threshold value.
The occupant determination unit determines that the first face is a face other than that of the driver if the first head position falls within the second range and the absolute value of the first facial direction is less than a first threshold, and determines that the first face is the face of the driver if the first head position does not fall within the second range or the absolute value of the first facial direction is equal to or greater than the first threshold.
Here, the third range may be appropriately set by a manager of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1 .

If the first head position falls within the second range 7 and the first facial orientation falls within the third range, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54.
If the first head position does not fall within the second range 7 or the first facial orientation does not fall within the third range, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

Next, an occupant determination performed using the first head position, which is the first determination feature amount, and the second head position, which is the third determination feature amount, will be described.
The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the head position difference, which is the difference between the first head position and the third head position, is greater than or equal to a second threshold value.
The occupant determination unit 53 determines that the first face is the face of a person other than the driver if the first head position falls within the second range 7 and the absolute value of the head position difference is greater than or equal to the second threshold, and determines that the first face is the face of the driver if the first head position does not fall within the second range 7 or the absolute value of the head position difference is less than the second threshold.
Here, the second threshold value may be appropriately set by a manager of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1 .

If the first head position falls within the second range 7 and the absolute value of the head position difference is greater than or equal to the second threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54.
If the first head position does not fall within the second range 7 or the absolute value of the head position difference is less than the second threshold value, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

Next, an occupant determination performed using the first head position and first face direction, which are the first determination feature amounts, and the second face direction, which is the third determination feature amount, will be described.
The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the facial direction difference, which is the difference between the first facial direction and the third facial direction, is equal to or greater than a third threshold value.
The occupant determination unit 53 determines that the first face is the face of a person other than the driver if the first head position falls within the second range 7 and the absolute value of the facial direction difference is equal to or greater than the third threshold, and determines that the first face is the face of the driver if the first head position does not fall within the second range 7 or the absolute value of the facial direction difference is less than the third threshold.
Here, the third threshold value may be appropriately set by a manager of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1 .

If the first head position falls within the second range 7 and the absolute value of the facial direction difference is greater than or equal to the third threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54.
If the first head position does not fall within the second range 7 or the absolute value of the facial direction difference is less than the third threshold value, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

Next, we will explain occupant determination performed using the first head position, which is the first determination feature, and recognition information acquired from a recognition unit (not shown) configured to recognize people in the first captured image.
The occupant determination unit 53 determines whether or not the first recognition information, which is information for recognizing the first face, matches the driver recognition information, which is information for recognizing the face of the driver.
If the first head position falls within the second range 7 and the first recognition information does not match the driver recognition information, the occupant determination unit 53 determines that the first face is the face of a person other than the driver, and if the first head position does not fall within the second range 7 or the first recognition information matches the driver recognition information, the occupant determination unit 53 determines that the first face is the face of the driver.

If the first head position falls within the second range 7 and the first recognition information does not match the driver identification information, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54.
If the first head position does not fall within the second range 7 or the first recognition information matches the driver identification information, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

When the abnormal posture judgment unit 54 obtains a judgment result from the second face detection judgment unit 52 indicating that the occupant's face has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the driver's face, it performs an abnormal posture judgment using the first captured image.
When the abnormal posture judgment unit 54 obtains a judgment result from the first face detection judgment unit 51 indicating that the occupant's face could not be detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the face of a person other than the driver, the abnormal posture judgment unit 54 performs an abnormal posture judgment using the third captured image.

Specifically, when the abnormal posture judgment unit 54 obtains a judgment result from the second face detection judgment unit 52 indicating that the occupant's face has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the driver's face, the abnormal posture judgment unit 54 performs an abnormal posture judgment using the first judgment feature.
When the abnormal posture judgment unit 54 obtains a judgment result from the first face detection judgment unit 51 indicating that the occupant's face could not be detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the face of a person other than the driver, it performs an abnormal posture judgment using the third judgment feature.

Here, we will explain why the abnormal posture judgment unit 54 uses the first judgment feature to make an abnormal posture judgment when it obtains a judgment result from the second face detection judgment unit 52 indicating that an occupant's face has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the driver's face.

First, when the abnormal posture determination unit 54 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face has been detected (hereinafter referred to as "Case C"), this refers to a case where the occupant's face is detected in the first range 6 set in the first captured image and also in the first range 6 set in the second captured image. In other words, Case C refers to a state where the occupant's face is continuously detected by the occupant state determination device 1. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be the driver's face. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the first determination feature.

Next, when the abnormal posture determination unit 54 obtains a determination result from the occupant determination unit 53 indicating that the first face is the driver's face, this refers to a case where the occupant's face is detected again after face loss recovery and is the driver's face. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be the driver's face. Therefore, the abnormal posture determination unit 54 performs an abnormal posture determination using the first determination feature.

Next, we will explain why the abnormal posture judgment unit 54 uses the third judgment feature to make an abnormal posture judgment when it obtains a judgment result from the first face detection judgment unit 51 indicating that the occupant's face could not be detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the face of someone other than the driver.

First, when the abnormal posture determination unit 54 obtains a determination result from the first face detection determination unit 51 indicating that the occupant's face could not be detected, this refers to a case where face loss has occurred or face loss has occurred. In other words, since the occupant's face could not be detected in the first range 6 set in the first captured image, abnormal posture determination is performed using information about the occupant's face detected in the first range 6 set in the third captured image. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the third determination feature.

Next, when the abnormal posture determination unit 54 obtains a determination result from the occupant determination unit 53 indicating that the first face is a face other than that of the driver, this refers to a case where the face of the occupant detected again is a face other than that of the driver after face loss recovery. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be a face other than that of the driver. Furthermore, because the occupant's face cannot be detected in the first range 6 set in the second captured image, the abnormal posture determination unit 54 performs the abnormal posture determination using information about the occupant's face detected in the first range 6 set in the third captured image. Therefore, the abnormal posture determination unit 54 performs the abnormal posture determination using the third determination feature.

The abnormal posture determination unit 54 will be described in detail below.
The abnormal posture judgment unit 54 determines whether to use the first captured image or the third captured image as the judgment image, which is the captured image used for abnormal posture judgment, based on the judgment results obtained from the first face detection judgment unit 51, the second face detection judgment unit 52, or the occupant judgment unit 53.
Then, the abnormal state determination unit 55 sets the first feature amount detected in the captured image adopted as the determination-use captured image as the determination-use feature amount.
Then, the abnormal posture determination unit 54 determines whether the driver's posture corresponds to any of a plurality of predetermined posture imbalance types (hereinafter referred to as "posture imbalance types") by comparing the determination feature with the reference feature obtained from the reference feature estimation unit 14.
The abnormal state determination unit 55 determines that the occupant has an abnormal posture if the driver's posture corresponds to any of the posture error types, and determines that the occupant does not have an abnormal posture if the driver's posture does not correspond to any of the posture error types.

Figure 3 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram illustrating several types of poor posture. As shown in Figure 3, there are several types of poor posture. Figure 3 illustrates the following: "head down," in which the driver leans forward and continues to have their face close to the steering wheel; "head down," in which the driver continues to have their face facing down; "backward lean," in which the driver's upper body leans backward and continues to have their face facing up; "backward lean," in which the driver's upper body leans up and continues to have their face facing up; "head tilted to the side," in which the driver's face is tilted to the left or right; "sideways lean," in which the driver's upper body leans to the left or right and their face is tilted in the same direction; and "leaning to the side," in which the driver's upper body is tilted to the left or right. In the present disclosure, the multiple pre-set posture abnormality types are "head down," "head down," "backward arch," "arched back," "head tilted to the side only," "side tilt," and "leaning to the side" shown in Fig. 3. Information on the multiple posture abnormality types is stored in a location that can be referenced by the abnormal posture determination unit 54.

In detail, the abnormal posture determination unit 54 determines which of multiple posture abnormality types the driver falls into based on how much the determination feature has changed compared to the reference feature. The conditions for determining which posture abnormality type the driver falls into based on how much the determination feature has changed compared to the reference feature are set in advance and stored in the abnormal posture determination unit 54.

If the driver's posture corresponds to one of the posture error types, the abnormal posture determination unit 54 outputs a determination result including information indicating that the driver has an abnormal posture to the abnormal state determination unit 55, and if the driver's posture does not correspond to one of the posture error types, the abnormal posture determination unit 54 outputs a determination result including information indicating that the driver does not have an abnormal posture to the abnormal state determination unit 55.

The abnormal state determination unit 55 uses the determination result output by the abnormal posture determination unit 54 to determine whether the driver is in an abnormal state.

When the abnormal posture determination unit 54 determines that the driver has an abnormal posture, the abnormal condition determination unit 55 determines the duration of the state in which the driver has an abnormal posture. For example, the abnormal condition determination unit 55 associates the determination result obtained from the abnormal posture determination unit 54 with information on the date and time when the determination result was obtained and stores the result in a memory unit (not shown). The abnormal condition determination unit 55 can determine the duration of the state in which the driver has an abnormal posture from the stored determination result. When the duration of the state in which the driver has an abnormal posture reaches a sixth threshold, the abnormal condition determination unit 55 determines that the driver is in an abnormal state.

The abnormal state determination unit 55 outputs the determination result as to whether the driver is in an abnormal state to the output device 3.

Figure 4 is a flowchart showing an example of the operation of the occupant state determination device 1 according to this embodiment. The processing executed by the occupant state determination device 1 according to this embodiment will be described with reference to the flowchart shown in Figure 4. Note that the operation of the occupant state determination device 1 is initiated, for example, after the vehicle ignition is turned on. It is also preferable to perform the occupant state determination processing each time an image is acquired from the imaging device 2, that is, for each frame.

The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1).
The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12 .

After step ST1, the face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2).
The face detection unit 12 outputs the face information to the feature amount detection unit 13 .

After step ST2, the face direction detection unit 31 detects the face direction of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3).
The face direction detection unit 31 outputs face direction information to the reference feature amount estimation unit 14 and the determination unit 15 .

After step ST2, the head position detection unit 32 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3).
The head position detection unit 32 outputs the head position information to the reference feature estimation unit 14 and the determination unit 15 .

After step ST2, the part position detection unit 33 detects the part positions of the driver's face in real space based on the face information output from the face detection unit 12 in step ST2, specifically based on the driver's face in the captured image detected by the face detection unit 12 (step ST3).
The part position detection unit 33 outputs the part position information to the determination unit 15 .

After step ST3, the detection reliability calculation unit 34 calculates the face detection reliability and the facial part detection reliability based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4).
The detection reliability calculation unit 34 outputs the face detection reliability and the face part detection reliability to the determination unit 15 .

After step ST4, the reference feature estimation unit 14 estimates a reference feature using at least one of the face direction information and the head position information (step ST5).
The detection reliability calculation unit 34 outputs the reference feature amount to the determination unit 15 .

After step ST5, the first face detection determination unit 51 performs a first face detection determination as to whether or not the face of an occupant has been detected in the first range 6 using the first face detection feature amount (step ST6).
Step ST6 indicates whether the first face detection determination unit 51 has succeeded in detecting the face of the occupant in the current frame.

If the first face detection judgment unit 51 is able to detect the occupant's face (if "YES" in step ST6), the second face detection judgment unit 52 uses the second captured image to make a second face detection judgment as to whether or not the occupant's face was detected in the first range 6 (step ST7).
Step ST7 indicates whether the second face detection determination unit 52 has succeeded in detecting the face of the occupant in the immediately preceding frame.

If the second face detection judgment unit 52 is unable to detect the face of an occupant (if "NO" in step ST7), the occupant judgment unit 53 uses the second range 7 to make an occupant judgment as to whether the first face is a face other than that of the driver (step ST8).
Step ST9 means that the occupant determination unit 53 determines whether the face of the occupant detected in the current frame is a face other than that of the driver.

If the first face is a face other than that of the driver (if the answer is "YES" in step ST8), the occupant determination unit 53 and the abnormal posture determination unit 54 use the frame before face loss occurred as the captured image for determination (step ST9).

If the second face detection judgment unit 52 is able to detect the face of an occupant (if "YES" in step ST7), or if the first face is the face of the driver (if "NO" in step ST8), the abnormal posture judgment unit 54 adopts the current frame as the captured image for judgment (step ST10).

If the first face detection judgment unit 51 is unable to detect the occupant's face (if the answer is "NO" in step ST6), the occupant judgment unit 53 and the abnormal posture judgment unit 54 will use the frame before the face loss occurred as the captured image for judgment (step ST11).

After step ST9, step ST10, or step ST11, the abnormal posture determination unit 54 performs abnormal posture determination using the determination feature and the reference feature (step ST12).

If the abnormal posture determination unit 54 determines that the driver is in an abnormal posture ("YES" in step ST12), the abnormal state determination unit 55 determines whether the duration of the driver's abnormal posture is equal to or greater than the sixth threshold value (step ST13).

If the duration of the driver's abnormal posture is equal to or greater than the sixth threshold ("YES" in step ST13), the abnormal state determination unit 55 determines that the driver is in an abnormal state (step ST14).

If the abnormal posture determination unit 54 determines that the driver is not in an abnormal posture (if "NO" in step ST12), or if the duration of the driver's abnormal posture is less than the sixth threshold (if "NO" in step ST13), the abnormal state determination unit 55 determines that the driver is not in an abnormal state (step ST15).

Figures 5 and 6 are diagrams showing an example of the hardware configuration of the occupant state determination device 1 according to embodiment 1. In embodiment 1, the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference feature estimation unit 14, and determination unit 15 are realized by a processing circuit 1001. That is, the occupant state determination device 1 includes a processing circuit 1001 for controlling the determination of whether a vehicle occupant is in an abnormal state based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in Figure 5, or a processor 1004 that executes a program stored in memory 1005 as shown in Figure 6.

If the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

When the processing circuit is a processor 1004, the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference feature estimation unit 14, and judgment unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 reads and executes the program stored in memory 1005 to perform the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference feature estimation unit 14, and judgment unit 15. In other words, the occupant state judgment device 1 includes memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST15 of FIG. 4 described above. It can also be said that the programs stored in the memory 1005 cause the computer to execute the processing procedures or methods of the image acquisition unit 11, face detection unit 12, feature amount detection unit 13, reference feature amount estimation unit 14, and determination unit 15. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, a DVD (Digital Versatile Disc), or the like.

The functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference feature estimation unit 14, determination unit 15, and output control unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the image acquisition unit 11 may be realized by a processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, feature detection unit 13, reference feature estimation unit 14, and determination unit 15 may be realized by the processor 1004 reading and executing programs stored in memory 1005.

The storage unit (not shown) is, for example, composed of a memory 1005. The occupant state determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as an imaging device 2 or an output device 3.

The occupant state determination device 1 of this embodiment includes a face detection unit 12 that detects the face of an occupant in a plurality of captured images that each include both a first range 6, which is a range in which the face of the driver of the vehicle may be present, and a second range 7, which is a range in which the face of an occupant other than the driver may be present; a first face detection determination unit 51 that performs a first face detection determination as to whether the face of the occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination unit 52 that, when a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit 51, performs a second face detection determination as to whether the face of the occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and a determination result indicating that the face of the occupant has not been detected from the second face detection determination unit 52. an occupant determination unit that, when a determination result indicating that the first face is a face other than the driver's, determines that the first face is a face other than the driver's when a first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and that the first face is a face of the driver when the first face is not located in the second range 7 of the first captured image; an abnormal attitude determination unit that, when a determination result indicating that the first face is a face other than the driver's, is obtained from the occupant determination unit 53, determines whether or not the driver is in an abnormal attitude using a third captured image, which was taken before the first captured image among the multiple captured images and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit 51; and an abnormal state determination unit that, using the determination result of the abnormal attitude determination unit 54, determines whether or not the driver is in an abnormal state.
When an occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination device 1 uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, the occupant state determination device 1 can determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about a face other than that of the driver, which is not the target of abnormal state determination, is used in abnormal posture determination. As a result, the occupant state determination device 1 can improve its detection accuracy of the occupant state.

The occupant status determination system 100 of this embodiment is equipped with the occupant status determination device 1 of this embodiment, an imaging device 2 installed in the vehicle so as to be able to capture an image of at least the face of the occupant, and an output device 3 mounted on the vehicle and outputting a warning that the occupant is in an abnormal state, where the occupant status determination device 1 uses the image captured by the imaging device 2 to determine whether the occupant is in an abnormal state, and if it determines that the occupant is in an abnormal state, causes the output device 3 to output a warning that the occupant is in an abnormal state.
When an occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination system 100 uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, the occupant state determination system 100 can determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about a face other than that of the driver, which is not the target of abnormal state determination, is used in abnormal posture determination. As a result, the occupant state detection accuracy in the occupant state determination system 100 can be improved.

The occupant state determination method of this embodiment includes a face detection step in which the face detection unit 12 detects the face of an occupant in a plurality of captured images each including both a first range 6, which is a range in which the face of the driver of the vehicle occupants can be present, and a second range 7, which is a range in which faces other than the driver can be present; a first face detection determination step in which the first face detection determination unit 51 makes a first face detection determination as to whether the face of the occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination step in which the second face detection determination unit 52, when obtaining from the first face detection determination unit 51 a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether the face of the occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and a second face detection determination step in which the occupant determination unit 53 determines whether the face of the occupant has been detected from the second face detection determination unit 52. an occupant determination step in which, when a determination result indicating that the first face was not located in the second range 7 of the first captured image is obtained, the abnormal posture determination unit 54 performs an occupant determination that the first face is a face other than that of the driver if the first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and that the first face is the face of the driver if the first face is not located in the second range 7 of the first captured image; an abnormal posture determination step in which, when the abnormal posture determination unit 54 obtains a determination result indicating that the first face is a face other than that of the driver from the occupant determination unit 53, the abnormal posture determination unit 54 performs an abnormal posture determination as to whether or not the driver is in an abnormal posture using a third captured image among the multiple captured images that was taken before the first captured image and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit 51; and an abnormal state determination step in which the abnormal state determination unit 55 performs an abnormal state determination as to whether or not the driver is in an abnormal state using the determination result of the abnormal posture determination unit 54.
In the occupant state determination method, when the occupant's face disappears from the first range 6 and then reappears in the first range 6, the second range 7 is used to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, it is possible to determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about a face other than that of the driver, which is not the target of abnormal state determination, is used in abnormal posture determination. As a result, the occupant state detection accuracy can be improved in the occupant state determination method.

The occupant state determination program of the present embodiment includes a computer having a face detection function in which a face detection unit detects the face of an occupant in a plurality of captured images each including both a first range 6, which is a range in which the face of the driver of the vehicle occupants may be present, and a second range 7, which is a range in which the faces of persons other than the driver may be present; a first face detection determination function in which a first face detection determination unit 51 makes a first face detection determination as to whether or not the face of an occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination function in which a second face detection determination unit 52, when obtaining from the first face detection determination unit 51 a determination result indicating that the face of an occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether or not the face of an occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and an occupant determination unit 53, when obtaining from the second face detection determination unit 52 a determination result indicating that the face of an occupant has been detected in the first range 6. an occupant determination function that, when a determination result indicating that the face of an occupant could not be detected is acquired, determines that the first face, which is the face of an occupant detected in the first captured image, is a face other than that of the driver if the first face is located in the second range 7 of the first captured image, and that the first face is the face of the driver if the first face is not located in the second range 7 of the first captured image; an abnormal attitude determination function that, when the abnormal attitude determination unit 54 acquires a determination result indicating that the first face is a face other than that of the driver from the occupant determination unit 53, determines whether or not the driver has an abnormal attitude using a third captured image among the multiple captured images that was captured before the first captured image and for which a determination result indicating that the face of an occupant could be detected was output by the first face detection determination unit 51; and an abnormal state determination function that the abnormal state determination unit 55 uses the determination result of the abnormal attitude determination unit 54 to determine whether or not the driver is in an abnormal state.
When the occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination program uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, it is possible to determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about faces other than the driver, which are not subject to abnormal state determination, is used in abnormal posture determination. As a result, the occupant state determination program can improve its detection accuracy for the occupant state.

In this embodiment, an example has been shown in which the most recent captured image among the captured images acquired by the image acquisition unit 11 is set as the first captured image, but this is merely an example, and the first captured image may be an image captured at any point in time among the captured images acquired by the image acquisition unit 11.

1 Occupant state determination device, 2 Imaging device, 3 Output device, 11 Image acquisition unit, 12 Face detection unit, 13 Feature detection unit, 14 Reference feature estimation unit, 15 Determination unit, 31 Face direction detection unit, 32 Head position detection unit, 33 Part position detection unit, 34 Detection reliability calculation unit, 51 First face detection determination unit, 52 Second face detection determination unit, 53 Occupant determination unit, 54 Abnormal posture determination unit, 55 Abnormal state determination unit 100 Occupant state determination system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory

Claims (13)

a face detection unit that detects the faces of the vehicle occupants in a plurality of captured images that each include both a first range, which is a range in which the face of the driver of the vehicle may be present, and a second range, which is a range in which the faces of persons other than the driver may be present and has a range that overlaps with the first range;
a first face detection determination unit that performs a first face detection determination as to whether or not the face of the occupant has been detected within the first range in the plurality of captured images;
a second face detection determination unit that, when a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit, performs a second face detection determination as to whether the face of the occupant has been detected in the first range using a second captured image of the plurality of captured images that has been captured before the first captured image; and
an occupant determination unit that, when a determination result indicating that the face of the occupant could not be detected is obtained from the second face detection determination unit, determines that a first face, which is the face of the occupant detected in the first captured image, is a face other than that of the driver if the first face is located within the second range of the first captured image, and determines that the first face is the face of the driver if the first face is not located within the second range of the first captured image;
an abnormal posture determination unit that, when a determination result indicating that the first face is a face other than that of the driver is obtained from the occupant determination unit, performs an abnormal posture determination as to whether the driver is in an abnormal posture using a third captured image among the plurality of captured images that was captured before the first captured image and for which a determination result indicating that the face of the occupant was detected was output by the first face detection determination unit;
an abnormal state determination unit that determines whether the driver is in an abnormal state using a determination result of the abnormal posture determination unit;
An occupant state determination device comprising: the abnormal posture determination unit, when acquiring a determination result from the second face detection determination unit indicating that the face of the occupant has been detected, or when acquiring a determination result from the occupant determination unit indicating that the first face is the face of the driver, performs the abnormal posture determination using the first captured image;
the abnormal attitude determination unit, when acquiring a determination result from the first face detection determination unit indicating that the face of the occupant could not be detected, performs the abnormal attitude determination using the third captured image.
The occupant condition determination device according to claim 1 . a feature amount detection unit that detects a feature amount of the occupant's face using the face of the occupant detected by the face detection unit, determines a first feature amount that is a specific feature amount from the detected feature amount, and calculates a second feature amount derived from the feature amount,
the feature amount includes a head position of the occupant, a facial direction of the occupant, and a position of a facial feature of the occupant;
the first feature amount includes at least one of a head position of the occupant and a facial orientation of the occupant,
the second feature amount includes at least one of a face detection reliability indicating a detection reliability of at least one of a head position of the occupant and a facial orientation of the occupant, and a face part detection reliability indicating a detection reliability of the feature position;
the face detection reliability is derived from at least one of a head position of the occupant and a facial orientation of the occupant;
the facial feature detection reliability is derived from the facial feature positions ;
the first face detection determination unit performs the first face detection determination in the first captured image by using a first face detection feature amount, which is the second feature amount calculated in the first captured image by the feature amount detection unit;
the second face detection determination unit performs the second face detection determination using a second face detection feature amount, which is the second feature amount calculated for the second captured image by the feature amount detection unit; and
the occupant determination unit performs the occupant determination using at least a first determination feature amount, which is the first feature amount detected in the first captured image by the feature amount detection unit.
The occupant condition determination device according to claim 1 or 2. the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range;
the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range;
the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range.
The occupant condition determination device according to claim 3 . the occupant determination unit determines whether a first head position, which is the first feature for determination, falls within the second range, and whether an absolute value of a first face direction, which is the first feature for determination and which becomes 0 when the face is turned in a direction facing the traveling direction of the vehicle or when the face is turned in a direction facing the center of an optical axis of an image capturing device installed in the vehicle so as to be able to capture an image of at least the face of the occupant, is less than a first threshold value;
the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the absolute value of the first face direction is less than the first threshold value;
the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the absolute value of the first face direction is equal to or greater than the first threshold value.
The occupant condition determination device according to claim 3 . the occupant determination unit performs the occupant determination using the first determination feature amount and a third determination feature amount that is the first feature amount detected in the third captured image by the feature amount detection unit.
The occupant condition determination device according to claim 3 . the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range, and whether an absolute value of a head position difference, which is a difference between the first head position and a second head position, which is the third determination feature, is equal to or greater than a second threshold;
the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the absolute value of the head position difference is equal to or greater than the second threshold;
the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the absolute value of the head position difference is less than the second threshold value.
The occupant state determination device according to claim 6 . the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range, and whether an absolute value of a face direction difference, which is a difference between a first face direction, which is the first determination feature, and a second face direction, which is the third determination feature, is equal to or greater than a third threshold;
the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the absolute value of the facial direction difference is equal to or greater than the third threshold;
the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the absolute value of the facial direction difference is less than the third threshold value.
The occupant state determination device according to claim 6 . the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range, and whether first recognition information, which is information for recognizing the first face, matches driver recognition information, which is information for recognizing the face of the driver;
the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the first recognition information does not match the driver recognition information,
the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the first recognition information matches the driver recognition information;
The occupant condition determination device according to claim 3 . the first range includes a range in which a face of the driver seated in the driver's seat of the vehicle is located, and the second range includes a range in which a face of another occupant other than the driver seated in a seat other than the driver's seat is located.
The occupant condition determination device according to claim 1 or 2. 3. An occupant state determination system comprising: the occupant state determination device according to claim 1 or 2; an imaging device installed in the vehicle so as to be able to capture an image of at least the face of the occupant; and an output device mounted in the vehicle and configured to output a warning that the occupant is in an abnormal state,
the occupant state determination device determines whether the occupant is in the abnormal state using the captured image captured by the imaging device, and when it determines that the occupant is in the abnormal state, causes the output device to output a warning that the occupant is in the abnormal state.
Occupant status determination system. a face detection step in which a face detection unit detects the faces of the vehicle occupants in a plurality of captured images each including both a first range, which is a range in which the face of the driver of the vehicle occupants may be present, and a second range, which is a range in which the face of a person other than the driver may be present and has a range overlapping with the first range;
a first face detection determination step in which a first face detection determination unit performs a first face detection determination as to whether or not the face of the occupant has been detected within the first range in the plurality of captured images;
a second face detection determination step of performing a second face detection determination as to whether or not the face of the occupant has been detected within the first range, using a second captured image of the plurality of captured images that has been captured before the first captured image, when the second face detection determination unit acquires from the first face detection determination unit a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images;
an occupant determination step in which, when the occupant determination unit obtains a determination result from the second face detection determination unit indicating that the face of the occupant could not be detected, the occupant determination unit determines that a first face, which is the face of the occupant detected in the first captured image, is a face other than that of the driver if the first face is located within the second range of the first captured image, and determines that the first face is the face of the driver if the first face is not located within the second range of the first captured image;
an abnormal posture determination step of, when the abnormal posture determination unit acquires from the occupant determination unit a determination result indicating that the first face is a face other than that of the driver, determining whether or not the driver is in an abnormal posture by using a third captured image among the plurality of captured images that was captured before the first captured image and for which a determination result indicating that the face of the occupant was detected was output by the first face detection determination unit;
an abnormal state determination step in which an abnormal state determination unit determines whether the driver is in an abnormal state by using a determination result of the abnormal posture determination unit;
An occupant state determination method comprising: On the computer,
a face detection function in which a face detection unit detects the faces of the vehicle occupants in a plurality of captured images each including both a first range in which the face of the driver of the vehicle may be present and a second range in which the face of a person other than the driver may be present, the second range overlapping with the first range;
a first face detection determination function in which a first face detection determination unit performs a first face detection determination as to whether or not the face of the occupant has been detected within the first range in the plurality of captured images;
a second face detection determination function that, when a second face detection determination unit acquires from the first face detection determination unit a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images, performs a second face detection determination as to whether the face of the occupant has been detected in the first range using a second captured image of the plurality of captured images that has been captured before the first captured image;
an occupant determination function that, when the occupant determination unit obtains a determination result from the second face detection determination unit indicating that the face of the occupant could not be detected, determines that a first face, which is the face of the occupant detected in the first captured image, is a face other than that of the driver if the first face is located within the second range of the first captured image, and determines that the first face is the face of the driver if the first face is not located within the second range of the first captured image;
an abnormal posture determination function that, when the abnormal posture determination unit obtains a determination result from the occupant determination unit indicating that the first face is a face other than that of the driver, performs an abnormal posture determination as to whether the driver is in an abnormal posture by using a third captured image among the plurality of captured images that was captured before the first captured image and for which a determination result indicating that the face of the occupant was detected was output by the first face detection determination unit;
an abnormal state determination function in which an abnormal state determination unit determines whether the driver is in an abnormal state by using the determination result of the abnormal posture determination unit;
An occupant status determination program to achieve this.