JP7552159B2 - Drowsiness determination program, drowsiness determination device, and drowsiness determination method - Google Patents

Description

The present invention relates to a drowsiness determination program, a drowsiness determination device, and a drowsiness determination method.

Information regarding the opening and closing of a user's eyes is information that is closely related to the user's drowsiness. For example, there is known technology that determines a user's drowsiness based on features related to the opening and closing of the eyes detected from an image of the user's face.

JP 2019-111092 A International Publication No. 2011/125166

Even if the feature amounts related to eye opening and closing are the same, the degree of drowsiness may differ. Therefore, if the feature amounts detected from different face images are comparable, an attempt to determine drowsiness using the same parameters for each feature amount may result in an erroneous determination.

In one aspect, the present invention aims to improve the accuracy of determining a user's drowsiness.

In one aspect, the drowsiness determination program may cause a computer to execute the following process. The process may include acquiring a plurality of captured images arranged in time series, in which an area including a user's face is captured. The process may also include detecting a change in a feature amount related to the opening and closing of eyes included in the face based on the plurality of captured images. The process may also include determining a first weight of the feature amount related to the opening and closing of the eyes and a second weight of the feature amount related to muscle tension of the face that does not affect the opening and closing of the eyes , according to the detected change in the feature amount related to the opening and closing of the eyes. The process may also include determining the drowsiness of the user based on at least one of the feature amount related to the opening and closing of the eyes and the feature amount related to muscle tension in accordance with the determined first weight and second weight. In addition, the process of determining the first weight and the second weight may include a process of determining the first weight and the second weight such that the first weight is greater than the second weight when the amount of change in the detected feature related to eye opening and closing is equal to or less than a first threshold, and determining the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.

In one aspect, the present invention can improve the accuracy of determining a user's drowsiness.

FIG. 1 is a diagram showing an example of a New Energy and Industrial Technology Development Organization (NEDO) evaluation value. 11 is a diagram showing an example of a relationship between feature amounts relating to eye open/closed states and blinking and a NEDO evaluation value. FIG. 11A and 11B are diagrams illustrating an example of an eye open/close state and the number of blinks in a first exceptional situation. FIG. 13 is a diagram showing an example of a determination result of the degree of drowsiness in a first exceptional situation. 11A and 11B are diagrams illustrating an example of an eye open/close state and the number of blinks in a second exceptional situation. FIG. 13 is a diagram showing an example of a determination result of the degree of drowsiness in a second exceptional situation. 11A and 11B are diagrams illustrating an example of changes in feature amount related to the tension state of facial muscles. 11A and 11B are diagrams illustrating an example of changes in feature amount related to the tension state of facial muscles. 1 is a block diagram showing an example of a functional configuration of a drowsiness determination device according to an embodiment; 5 is a diagram for explaining an example of an operation of a drowsiness determination device according to an embodiment. FIG. 11 is a diagram showing an example of determining a drowsiness level using a method according to an embodiment; 11 is a diagram showing an example of determining a drowsiness level using a method according to an embodiment; 10 is a flowchart illustrating an example of an operation of a drowsiness determination process performed by a drowsiness determination device according to an embodiment. 2 is a block diagram showing an example of a hardware (HW) configuration of a computer that realizes the functions of the drowsiness determination device. FIG. 1 is a block diagram showing a configuration example of a safe driving support system according to a first application example. FIG. 11 is a diagram illustrating an example of installation of an imaging device according to a first application example. FIG. 5 is a flowchart illustrating an example of the operation of the safe driving support system according to the first application example. FIG. 13 is a block diagram showing a configuration example of a remote learning support system according to a second application example. 13 is a diagram showing an example of installation of an imaging device according to a second application example. FIG. 13 is a flowchart illustrating an example of the operation of the remote learning support system according to the second application example.

The following describes an embodiment of the present invention with reference to the drawings. However, the embodiment described below is merely an example, and is not intended to exclude the application of various modifications or techniques not specifically described below. For example, this embodiment can be modified in various ways without departing from the spirit of the invention. In the drawings used in the following description, parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[1] One embodiment [1-1] Regarding drowsiness determination First, drowsiness determination will be described. A sign of drowsiness in a user is determined based on, for example, a feature amount related to the opening and closing of the user's eyes obtained from a captured image of the user's face. An example of the feature amount related to the opening and closing of the user's eyes is the degree of eyelid opening. The degree of eyelid opening may be obtained as a value obtained by dividing the distance from the upper eyelid to the lower eyelid by the diameter of the iris. The degree of eyelid opening is an example of the degree of eye opening.

PERCLOS (PERcent of eye lid CLOSure) is one index for measuring the degree of fatigue of a user, for example, a driver of a vehicle such as an automobile. PERCLOS is an index approved by the National Highway Traffic Safety Administration, and is the percentage of time that the eyes are closed in the last minute. For example, in PERCLOS, eyelid movement with an eyelid opening rate of 20% or less and lasting for 500 milliseconds or more is determined to be a state in which the driver has his/her eyes closed. With such a determination, for example, when drowsiness determination is used in a vehicle safety driving support system, it is possible to detect the driver's signs of drowsiness and support safe driving in a timely manner.

One of the indices for assessing the degree of drowsiness is the facial drowsiness assessment scale (hereinafter, sometimes referred to as the "NEDO assessment value") by NEDO (New Energy and Industrial Technology Development Organization). Figure 1 is a diagram showing an example of the NEDO assessment value. The drowsiness assessment scale illustrated in Figure 1 can be used to assess the degree of drowsiness based on eye opening and closing and drowsiness.

Furthermore, there is a method for estimating the level of the drowsiness assessment scale (drowsiness value, degree of drowsiness) in the NEDO evaluation value based on features related to the opening and closing of the eyes, for example, features related to the opening and closing state of the eyes and blinking. Examples of features related to the opening and closing state of the eyes and blinking include one or more parameters such as the degree of opening when the eyes are open, the average blinking time, the distribution of the continuous closing time of the eyes, the percentage of the eyes closed, and the number of blinks. The degree of opening of the eyes may be, for example, the value obtained by dividing the distance from the upper eyelid to the lower eyelid by the maximum distance from the upper eyelid to the lower eyelid, or the above-mentioned degree of opening of the eyes. For example, the above method may be a method described in "Development of Drowsiness State Estimation and Driver Status Monitor Using Image Sensors," "DENSO TECHNICAL REVIEW Vol. 21 2016," Denso (registered trademark)" or the like.

In the above method, when information on features related to eye opening/closing and blinking is used, the reliability of drowsiness detection may be insufficient. For example, in the wakefulness state and the strong drowsiness state, which are examples of the degree of drowsiness, exceptional situations may occur, such as a situation in which the features related to eye opening/closing and blinking are reversed, and the user's degree of drowsiness may be misclassified.

Figure 2 is a diagram showing an example of the relationship between the feature quantities related to eye opening/closing and blinking and the NEDO evaluation value. Figure 3 is a diagram showing an example of the eye opening/closing state and the number of blinks in a first exceptional situation, and Figure 4 is a diagram showing an example of the determination result of the degree of drowsiness in the first exceptional situation. Figure 5 is a diagram showing an example of the eye opening/closing state and the number of blinks in a second exceptional situation, and Figure 6 is a diagram showing an example of the determination result of the degree of drowsiness in the second exceptional situation.

(First Exception)
The area indicated by the symbol A in FIG. 2, as well as FIGS. 3 and 4, show cases where an alert state is erroneously determined as a drowsy state.

As shown in FIG. 2, when a user is in an awake state and the NEDO evaluation value is "1" (see FIG. 1), the degree of eye opening is generally "wide" and the number of blinks is "many," which is a typical feature of the definition of an awake state.

However, there is an exception to this, and when the user is awake and concentrating, the degree of eye opening may be "small" and the number of blinks may be "low" (see the area indicated by symbol A and Figure 3).

For example, as shown in Figure 4, when a user is concentrating, the opening and closing of the eyes decreases, which may result in the user being misclassified or misdetected as being in a drowsy state (medium to high degree of drowsiness) even though the correct answer is that the user is actually in an awake state.

(Second Exception)
The area indicated by the symbol B in FIG. 2, as well as FIGS. 5 and 6, show cases where a drowsy state is erroneously determined as an alert state.

As shown in FIG. 2, when a user is in a drowsy state (medium to high degree of drowsiness) with a NEDO evaluation value of "4, 5" (see FIG. 1), the following features are generally defined as the drowsy state. These features are: degree of eye opening "small", average blink duration "long", distribution of continuous eye closure duration "large average, small variance", percentage of eyes closed "medium to large", and number of blinks "medium to small".

However, as an exception, if a user intentionally (intentionally) blinks to combat drowsiness while in a drowsy state, the following features may result (see the area indicated by symbol B and Figure 5). In this case, the features are: degree of eye opening "large", average blink duration "short", distribution of continuous eye closure duration "small average, small variance", percentage of eye closure "small", and number of blinks "many".

For example, as shown in Figure 6, if the opening and closing of the eyes when a user is fighting drowsiness matches the general pattern of an awakened state, the user may be misclassified or misdetected as being in an awakened state even though the correct answer is that the user is in a drowsy state (medium to high degree of drowsiness).

Note that the expressions showing the degree such as "small", "few", "short", "medium", "large", "many", and "long" shown in FIG. 2 may be set by a relative comparison between the features, or may be set by a comparison with a threshold value or range corresponding to each degree.

As such, even if the feature amounts related to eye opening and closing are the same, the degree of drowsiness may differ. Therefore, if the feature amounts detected from different face images are comparable, an attempt to determine drowsiness using the same parameters for each feature amount may result in an erroneous determination.

Therefore, in one embodiment, a method for improving the accuracy of determining a user's drowsiness, for example, a method for accurately determining the user's level of drowsiness even when the first and second exceptions occur, is described.

[1-2] Regarding One Embodiment For example, a drowsiness determination device according to one embodiment executes the following processes (a) to (d).
(a) A plurality of captured images including a user's face are acquired.
(b) A change in a feature amount relating to opening and closing of the eyes included in the face is detected based on a plurality of captured images.
(c) determining a first weight for the feature amount relating to eye opening/closing and a second weight for the feature amount relating to facial muscle tension according to a change in the detected feature amount relating to eye opening/closing;
(d) In accordance with the determined first weight and second weight, the drowsiness of the user is determined based on at least one of the feature amount relating to eye opening/closing and the feature amount relating to facial muscle tension.

For example, in a state including the first and second exceptions, the drowsiness determination device can accurately determine the user's level of drowsiness by determining the level of drowsiness using features related to facial muscle tension, thereby improving the accuracy of drowsiness detection.

Examples of facial muscles include the superficial head muscles (facial muscles) that experience small changes in tension when opening and closing the eyes, including blinking, in other words, muscles that do not affect the opening and closing of the eyes. Examples of such muscles include the muscles of the mouth or nose, such as the orbicularis oris, mentalis, and levator labii superioris alae naris.

The feature amount related to facial muscle tension (tension state) may be based on, for example, the AU (Action Unit) value classified by FACS (Facial Action Coding System). The AU value may be obtained, for example, by image analysis processing of the captured image.

Feature amounts relating to the state of tension in facial muscles may be calculated, for example, based on information relating to changes in facial muscles, such as the amount of change in AU value between multiple captured images (e.g., between captured images in a time series).

Figures 7 and 8 are diagrams showing an example of changes in feature quantities related to the tension state of facial muscles. Figure 7 shows an example in which the range of change in the value of the orbicularis oris muscle defined in AU23 is used as the feature quantities related to the tension state of facial muscles. Figure 8 shows an example in which the range of change in the value of the mentalis muscle defined in AU17, the range of change in the value of the orbicularis oris muscle, the number of peaks in the change in the value of the mentalis muscle, and the range of change in the value of the levator labii superioris alae naris defined in AU9 are used as the feature quantities related to the tension state of facial muscles.

As shown in the top table of Figure 7 and Figure 8, when we focus on the range of change in the orbicularis oris muscle, we can see that the features differ greatly between a wakefulness state, which includes both general and exceptional (concentration) states, and a state of strong drowsiness, which includes both general and exceptional (conscious blink) states.

For example, the range of change in the orbicularis oris muscle, which is generally "indefinite" in an awake state, becomes "large" when one is concentrating. On the other hand, when one is very sleepy, the range of change in the orbicularis oris muscle becomes "small" in both the general state and when one consciously blinks.

In this way, the feature changes significantly between a state of concentration and a general state of strong sleepiness, whereas the feature changes almost unchanged between a general state of strong sleepiness and a state in which conscious blinking has been performed. Such characteristics of changes in feature values relating to the tension state of facial muscles are often commonly obtained for the muscles of the mouth or nose, in other words, the superficial head muscles (facial muscles) that show small changes in tension state due to opening and closing of the eyes, including blinking. For example, as shown in Figure 8, when the change in feature value is the range of change of the mentalis muscle, the number of peaks of change of the orbicularis oris muscle, and the range of change of the levator labii superioris alae naris muscle, a tendency similar to that of the range of change of the orbicularis oris muscle is obtained.

Note that the expressions showing the degree such as "small", "few", "medium", "large", and "many" shown in Figures 7 and 8 may be set by a relative comparison between the features, or may be set by comparison with a threshold value (range) corresponding to each degree.

Here, as shown by symbol A in FIG. 8, the reason why the orbicularis oris muscle is generally "inconstant" in an awake state is that it is a muscle that is easily changed (moved) by, for example, movements other than eye opening and closing and blinking, as well as by emotions (e.g., "dislike"). For this reason, when estimating the degree of drowsiness using features related to the tension state of facial muscles together with features related to eye opening and closing and blinking, there is a possibility that the features related to the tension state of facial muscles may fluctuate even if the level of drowsiness is constant, which may reduce the accuracy of the assessment result.

As shown in the lower graph of FIG. 7 and symbol B of FIG. 8, for example, when a user is performing a predetermined task such as driving, the degree of drowsiness can be estimated by limiting the opportunities for using the facial muscle tension feature to a state of concentration in an awake state and a state of drowsiness. For example, by using the facial muscle tension feature only to a state of concentration and a state of drowsiness according to the timing of changes in the degree of drowsiness estimated from the features related to eye opening and closing and blinking, it is possible to avoid false detections including the first and second exceptional cases described above.

In one embodiment, the drowsiness determination device uses features related to the state of tension in the facial muscles, but this is not limited to this, and features related to the state of tension in muscles in other parts of the user other than the face may also be used.

[1-3] Example of functional configuration FIG. 9 is a block diagram showing an example of the functional configuration of a drowsiness determination device 1 according to one embodiment, and FIG. 10 is a diagram for explaining an example of the operation of the drowsiness determination device 1 according to one embodiment.

The drowsiness determination device 1 executes a drowsiness determination process that determines the drowsiness of a user based on a plurality of captured images. As shown in FIG. 9, the drowsiness determination device 1 may, for example, include an imaging unit 2, a memory unit 3, and an estimation unit 4.

The imaging unit 2 acquires multiple captured images of the user's face. For example, the imaging unit 2 captures multiple captured images, such as moving images, including the user's face through control of an imaging device such as a camera, and records the multiple captured images 31 in the memory unit 3. The imaging device may be provided outside the drowsiness determination device 1, for example, in the vicinity of the user. In this case, the imaging unit 2 may control the imaging device and acquire captured images via a network (not shown).

The memory unit 3 is an example of a storage area, and stores various data used by the drowsiness determination device 1. As shown in FIG. 9, the memory unit 3 may be capable of storing, for example, a plurality of captured images 31, a plurality of feature amounts 32, and an output result 33.

The estimation unit 4 is an example of a control unit, and determines the user's drowsiness, for example, estimating the degree of drowsiness (impression estimation). For example, the estimation unit 4 may perform image analysis processing on each of the multiple captured images 31 (e.g., frames of a moving image) stored in the memory unit 3, and detect the user's face (face area) and AU from each of the multiple captured images 31 (see process P1 in Figure 10).

The estimation unit 4 may detect the AU from the captured image 31 using various known methods. For example, the estimation unit 4 may detect the AU using the method described in "Facial Non-Linear Kernel Approximation, Proceedings of the IEEE international Conference on Computer Vision Workshops, 2015."

As illustrated in FIG. 9, the estimation unit 4 may include an open/closed state blink estimation unit 41, a drowsiness level change estimation unit 42, a tension state estimation unit 43, a feature amount determination unit 44, and a drowsiness estimation unit 45.

The open/closed blink estimation unit 41 estimates features related to the open/closed state of the eyes included in the user's face, for example features related to the open/closed state of the eyes and blinking, based on the area of the user's face portion detected from each of the multiple captured images 31 (see process P2 in FIG. 10). The open/closed blink estimation unit 41 may store the acquired features as features 32 in the memory unit 3. In the following description, the "feature related to the open/closed state of the eyes" (feature 32) estimated by the open/closed blink estimation unit 41 may be referred to as the "first feature" (first feature 32a).

For example, the open/closed state blink estimation unit 41 detects eye feature points from the area of the user's face portion detected from each of the multiple captured images 31. The eye feature points may include, for example, at least one of the feature points of the eyeball (cornea or iris, etc.), upper eyelid, lower eyelid, etc.

Then, the open/closed state blink estimation unit 41 calculates the first feature amount 32a by image analysis processing based on the feature points. As the image analysis processing, various known methods may be used. For example, the image analysis processing may include image recognition using a trained model in machine learning, for example, deep learning.

In one embodiment, the first feature amount 32a may be any one or a combination of two or more of the following: eyelid opening degree, eye opening degree when the eyes are open, average blink duration, distribution of continuous eye closure duration, eye closure rate, and blink count. The open/closed state blink estimation unit 41 may acquire these feature amounts based on various known methods, for example, "Development of Drowsiness State Estimation and Driver Status Monitor Using Image Sensors," "DENSO TECHNICAL REVIEW Vol. 21 2016," DENSO (registered trademark).

The drowsiness level change estimation unit 42 estimates a change in the user's drowsiness level based on the first feature amount 32a acquired by the open/closed state blink estimation unit 41. For example, the drowsiness level change estimation unit 42 calculates the amount of change in the first feature amount 32a between the captured images 31 arranged in time series (see process P3 in FIG. 10). As an example, the drowsiness level change estimation unit 42 may calculate the amount of change in the first feature amount 32a between the target captured image 31 and the most recently captured image 31 (the image immediately preceding it in time series).

The drowsiness level change estimation unit 42 determines (detects) whether or not there is a change in the user's drowsiness level by comparing the calculated amount of change with the first threshold value Th1 (see process P4 in FIG. 10). For example, the drowsiness level change estimation unit 42 determines whether or not the calculated amount of change is greater than the first threshold value Th1.

The first threshold Th1 may be, for example, a threshold capable of detecting a change in the first characteristic amount 32a from "small (few, short)" to "large (many, long)" or from "large (many, long)" to "small (few, short)" in the table shown in FIG. 2 (in other words, a change in the degree of drowsiness). As an example, the first threshold Th1 may be the difference between the "threshold" or "median of the range" at which the first characteristic amount 32a is judged to be "large (many, long)" and the "threshold" or "median of the range" at which the first characteristic amount 32a is judged to be "small (few, short)", or may be a value calculated by various other methods.

As shown in FIG. 2, in an awake state and a drowsy state (medium to high level of drowsiness), when the level of drowsiness changes between a typical state and an exceptional state, the amount of change in the first characteristic amount 32a is greater than the first threshold value Th1, except in some cases (see the areas indicated by symbols A and B in FIG. 2). In other words, when the amount of change in the first characteristic amount 32a is greater than the first threshold value Th1, there is a high possibility that the level of drowsiness has changed between "typical" and "exceptional."

On the other hand, when the amount of change in the first characteristic amount 32a is equal to or less than the first threshold value Th1, as shown in FIG. 2, in the wakefulness state and the drowsiness state (level of drowsiness (medium to high)), it is unlikely that the level of drowsiness has changed between "general" and "exceptional."

Therefore, the drowsiness level change estimation unit 42 may determine that there is a change in the user's drowsiness level when the change in the first characteristic amount 32a is greater than the first threshold value Th1, and may determine that there is no change in the user's drowsiness level when the change in the first characteristic amount 32a is equal to or less than the first threshold value Th1.

The tension state estimation unit 43 calculates a feature amount related to the tension state of the facial muscles based on the AU value of the face detected from each of the multiple captured images 31 (see process P5 in FIG. 10). The tension state estimation unit 43 may store the acquired feature amount as feature amount 32 in the memory unit 3. Note that in the following description, the "feature amount related to the tension state of the facial muscles" (feature amount 32) estimated by the tension state estimation unit 43 may be referred to as a "second feature amount" (second feature amount 32b).

For example, the tension state estimation unit 43 may calculate, as the second feature 32b, any one or a combination of two or more of the change range and number of peaks of the mentalis muscle (AU17), the change range and number of peaks of the orbicularis oris muscle (AU23), and the change range of the levator labii superioris alae naris muscle (AU9).

When calculating the range of change in AU as the second feature 32b, the tension state estimation unit 43 may, for example, record the maximum and minimum AU values in multiple captured images 31 acquired during a specified calculation period, and obtain the difference between the maximum and minimum values as the range of change.

When calculating the number of AU peaks as the second feature 32b, the tension state estimation unit 43 may, for example, record the number of times that the AU value reaches a peak (e.g., a maximum) in multiple captured images 31 acquired during a specified calculation period, and acquire this number as the number of peaks.

The predetermined calculation period may be, for example, several seconds to several tens of seconds, for example, approximately 10 seconds. The tension state estimation unit 43 may calculate the second feature amount 32b every 10 seconds, for example, based on the AU value acquired in the most recent 10 seconds. Note that the drowsiness estimation process by the drowsiness determination device 1 may be performed every predetermined calculation period described above.

In addition, the tension state estimation unit 43 may, for example, limit the calculated second feature 32b to one or more feature values in the following group (A) or (B) depending on the degree of drowsiness estimated based on the first feature value 32a.
(A) When the level of drowsiness is low to medium during the drowsy state.
- Range of change and number of peaks in the mentalis muscle (AU17) - Range of change and number of peaks in the orbicularis oris muscle (AU23) - Range of change in the levator labii superioris alae naris muscle (AU9) (B) When the degree of sleepiness is medium to high in a drowsy state.
・Number of peaks in the orbicularis oris muscle (AU23) ・Range of change in the levator labii superioris alae nasalis muscle (AU9)

The feature amount determination unit 44 selects the feature amount 32 used to estimate the user's drowsiness (see process P6 in FIG. 10). For example, the feature amount determination unit 44 may determine a first weight for the first feature amount 32a and a second weight for the second feature amount 32b depending on whether or not there is a change in the level of drowsiness estimated by the drowsiness level change estimation unit 42. For example, the feature amount determination unit 44 may determine the first and second weights in accordance with the following (i) and (ii).

(i) When there is no change in the level of drowsiness (when the change in the first characteristic amount 32a is equal to or smaller than the first threshold value Th1).
When the drowsiness level change estimation unit 42 determines that there is no change in the level of drowsiness, the feature determination unit 44 may determine that there is a low possibility that the state corresponds to the first and second exceptions described above (that the state does not correspond to the first and second exceptions), and may perform weighting so that the first weight is greater than the second weight.

When the possibility of the state corresponding to the first and second exceptions is low, the user's drowsiness can be determined based on the first feature 32a, as described with reference to FIG. 2. Therefore, the feature determination unit 44 may determine the ratio (proportion) of the first and second weights within a range such as first weight:second weight = "0.5 to 1.0": "0.5 to 0.0" (as an example, "1.0": "0.0") so that the first weight is greater than the second weight.

(ii) When there is a change in the level of drowsiness (when the amount of change in the first characteristic amount 32a is greater than the first threshold value Th1).
When the drowsiness level change estimation unit 42 determines that there is a change in the level of drowsiness, the feature determination unit 44 may determine that there is a high possibility that the state corresponds to the first and second exceptions described above (that the state corresponds to the first and second exceptions), and may perform weighting so that the first weight is less than the second weight.

When the user is in a state corresponding to the first and second exceptions, the user's drowsiness can be determined based on the second feature 32b, as described with reference to Figures 7 and 8. Therefore, the feature determination unit 44 may determine the ratio (proportion) of the first and second weights within a range such as first weight:second weight = "0.5 to 0.0": "0.5 to 1.0" (as an example, "0.0": "1.0") so that the second weight is greater than the first weight.

The drowsiness estimation unit 45 determines the user's drowsiness based on at least one of the first feature amount 32a and the second feature amount 32b in accordance with the first and second weights determined by the feature amount determination unit 44 (see process P7 in Figure 10).

The drowsiness estimation unit 45 may output the drowsiness judgment result. Examples of output of the judgment result include displaying it on an output device such as a monitor, storing it in the memory unit 3 or a storage device, transmitting it to the outside via a communication interface (IF), and the like. For example, the drowsiness estimation unit 45 may store the judgment result in the memory unit 3 as the output result 33.

As an example of drowsiness determination by the drowsiness estimation unit 45, a case where the first and second weights are "1.0":"0.0" or "0.0":"1.0", in other words, a case where only the first feature amount 32a (see (i) above) or only the second feature amount 32b (see (ii) above) of the first feature amount 32a and the second feature amount 32b is used to determine drowsiness is assumed.

(I) A case where the first and second weights are "1.0":"0.0" (a case where only the first feature amount 32a is used to determine drowsiness).
In this case, the second feature amount 32b having the weight of “0.0” is not used for determining drowsiness. For example, the drowsiness estimation unit 45 may determine (estimate) the level of drowsiness of the user by comparing the first feature amount 32a with the second threshold value Th2.

The second threshold Th2 may be, for example, a value between "small (few, short)" and "large (many, long)" of the first characteristic amount 32a in the table shown in FIG. 2. As an example, the second threshold Th2 may be a "threshold" or "median of a range" at which the first characteristic amount 32a is determined to be "medium," or may be a value determined by various other methods.

In the following, it is assumed that the first feature amount 32a is the degree of eye opening. When the first feature amount 32a is greater than the second threshold value Th2, the drowsiness estimation unit 45 may determine that the user's degree of drowsiness is an awake state (see "Awake state: general" in which "degree of eye opening" is "large" in FIG. 2). On the other hand, when the first feature amount 32a is equal to or less than the second threshold value Th2, the drowsiness estimation unit 45 may determine that the user's degree of drowsiness is a drowsy state (degree of drowsiness: medium to high) (see "degree of drowsiness (medium to high): general" in which "degree of eye opening" is "small" in FIG. 2).

Note that the value of the second threshold Th2 and the direction of the inequality sign when comparing the first feature 32a with the second threshold Th2 may differ depending on the type of feature used as the first feature 32a (see FIG. 2).

In addition, the drowsiness estimation unit 45 may determine the degree of drowsiness from three or more levels (e.g., "awake state," "low drowsiness level," "medium to high drowsiness level," etc.) by comparing the first feature 32a with multiple second threshold values Th2 that are different from each other.

(II) A case where the first and second weights are "0.0":"1.0" (a case where only the second feature amount 32b is used to determine drowsiness).
In this case, the first feature amount 32a having the weight of “0.0” is not used for determining drowsiness. For example, the drowsiness estimation unit 45 may determine (estimate) the level of drowsiness of the user by comparing the second feature amount 32b with a third threshold value Th3.

The third threshold Th3 may be, for example, a value between "small" and "large" of the second characteristic amount 32b in the table shown in FIG. 8. As an example, the third threshold Th3 may be a "threshold" or "median of a range" at which the second characteristic amount 32b is determined to be "medium," or may be a value determined by various other methods.

In the following, it is assumed that the second feature 32b is the range of change in the orbicularis oris muscle (AU23). When the second feature 32b is greater than the third threshold Th3, the drowsiness estimation unit 45 may determine that the user's level of drowsiness is an awake state (see "Awake state: Exception (concentration)" in FIG. 8 where the "range of change in the orbicularis oris muscle" is "large").

As a result, when the awake state changes from "general" to "exceptional (concentration)" as shown in the upper part of FIG. 11, the drowsiness determination device 1 can determine drowsiness using an appropriate feature amount 32 at the timing of the change in the degree of drowsiness. For example, as shown in the lower part of FIG. 11, the drowsiness determination device 1 determines drowsiness based on the second feature amount 32b related to the tension state of the facial muscles instead of the first feature amount 32a related to the opening and closing of the eyes at the timing of the change from "general" to "exceptional (concentration)". In this way, the drowsiness determination device 1 can avoid erroneously detecting the concentration state in the awake state as a drowsy state in the first exception shown in FIG. 4.

On the other hand, when the second feature amount 32b is equal to or less than the third threshold value Th3, the drowsiness estimation unit 45 may determine that the user's drowsiness level is a drowsy state (drowsiness level: medium to high) (see "Drowsiness level (medium to high): exception (conscious blinking)" in which the "range of change in the orbicularis oris muscle" in Figure 2 is "small").

As a result, when the drowsiness state (level of drowsiness: medium to high) changes from "general" to "exception (conscious blink)" as shown in the upper part of FIG. 12, the drowsiness determination device 1 can determine drowsiness using an appropriate feature amount 32 at the timing of the change in the level of drowsiness. For example, as shown in the lower part of FIG. 12, the drowsiness determination device 1 determines drowsiness based on the second feature amount 32b related to the tension state of the facial muscles instead of the first feature amount 32a related to the opening and closing of the eyes at the timing of the change from "general" to "exception (conscious blink)". In this way, the drowsiness determination device 1 can avoid erroneously detecting a conscious blink in a drowsiness state (level of drowsiness: medium to high) as an awakening state in the second exception shown in FIG. 6.

Note that, although an example has been described in which the second feature amount 32b is the range of change of the orbicularis oris muscle (AU23), this is not limiting. The drowsiness estimation unit 45 may make the following determination depending on the type of feature amount used as the second feature amount 32b.

Type: When the change range is the mentalis muscle (AU17).
In this case, the drowsiness estimation unit 45 may determine the degree of drowsiness as an awake state when the second feature 32b is greater than the third threshold Th3, and may determine the degree of drowsiness as a drowsy state when the second feature 32b is equal to or less than the third threshold Th3.

Type: Peak number of the orbicularis oris muscle (AU23).
In this case, the drowsiness estimation unit 45 may determine the degree of drowsiness as an awake state when the second feature 32b is greater than (has a large value for) the third threshold Th3, and may determine the degree of drowsiness as a drowsy state when the second feature 32b is less than or equal to the third threshold Th3.

- Type: When the change is within the range of the levator labii superioris alae nasii (AU9).
In this case, the drowsiness estimation unit 45 may determine the degree of drowsiness as an awake state when the second feature 32b is greater than the third threshold Th3, and may determine the degree of drowsiness as a drowsy state when the second feature 32b is equal to or less than the third threshold Th3.

Note that the value of the third threshold Th3 and the direction of the inequality sign when comparing the second feature 32b with the third threshold Th3 may differ depending on the type of feature used as the second feature 32b (see FIG. 8).

Next, we will explain the case where the first and second weights are greater than "0.0" and less than "1.0", in other words, where both the first feature amount 32a and the second feature amount 32b are used to determine drowsiness.

The drowsiness estimation unit 45 may estimate drowsiness by applying a first weight to the first feature 32a relating to eye opening and closing, and applying a second weight to the second feature 32b relating to facial muscle tension.

As an example, the drowsiness estimation unit 45 may perform both a determination of the degree of drowsiness based on the first feature amount 32a and a determination of the degree of drowsiness based on the second feature amount 32b. Then, the drowsiness estimation unit 45 may output the results of applying the first weight and the second weight to the degree determined based on each feature amount 32 as the output result 33.

For example, the drowsiness estimation unit 45 may add a reliability of the judgment result based on a first weight to the judgment result of the degree of drowsiness based on the first feature amount 32a, and add a reliability of the judgment result based on a second weight to the judgment result of the degree of drowsiness based on the second feature amount 32b. This makes it possible to comprehensively judge drowsiness based on each of the judgment results of the first feature amount 32a and the second feature amount 32b and their respective reliability in a system that uses the judgment result (output result 33). Note that the drowsiness estimation unit 45 may make a comprehensive judgment using the reliability and output the judgment result.

As described above, according to one embodiment of the drowsiness determination device 1, the user's level of drowsiness is estimated based on one or both of the features related to eye opening and closing and the features related to the tension state of facial muscles in response to changes in the level of drowsiness detected from the features related to eye opening and closing.

This makes it possible to properly detect the level of drowsiness even in situations where the level of drowsiness cannot be detected correctly using features related to eye opening and closing, such as when the user is concentrating (first exception) or when the user opens and closes their eyes while consciously fighting drowsiness (second exception).

[1-4] Operation Example Hereinafter, a description will be given of an operation example of the drowsiness determination process performed by the drowsiness determination device 1. Fig. 13 is a flowchart illustrating an operation example of the drowsiness determination process performed by the drowsiness determination device 1 according to one embodiment.

The estimation unit 4 detects the user's face (face area) and AU from each of the multiple captured images 31 stored in the memory unit 3. For example, as shown in FIG. 13, the estimation unit 4 calculates the AU value from each of the multiple captured images 31 (step S1).

As illustrated in FIG. 13, in the drowsiness assessment device 1, the tension state estimation unit 43 calculates the range of change in the second feature amount 32b, for example, the orbicularis oris muscle (AU23) (step S2).

In parallel with steps S1 and S2, the open/closed state blink estimation unit 41 detects eye feature points from the area of the user's face detected by the estimation unit 4 (step S3).

The open/closed state blink estimation unit 41 calculates the first feature amount 32a, for example the degree of eye opening, based on the eye feature points (step S4).

The drowsiness level change estimation unit 42 calculates the amount of change in the first feature 32a (e.g., eye opening degree) between the captured images 31 arranged in time series (step S5).

The drowsiness level change estimation unit 42 compares the calculated amount of change with the first threshold Th1 and determines whether the calculated amount of change is greater than the first threshold Th1 (step S6).

When the amount of change in the first feature 32a is equal to or less than the first threshold Th1 (NO in step S6), the drowsiness level change estimation unit 42 determines that there is no change in the user's level of drowsiness. In this case, since it is unlikely that the state corresponds to the first and second exceptions described above, the feature determination unit 44 may perform weighting so that the first weight is greater than the second weight. Hereinafter, for simplicity, it is assumed that the feature determination unit 44 determines the first weight:second weight = "1.0":"0.0".

The drowsiness estimation unit 45 determines the user's drowsiness based on the first feature 32a, based on the weighting (first weight: second weight = "1.0": "0.0") set by the feature determination unit 44. For example, the drowsiness estimation unit 45 compares the first feature 32a (e.g., eye opening degree) with the second threshold Th2, and determines whether the first feature 32a is greater than the second threshold Th2 (step S7).

If the first characteristic amount 32a is greater than the second threshold value Th2 (YES in step S7), the drowsiness estimation unit 45 determines that the user's level of drowsiness is an awake state (step S8), outputs the determination result, and the process ends.

If the first feature amount 32a is equal to or less than the second threshold value Th2 (NO in step S7), the drowsiness estimation unit 45 determines that the user's level of drowsiness is in a drowsy state (step S9), outputs the determination result, and the process ends.

On the other hand, if the change in the first feature 32a is greater than the first threshold Th1 (YES in step S6), the drowsiness level change estimation unit 42 determines that there has been a change in the user's drowsiness level. In this case, since there is a high possibility that the state corresponds to the first and second exceptions described above, the feature determination unit 44 may perform weighting so that the first weight is smaller than the second weight. Hereinafter, for simplicity, it is assumed that the feature determination unit 44 determines the first weight:second weight = "0.0":"1.0".

The drowsiness estimation unit 45 determines the user's drowsiness based on the second feature amount 32b calculated in step S2, based on the weighting (first weight: second weight = "0.0": "1.0") set by the feature amount determination unit 44. For example, the drowsiness estimation unit 45 compares the second feature amount 32b (e.g., the range of change in the orbicularis oris muscle) with a third threshold value Th3, and determines whether the second feature amount 32b is greater than the third threshold value Th3 (step S10).

If the second feature 32b is greater than the third threshold Th3 (YES in step S10), the drowsiness estimation unit 45 determines that the user's level of drowsiness is an awake state (step S11), outputs the determination result, and the process ends.

If the second feature 32b is equal to or less than the third threshold Th3 (NO in step S10), the drowsiness estimation unit 45 determines that the user's level of drowsiness is in a drowsy state (step S12), outputs the determination result, and the process ends.

The drowsiness estimation unit 45 may store the processing results of steps S8, S9, S11, and S12 in the memory unit 3 as output results 33.

[1-5] Hardware Configuration Example The drowsiness determination device 1 according to an embodiment may be a virtual server (VM) or a physical server. Furthermore, the functions of the drowsiness determination device 1 may be realized by one computer or two or more computers. Furthermore, at least a part of the functions of the drowsiness determination device 1 may be realized using HW (Hardware) resources and NW (Network) resources provided by a cloud environment.

FIG. 14 is a block diagram showing an example of the hardware (HW) configuration of a computer 10 that realizes the functions of the drowsiness determination device 1. When multiple computers are used as HW resources that realize the functions of the drowsiness determination device 1, each computer may have the HW configuration shown in FIG. 14.

As shown in FIG. 14, the computer 10 may, as a HW configuration, illustratively include a processor 10a, a memory 10b, a storage unit 10c, an IF (Interface) unit 10d, an IO (Input/Output) unit 10e, and a reading unit 10f.

Processor 10a is an example of a processing unit that performs various controls and calculations. Processor 10a may be connected to each block in computer 10 via bus 10i so that they can communicate with each other. Processor 10a may be a multiprocessor including multiple processors, a multicore processor having multiple processor cores, or a configuration having multiple multicore processors.

The processor 10a may be, for example, an integrated circuit (IC) such as a CPU, MPU, GPU, APU, DSP, ASIC, or FPGA. Note that a combination of two or more of these integrated circuits may be used as the processor 10a. CPU is an abbreviation for Central Processing Unit, and MPU is an abbreviation for Micro Processing Unit. GPU is an abbreviation for Graphics Processing Unit, and APU is an abbreviation for Accelerated Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

Memory 10b is an example of HW that stores various data, programs, and other information. Examples of memory 10b include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

The storage unit 10c is an example of HW that stores various data, programs, and other information. Examples of the storage unit 10c include various storage devices such as magnetic disk devices such as HDDs (Hard Disk Drives), semiconductor drive devices such as SSDs (Solid State Drives), and non-volatile memories. Examples of non-volatile memories include flash memories, SCMs (Storage Class Memory), and ROMs (Read Only Memory).

The memory unit 3 shown in FIG. 9 may be realized by at least one of the storage areas of the memory 10b and the storage unit 10c. In other words, the captured image 31, the feature amount 32, and the output result 33 may be stored in at least one of the storage areas of the memory 10b and the storage unit 10c.

The memory unit 10c may also store a program 10g (drowsiness determination program) that realizes all or part of the various functions of the computer 10.

For example, the processor 10a of the drowsiness determination device 1 can realize the functions of the drowsiness determination device 1 (e.g., the imaging unit 2 and the estimation unit 4) illustrated in FIG. 9 by expanding the program 10g stored in the storage unit 10c into the memory 10b and executing it.

The IF unit 10d is an example of a communication IF that controls connection and communication with a network. For example, the IF unit 10d may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods. For example, the drowsiness determination device 1 may be connected to an information processing device and an imaging device (not shown) via the IF unit 10d so that they can communicate with each other. For example, the imaging unit 2 shown in FIG. 9 may acquire an image 31 from the imaging device via a network. In addition, the program 10g may be downloaded from the network to the computer 10 via the communication IF and stored in the storage unit 10c.

The IO unit 10e may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, a touch panel, etc. Examples of the output device include a monitor, a projector, a printer, etc. For example, the estimation unit 4 shown in FIG. 9 may output the determination result to the output device of the IO unit 10e and display it.

The reading unit 10f is an example of a reader that reads data and program information recorded on the recording medium 10h. The reading unit 10f may include a connection terminal or device to which the recording medium 10h can be connected or inserted. Examples of the reading unit 10f include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. The recording medium 10h may store a program 10g, and the reading unit 10f may read the program 10g from the recording medium 10h and store it in the memory unit 10c.

Examples of the recording medium 10h include non-transitory computer-readable recording media such as magnetic/optical disks and flash memories. Examples of magnetic/optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

The HW configuration of the computer 10 described above is an example. Therefore, the HW in the computer 10 may be increased or decreased (e.g., adding or deleting any block), divided, or integrated in any combination, or buses may be added or deleted, as appropriate. For example, in the drowsiness determination device 1, at least one of the IO unit 10e and the reading unit 10f may be omitted.

(1-6) Application Examples Next, application examples of the drowsiness determination device 1 according to an embodiment will be described.

15 is a block diagram showing a configuration example of a safe driving support system 11 according to application example 1. The safe driving support system 11 judges the drowsiness of a driver of a vehicle, which is an example of a user, based on a first feature amount 32a relating to the opening and closing of the eyes of the driver, and a second feature amount 32b relating to the tension state of the facial muscles, and utilizes the judgment result for supporting safe driving.

As shown in FIG. 15, the safe driving support system 11 may, for example, include a determination device 12, a safe driving support unit 17, and a data storage unit 18.

The determination device 12 is a device that determines the drowsiness of a user based on an image 31 including the face of the user who is driving the vehicle, and is an example of the drowsiness determination device 1 shown in FIG. 9. As shown in FIG. 15, the determination device 12 may, for example, include an image capture device 13, a communication unit 14, a memory unit 15, and a control unit 16.

Figure 16 is a diagram showing an example of installation of the imaging device 13. The imaging device 13 may be any of various cameras that capture images, such as moving images (video), to capture multiple captured images 31. As shown in Figure 16, the camera may be an in-vehicle camera that captures an area including the user's face, a smartphone, or a tablet. The imaging device 13 may be provided outside the determination device 12, and in this case may include a communication IF that transmits the captured images 31 to the determination device 12.

The communication unit 14 communicates with each of the safe driving support unit 17 and the data storage unit 18, and may be realized by the IF unit 10d shown in FIG. 14. In addition, when the imaging device 13 is provided outside the determination device 12, the communication unit 14 may communicate with the imaging device 13, for example, transmit control commands related to imaging to the imaging device 13, and receive captured images 31 transmitted from the imaging device 13.

The memory unit 15 stores various information related to the operation of the determination device 12. The memory unit 15 is an example of the memory unit 3 shown in FIG. 9, and may store the same information as the memory unit 3.

The control unit 16 performs various controls related to the operation of the determination device 12. For example, the control unit 16 may include a transmission/reception unit 16a, a drowsiness determination unit 16b, and an information control unit 16c.

The transmitter/receiver 16a controls data transmission and reception with the imaging device 13, and data transmission and reception between the imaging device 13, the safe driving support unit 17, and the data storage unit 18 via the communication unit 14.

The drowsiness determination unit 16b is an example of the estimation unit 4 shown in FIG. 9, and may perform a drowsiness determination process similar to that of the estimation unit 4.

The information control unit 16c performs various controls regarding the data exchanged between the safe driving support unit 17 and the data storage unit 18.

The safe driving support unit 17 receives the judgment result judged by the judgment device 12 from the communication unit 14, and performs safe driving support based on the judgment result. For example, when the judgment result indicates a drowsy state, the safe driving support unit 17 may output a warning such as an alarm to the driver. The alarm may be, for example, any one of sound, vibration, and smell, or a combination of two or more of these. The alarm may be output by notifying the driver using a function such as a speaker or vibrator provided in the vehicle or a smartphone owned by the driver.

The data storage unit 18 may be a database (DB) that receives the judgment results determined by the judgment device 12 from the communication unit 14 and stores the received judgment results. The information stored in the data storage unit 18 may be used for various purposes, such as automobile insurance evaluation and accident prevention.

The imaging device 13, communication unit 14, memory unit 15, control unit 16, safe driving support unit 17, and data storage unit 18 may be combined in any combination, or their respective functions may be separated.

Figure 17 is a flowchart illustrating an example of the operation of the safe driving support system 11 according to the first application example.

As illustrated in FIG. 17, the determination device 12 acquires a plurality of captured images 31 captured by the imaging device 13 (step S21).

The determination device 12 executes a drowsiness determination process based on the acquired captured image 31 (step S22), and transmits the determination result of the degree of drowsiness to the safe driving support unit 17 and the data storage unit 18 via the communication unit 14. In the drowsiness determination process, the drowsiness determination unit 16b may execute processes similar to steps S1 to S12 shown in FIG. 13.

The safe driving support unit 17 implements safe driving support based on the received judgment result, and the data accumulation unit 18 accumulates the received judgment result (step S23), and the processing ends.

In this way, according to the first application example, the degree (state) of drowsiness of the user (driver) accurately determined by the drowsiness determination device 1 according to one embodiment can be utilized in implementing safe driving assistance.

[1-6-2] Second Application Example Fig. 18 is a block diagram showing a configuration example of a remote learning support system 21 according to a second application example. The remote learning support system 21 judges the drowsiness of a student, who is an example of a user, based on a first feature amount 32a relating to the opening and closing of the eyes of the student and a second feature amount 32b relating to the tension state of the facial muscles, and utilizes the judgment result for remote learning support. The remote learning may be, for example, remote education such as online classes and e-learning in which educational materials are delivered via the Internet to a computer such as a personal computer (PC) or a smartphone of the student.

As shown in FIG. 18, the remote learning support system 21 may, for example, include a determination device 22, a teacher feedback unit 27, an administrator feedback unit 28, and a teaching material adjustment unit 29.

The determination device 22 is a device that determines the drowsiness of a user based on an image 31 including the face of the user performing remote learning, and is an example of the drowsiness determination device 1 shown in FIG. 9. As shown in FIG. 18, the determination device 22 may exemplarily include an imaging device 23, a communication unit 24, a memory unit 25, and a control unit 26. Unless otherwise specified below, the imaging device 23, the communication unit 24, the memory unit 25, and the control unit 26 are similar to the imaging device 13, the communication unit 14, the memory unit 15, and the control unit 16 of the first application example shown in FIG. 15.

Figure 19 is a diagram showing an example of installation of the imaging device 23. As shown in Figure 19, the camera serving as the imaging device 23 may be a web camera, video camera, smartphone, or tablet that captures an area including the user's face, or may be provided in the participant's computer. The imaging device 23 may be provided outside the judgment device 22, in which case it may include a communication IF that transmits the captured image 31 to the judgment device 22.

The communication unit 24 communicates with each of the teacher feedback unit 27, the administrator feedback unit 28, and the teaching material adjustment unit 29, and may be realized by the IF unit 10d shown in FIG. 14. In addition, when the imaging device 23 is provided outside the determination device 22, the communication unit 24 may communicate with the imaging device 23.

The memory unit 25 stores various information related to the operation of the determination device 22. The memory unit 25 is an example of the memory unit 3 shown in FIG. 9, and may store the same information as the memory unit 3.

The control unit 26 performs various controls related to the operation of the determination device 22. For example, the control unit 26 may include a transceiver unit 26a, a drowsiness determination unit 26b, and an information control unit 26c.

The transmission/reception unit 26a controls the transmission and reception of data between the imaging device 23, and the transmission and reception of data between the imaging device 23, the teacher feedback unit 27, the administrator feedback unit 28, and the teaching material adjustment unit 29 via the communication unit 24.

The drowsiness determination unit 26b is an example of the estimation unit 4 shown in FIG. 9, and may execute a drowsiness determination process similar to that of the estimation unit 4. When the drowsiness determination unit 26b outputs an alertness state as the determination result, the drowsiness determination unit 26b may include in the determination result whether the alertness state is "general" or "exceptional (concentration)."

The information control unit 26c performs various controls regarding data exchanged between the teacher feedback unit 27, the administrator feedback unit 28, and the teaching material adjustment unit 29.

The teacher feedback unit 27 receives the judgment result judged by the judgment device 22 from the communication unit 24, and based on the judgment result, provides feedback of the judgment result to an instructor of the distance learning, for example, a teacher. For example, the teacher feedback unit 27 may accumulate and tally the judgment results to generate statistical data on the concentration and drowsiness information of the students, and output the statistical data to a computer owned by the teacher. This allows the teacher to adjust the content of the distance learning in real time based on the statistical data.

The administrator feedback unit 28 receives the judgment result judged by the judgment device 22 from the communication unit 24, and provides feedback of the judgment result to the administrator of the distance learning based on the judgment result. For example, the administrator feedback unit 28 may generate or obtain the statistical data described above and output it to a computer owned by the administrator. This allows the administrator to evaluate and improve the distance learning (classes) based on the statistical data.

The teaching material adjustment unit 29 receives the judgment results made by the judgment device 22 from the communication unit 24, stores them, and adjusts the teaching material content based on the judgment results or statistical data obtained from the judgment results. Adjustments to the teaching material content may include changing the teaching material content to be distributed, changing the playback speed or volume, etc. The adjustments made may be reflected in the teaching materials distributed to the students' computers.

The imaging device 23, communication unit 24, memory unit 25, control unit 26, teacher feedback unit 27, administrator feedback unit 28, and teaching material adjustment unit 29 may be combined in any combination, or their respective functions may be separated.

Figure 20 is a flowchart illustrating an example of the operation of the remote learning support system 21 according to the second application example.

As illustrated in FIG. 20, the determination device 22 acquires a plurality of captured images 31 captured by the imaging device 23 (step S31).

The determination device 22 executes a drowsiness determination process based on the acquired captured image 31 (step S32), and transmits the determination result of the degree of drowsiness to the teacher feedback unit 27, the administrator feedback unit 28, and the teaching material adjustment unit 29 via the communication unit 24. In the drowsiness determination process, the drowsiness determination unit 26b may execute processes similar to steps S1 to S12 shown in FIG. 13.

The teacher feedback unit 27 and the administrator feedback unit 28 transmit statistical data based on the received judgment results to the teacher's and administrator's computers, respectively, and the teaching material adjustment unit 29 adjusts the teaching material content based on the received judgment results (step S33), and the process ends.

In this way, according to the second application example, the degree (state) of drowsiness of a user (student) accurately determined by the drowsiness determination device 1 according to one embodiment can be utilized in implementing remote learning support.

[2] Others The above-described embodiment and the techniques according to the first and second application examples can be modified and changed as follows.

For example, the open/closed state blink estimation unit 41, the drowsiness level change estimation unit 42, the tension state estimation unit 43, the feature amount determination unit 44, and the drowsiness estimation unit 45 included in the estimation unit 4 of the drowsiness determination device 1 shown in FIG. 9 may be combined in any combination or may be separated.

The drowsiness determination device 1 shown in FIG. 9 may also be configured to realize each processing function by multiple devices cooperating with each other via a network. As an example, the imaging unit 2 may be a computer located near the user, the estimation unit 4 may be an application server, the memory unit 3 may be a DB server, etc. In this case, the computer, the application server, and the DB server may cooperate with each other via a network to realize each processing function of the drowsiness determination device 1.

Furthermore, when the estimation unit 4 (sleepiness estimation unit 45) of the drowsiness determination device 1 shown in FIG. 9 uses the second feature amount 32b to determine drowsiness, it may use multiple types of second feature amounts 32b to comprehensively determine drowsiness. That is, as the second feature amount 32b, two or more feature amounts, for example, the range of change of the orbicularis oris muscle (AU23) and the number of peaks of the mentalis muscle (AU17), may be used. In this case, the drowsiness estimation unit 45 may determine drowsiness based on the determination result based on the range of change of the orbicularis oris muscle (AU23) and the determination result based on the number of peaks of the mentalis muscle (AU17).

[3] Supplementary Note The following supplementary note is further disclosed with respect to the above embodiment and the first and second application examples.

(Appendix 1)
Acquire a plurality of captured images including a user's face;
detecting a change in a feature amount relating to opening and closing of the eyes included in the face based on the plurality of captured images;
determining a first weight for the feature quantity relating to eye opening/closing and a second weight for the feature quantity relating to facial muscle tension according to a change in the detected feature quantity relating to eye opening/closing;
determining drowsiness of the user based on at least one of a feature amount related to opening and closing of the eyes and a feature amount related to tension of the facial muscles in accordance with the determined first weight and the determined second weight;
A drowsiness determination program that causes a computer to execute processing.

(Appendix 2)
the process of determining the first weight and the second weight includes a process of determining the first weight and the second weight such that the first weight is greater than the second weight when an amount of change in the detected feature amount related to eye opening and closing is equal to or less than a first threshold, and determining the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.
A drowsiness determination program according to appendix 1.

(Appendix 3)
The process of determining the drowsiness of the user includes a process of determining that the drowsiness of the user is in an awake state when the feature amount related to the eye opening and closing is greater than a second threshold when the amount of change is equal to or less than the first threshold, and determining that the drowsiness of the user is in a drowsy state when the feature amount related to the eye opening and closing is equal to or less than the second threshold.
A drowsiness determination program according to appendix 2.

(Appendix 4)
The process of determining the drowsiness of the user includes a process of determining that the drowsiness of the user is in an awake state when the amount of change is greater than the first threshold and the feature amount related to the tension of the facial muscles is greater than a third threshold, and determining that the drowsiness of the user is in a drowsy state when the feature amount related to the tension of the facial muscles is equal to or less than the third threshold.
The drowsiness determination program according to claim 2 or 3.

(Appendix 5)
Calculating the facial muscle tension feature based on information about changes in facial mimic muscles.
The drowsiness determination program according to any one of claims 1 to 4, which causes the computer to execute a process.

(Appendix 6)
The information regarding the change in the facial mimic muscles includes a change in an AU (Action Unit) value in the plurality of captured images.
A drowsiness determination program according to claim 5.

(Appendix 7)
Acquire a plurality of captured images including a user's face;
detecting a change in a feature amount relating to opening and closing of the eyes included in the face based on the plurality of captured images;
determining a first weight for the feature quantity relating to eye opening/closing and a second weight for the feature quantity relating to facial muscle tension according to a change in the detected feature quantity relating to eye opening/closing;
determining drowsiness of the user based on at least one of a feature amount related to opening and closing of the eyes and a feature amount related to tension of the facial muscles in accordance with the determined first weight and the determined second weight;
A drowsiness determination device comprising a control unit.

(Appendix 8)
the control unit, in determining the first weight and the second weight, determines the first weight and the second weight such that the first weight is greater than the second weight when an amount of change in the detected feature amount related to eye opening and closing is equal to or less than a first threshold, and determines the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.
8. The drowsiness determination device according to claim 7.

(Appendix 9)
In determining the drowsiness of the user, when the amount of change is equal to or less than the first threshold and the feature amount related to eye opening and closing is greater than a second threshold, the control unit determines that the user is in an awake state, and when the feature amount related to eye opening and closing is equal to or less than the second threshold, determines that the user is in a drowsy state.
9. The drowsiness determination device according to claim 8.

(Appendix 10)
In determining the drowsiness of the user, the control unit determines that the user is in an awake state when the amount of change is greater than the first threshold and the feature amount related to the tension of the facial muscles is greater than a third threshold, and determines that the user is in a drowsy state when the feature amount related to the tension of the facial muscles is equal to or less than the third threshold.
10. The drowsiness determination device according to claim 8 or 9.

(Appendix 11)
The control unit calculates a feature amount related to tension of the facial muscles based on information about changes in the facial mimic muscles.
The drowsiness determination device according to any one of Supplementary Note 7 to Supplementary Note 10.

(Appendix 12)
The information regarding the change in the facial mimic muscles includes a change in an AU (Action Unit) value in the plurality of captured images.
12. The drowsiness determination device according to claim 11.

(Appendix 13)
Acquire a plurality of captured images including a user's face;
detecting a change in a feature amount relating to opening and closing of the eyes included in the face based on the plurality of captured images;
determining a first weight for the feature quantity relating to eye opening/closing and a second weight for the feature quantity relating to facial muscle tension according to a change in the detected feature quantity relating to eye opening/closing;
determining drowsiness of the user based on at least one of a feature amount related to opening and closing of the eyes and a feature amount related to tension of the facial muscles in accordance with the determined first weight and the determined second weight;
A method for determining drowsiness, the processing of which is performed by a computer.

(Appendix 14)
the process of determining the first weight and the second weight includes a process of determining the first weight and the second weight such that the first weight is greater than the second weight when an amount of change in the feature amount related to the detected eye opening and closing is equal to or less than a first threshold, and determining the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.
A method for determining drowsiness according to claim 13.

(Appendix 15)
The process of determining the drowsiness of the user includes a process of determining that the drowsiness of the user is in an awake state when the feature amount related to the eye opening and closing is greater than a second threshold when the amount of change is equal to or less than the first threshold, and determining that the drowsiness of the user is in a drowsy state when the feature amount related to the eye opening and closing is equal to or less than the second threshold.
A method for determining drowsiness according to claim 14.

(Appendix 16)
The process of determining the drowsiness of the user includes a process of determining that the drowsiness of the user is in an awake state when the amount of change is greater than the first threshold and the feature amount related to the tension of the facial muscles is greater than a third threshold, and determining that the drowsiness of the user is in a drowsy state when the feature amount related to the tension of the facial muscles is equal to or less than the third threshold.
The drowsiness determination method according to claim 14 or 15.

(Appendix 17)
Calculating the facial muscle tension feature based on information about changes in facial mimic muscles.
The drowsiness determination method according to any one of claims 13 to 16, wherein the processing is executed by the computer.

(Appendix 18)
The information regarding the change in the facial mimic muscles includes a change in an AU (Action Unit) value in the plurality of captured images.
A method for determining drowsiness according to claim 17.

REFERENCE SIGNS LIST 1 Drowsiness determination device 11 Safe driving support system 12, 22 Determination device 13, 23 Imaging device 14, 24 Communication unit 15, 25, 3 Memory unit 16, 26 Control unit 16a, 26a Transmitting/receiving unit 16b, 26b Drowsiness determination unit 16c, 26c Information control unit 17 Safe driving support unit 18 Data storage unit 2 Imaging unit 21 Remote learning support system 27 Teacher feedback unit 28 Administrator feedback unit 29 Teaching material adjustment unit 31 Captured image 32, 32a, 32b Feature amount 33 Output result 4 Estimation unit 41 Open/closed state blink estimation unit 42 Drowsiness level change estimation unit 43 Tension state estimation unit 44 Feature amount determination unit 45 Drowsiness estimation unit

Claims (7)

Acquire a plurality of captured images arranged in chronological order, each of which captures an area including a user's face;
detecting a change in a feature amount relating to opening and closing of the eyes included in the face based on the plurality of captured images;
determining a first weight for the feature amount related to the opening and closing of the eyes and a second weight for the feature amount related to tension of the facial muscles that does not affect the opening and closing of the eyes , according to a change in the feature amount related to the opening and closing of the eyes detected;
determining drowsiness of the user based on at least one of the feature amount related to eye opening/closing and the feature amount related to muscle tension in accordance with the determined first weight and the determined second weight;
The process is executed by a computer,
the process of determining the first weight and the second weight includes a process of determining the first weight and the second weight such that the first weight is greater than the second weight when an amount of change in the detected feature amount related to eye opening and closing is equal to or less than a first threshold, and determining the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.
Drowsiness assessment program. The process of determining the drowsiness of the user includes a process of determining that the drowsiness of the user is in an awake state when the feature amount related to the eye opening and closing is greater than a second threshold when the amount of change is equal to or less than the first threshold, and determining that the drowsiness of the user is in a drowsy state when the feature amount related to the eye opening and closing is equal to or less than the second threshold.
The drowsiness determination program according to claim 1 . The process of determining the drowsiness of the user includes a process of determining that the drowsiness of the user is in an awakened state when the amount of change is greater than the first threshold and the feature amount related to muscle tension is greater than a third threshold, and determining that the drowsiness of the user is in a drowsy state when the feature amount related to muscle tension is equal to or less than the third threshold.
The drowsiness determination program according to claim 1 or 2. Calculating the feature amount related to muscle tension based on information related to changes in the facial muscles.
The drowsiness determination program according to any one of claims 1 to 3, which causes the computer to execute a process. The information regarding the change in the facial mimic muscles includes a change in an AU (Action Unit) value in the plurality of captured images.
The drowsiness determination program according to claim 4 . Acquire a plurality of captured images arranged in chronological order, each of which captures an area including a user's face;
detecting a change in a feature amount relating to opening and closing of the eyes included in the face based on the plurality of captured images;
determining a first weight for the feature amount related to the opening and closing of the eyes and a second weight for the feature amount related to tension of the facial muscles that does not affect the opening and closing of the eyes , according to a change in the feature amount related to the opening and closing of the eyes detected;
determining drowsiness of the user based on at least one of the feature amount related to eye opening/closing and the feature amount related to muscle tension in accordance with the determined first weight and the determined second weight;
A control unit is provided,
the control unit, in the process of determining the first weight and the second weight, determines the first weight and the second weight such that the first weight is greater than the second weight when an amount of change in the detected feature amount related to eye opening and closing is equal to or less than a first threshold, and determines the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.
Drowsiness detection device. Acquire a plurality of captured images arranged in chronological order, each of which captures an area including a user's face;
detecting a change in a feature amount relating to opening and closing of the eyes included in the face based on the plurality of captured images;
determining a first weight for the feature amount related to the opening and closing of the eyes and a second weight for the feature amount related to tension of the facial muscles that does not affect the opening and closing of the eyes , according to a change in the feature amount related to the opening and closing of the eyes detected;
determining drowsiness of the user based on at least one of the feature amount related to eye opening/closing and the feature amount related to muscle tension in accordance with the determined first weight and the determined second weight;
The computer executes the process,
the process of determining the first weight and the second weight includes a process of determining the first weight and the second weight such that the first weight is greater than the second weight when an amount of change in the detected feature amount related to eye opening and closing is equal to or less than a first threshold, and determining the first weight and the second weight such that the second weight is greater than the first weight when the amount of change is greater than the first threshold.
Methods for assessing drowsiness.

Images