JP7580265B2 - Eye fatigue evaluation device, eye fatigue evaluation method, and program - Google Patents

Description

Embodiments of the present invention relate to an eye fatigue assessment device, an eye fatigue assessment method, and a program.

Conventionally, there is known a technique for measuring blinking as a technique for measuring eye fatigue (see, for example, Patent Document 1). This technique is a blink measurement device for determining the biological state of a measurement subject based on the blinking behavior of the measurement subject, and includes an imaging unit that images the eyes of the measurement subject, and an eyelid opening/closing measurement unit that calculates blink feature amounts related to the blinking behavior based on the image of the eye captured by the imaging unit, and determines the type of blinking behavior based on the blink feature amounts.

JP 2010-273954 A

The object of the present invention is to provide an eye fatigue evaluation device, an eye fatigue evaluation method, and a program that can evaluate eye fatigue.

One aspect of the present invention is an eye fatigue evaluation device that evaluates the eye fatigue of a subject, and includes a calculation unit that calculates features representing the shape of the eyes based on information identifying eye features obtained from an image of the subject's face, and a judgment unit that judges the eye fatigue of the subject based on the features calculated by the calculation unit , wherein the judgment unit judges the eye fatigue of the subject based on a statistical quantity derived based on the distribution of the features representing the eye shape calculated by the calculation unit at a predetermined time .

One aspect of the present invention is an eye fatigue evaluation method executed by a computer, comprising the steps of calculating features representing eye shape based on information identifying eye features obtained from an image of a subject's face , and judging the eye fatigue of the subject based on the features calculated in the calculating step, wherein in the judging step, the eye fatigue of the subject is judged based on a statistical quantity derived based on the distribution of the features representing the eye shape calculated in the calculating step at a predetermined time .

One aspect of the present invention is a program that causes a computer to execute the steps of calculating features representing eye shape based on information identifying eye features obtained from an image of the subject's face, and judging eye fatigue of the subject based on the features calculated in the calculating step , and in the judging step, judges eye fatigue of the subject based on statistics derived based on the distribution of the features representing eye shape calculated in the calculating step at a predetermined time .

According to an embodiment of the present invention, it is possible to provide an eye fatigue evaluation device, an eye fatigue evaluation method, and a program that can evaluate eye fatigue.

1 is a diagram showing an eye fatigue evaluation system according to an embodiment of the present invention. 5 is a diagram showing an example of feature points detected by the eye fatigue evaluation device according to the present embodiment. FIG. FIG. 4 is a diagram illustrating an example of a process of the eye fatigue evaluation device according to the present embodiment. FIG. 4 is a diagram illustrating an example of a process of the eye fatigue evaluation device according to the present embodiment. 1 is a diagram showing Example 1 of statistical data of aspect ratios calculated by the eye fatigue evaluation device according to the present embodiment; FIG. FIG. 11 is a diagram showing Example 2 of statistical data of aspect ratios calculated by the eye fatigue evaluation device according to the present embodiment. 11 is a diagram showing an example of a time change in probability density distribution of a right eye aspect ratio calculated by the eye fatigue evaluation device according to the present embodiment. FIG. 1 is a diagram showing an example of an aspect ratio interval, a probability density, and an eye fatigue index calculated by the eye fatigue evaluation device according to the present embodiment. FIG. 5 is a diagram showing an example of a time change in eye fatigue index calculated by the eye fatigue evaluation device according to the present embodiment. FIG. FIG. 4 is a flow chart showing an example of the operation of the eye fatigue evaluation system according to the present embodiment. FIG. 13 is a diagram showing an example of a display of an evaluation result of the eye fatigue evaluation system according to the present embodiment. 13A and 13B are diagrams illustrating an example of a process of an eye fatigue assessment device according to a modified example of an FIG. 11 is a diagram showing Example 1 of statistical data of the right opening angle calculated by the eye fatigue evaluation device according to the modified example of the embodiment. FIG. 11 is a diagram showing Example 2 of statistical data of the left opening angle calculated by the eye fatigue evaluation device according to the modified example of the embodiment. 13A and 13B are diagrams showing an example of intervals, probability densities, and eye fatigue indexes of opening angles calculated by the eye fatigue evaluation device according to the modified example of the embodiment. FIG. 11 is a diagram showing an example of a time change in eye fatigue index calculated by an eye fatigue evaluation device according to a modified example of an embodiment. FIG. 11 is a flowchart showing an example of the operation of the eye fatigue evaluation system according to the modified example of the embodiment. FIG. 13 is a diagram showing an example of a display of an evaluation result of the eye fatigue evaluation system according to the modified example of the embodiment.

<Embodiment>
Hereinafter, an eye fatigue evaluation system according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing the eye fatigue evaluation system according to the present embodiment. The eye fatigue evaluation system 100 includes an imaging unit 1, a display device 2, and an eye fatigue evaluation device 10. The imaging unit 1 and the display device 2 are connected to the eye fatigue evaluation device 10.
The imaging unit 1 captures an image of a subject S.
The display device 2 displays the image captured by the imaging unit 1 .
The eye fatigue assessment device 10 allows the subject S to check the positional relationship between the imaging unit 1 and the face of the subject S by displaying an image captured by the imaging unit 1 on the display device 2. The eye fatigue assessment device 10 displays a gaze point on the display device 2 and has the subject S gaze at the gaze point. This configuration allows the subject S to keep his/her line of sight constant. The eye fatigue assessment device 10 evaluates the eye fatigue of the subject S based on the image captured by the imaging unit 1.

In one example of the eye fatigue evaluation system 100, the imaging unit 1 is fixed on a desk. The subject S sits on a chair and gazes at the gaze point displayed on the display device 2.
The imaging unit 1 creates multiple image data of the subject S by imaging the subject S for a predetermined time. In this embodiment, the explanation will continue with a case where the predetermined time is 60 seconds as an example. The eye fatigue evaluation device 10 acquires multiple image data of the subject S created by the imaging unit 1. If the imaging unit 1 can capture 30 frames per second, 1800 frames can be captured in 60 seconds, so the eye fatigue evaluation device 10 acquires approximately 1800 pieces of image data. The eye fatigue evaluation device 10 records the multiple acquired image data of the subject S.

The eye fatigue assessment device 10 calls up the recorded multiple image data into a memory and processes the multiple image data called up into the memory to detect the face of the subject S from each of the multiple images of the subject S. For example, the eye fatigue assessment device 10 obtains information specifying a rectangle (quadrangle) enclosing the face of the subject S as a result of detecting the face of the subject S from the image of the subject S.
The eye fatigue evaluation device 10 detects facial feature points (landmarks) of the subject S from each of multiple images of the subject S based on information identifying a rectangle enclosing the subject S's face obtained from each of multiple images of the subject S.

The eye fatigue evaluation device 10 detects eye feature points of the subject S from each of multiple images of the subject S based on the detection results of the facial feature points of the subject S obtained from each of multiple images of the subject S.
The eye fatigue evaluation device 10 calculates features representing the shape of the subject S's eyes for each of the multiple images of the subject S based on the detection results of the feature points of the subject S's eyes obtained from each of the multiple images of the subject S.
The eye fatigue evaluation device 10 evaluates the eye fatigue of the subject S based on feature amounts representing the eye shape of the subject S calculated for each of a plurality of images of the subject S.

The eye fatigue evaluation device 10 will be described in detail below.
The eye fatigue evaluation device 10 includes an input/output I/F 11, a CPU (Central Processing Unit) 12, a memory 13, a recording device 14, a data reading unit 15, and a bus 16. The bus 16 connects the input/output I/F 11, the CPU 12, the memory 13, the recording device 14, and the data reading unit 15 to one another.
The input/output I/F 11 is an interface for inputting and outputting information between the image capturing unit 1 and the display device 2 and between the eye fatigue evaluation device 10 and the image capturing unit 1 and the display device 2 .
The data reading unit 15 acquires image data of the subject S from the imaging unit 1 via the bus 16. The data reading unit 15 associates the acquired image data with identification information of the subject S and records it in the recording device 14.

The CPU 12 executes a computer program (software) stored in the recording device 14 to function as a processing unit 12-1, a first detection unit 12-2, a second detection unit 12-3, a calculation unit 12-4, a determination unit 12-5, and a warning unit 12-6.
The processing unit 12-1 acquires a plurality of image data items recorded in association with identification information of the subject S from among the image data items recorded in the recording device 14. The processing unit 12-1 calls up the acquired plurality of image data items into the memory 13. The processing unit 12-1 processes each of the plurality of image data items called up into the memory 13 to acquire an image of the subject S.

The first detection unit 12-2 acquires a plurality of images of the subject S from the processing unit 12-1. The first detection unit 12-2 executes software stored in the recording device 14 to detect the face of the subject S from each of the acquired plurality of images of the subject S. As face detection software, various software are provided as a library. In this embodiment, the description will continue with a case where OpenCV's Haar Cascades is used as an example of face detection software. OpenCV's Haar Cascades provides both a learning machine and a detector.
In this embodiment, the learning machine has previously learned the relationship between images of people and images of people's faces based on multiple images of people and images of people's faces contained in each of the multiple images of people, and we will continue to explain the case where the detector detects images of people's faces from images of people using the learning results of the learning machine's relationship between images of people and images of people's faces.

As a result of detecting the face of the subject S from each of the multiple images captured of the subject S, the first detection unit 12-2 detects a rectangle (quadrant) enclosing the facial portion of the subject S in each of the multiple images captured of the subject S. Here, an example of information specifying the rectangle enclosing the facial portion of the subject S is coordinates VC(x, y, w, h). In the information specifying the rectangle enclosing the facial portion of the subject S, x and y are information specifying the X-coordinate and Y-coordinate of the upper left corner of the rectangle, respectively, and w and h are information specifying the width and height of the rectangle, respectively.
The first detection unit 12-2 acquires coordinates VC from each of a plurality of images captured of the subject S. By acquiring the coordinates VC from each of a plurality of images captured of the subject S, the first detection unit 12-2 can acquire (identify) the position of the face of the subject S from each of a plurality of images captured of the subject S based on each of the acquired coordinates VC.
The first detection unit 12-2 may change the size of the image of the subject S based on each of the acquired coordinates VC in order to make the size of the rectangle represented by the coordinates VC constant. Below, the case where the first detection unit 12-2 changes the size of the image of the subject S based on each of the coordinates VC will be described. By configuring in this way, it is possible to correct errors caused by differences in the size of the rectangle surrounding the face of the subject S specified by the coordinates VC.

The second detection unit 12-3 acquires from the first detection unit 12-2 a plurality of images of the subject S and information specifying the coordinates VC corresponding to each of the plurality of images of the subject S. The second detection unit 12-3 detects an image of the face of the subject S from each of the plurality of images of the subject S, based on the acquired plurality of images of the subject S and a plurality of pieces of information specifying the coordinates VC.
The second detection unit 12-3 detects facial feature points of the subject S from each of the detected images of the face of the subject S. In this embodiment, as an example, the case where the ensemble of regression trees is used to detect the facial feature points of the subject S will be described. By using the ensemble of regression trees, the facial feature points of the subject S can be detected in real time with high accuracy using regression tree analysis.
In this embodiment, the learning machine has previously learned the relationship between the human face images and the human face feature points based on the multiple human face images and the human face feature points contained in each of the multiple human face images, and the detector detects the human face feature points from the human face images using the learning result of the relationship between the human face images and the human face feature points by the learning machine. In this embodiment, the explanation will continue with the case where 68 human face feature points are detected from the human face image, as an example.

The second detection unit 12-3 detects 68 feature points FP01 to FP68 from one facial image of the subject S. The second detection unit 12-3 acquires the coordinates of each of the detected feature points FP01 to FP68. Here, an example of the coordinates is a coordinate for identifying the position of the subject S in the captured image.
2 is a diagram showing an example of feature points detected by the eye fatigue assessment device according to the present embodiment, in which each of the 68 feature points is indicated by a point and a number that identifies the point.
The second detection unit 12-3 acquires the coordinates of each of the feature points corresponding to the left eye and the coordinates of each of the feature points corresponding to the right eye from the coordinates of each of the acquired feature points FP01 to FP68. Specifically, the second detection unit 12-3 acquires the coordinates of each of the feature points FP37 to FP42 corresponding to the left eye, and acquires the coordinates of each of the feature points FP43 to FP48 corresponding to the left eye. The second detection unit 12-3 creates array data including information specifying the coordinates of each of the acquired feature points FP37 to FP42 and coordinates specifying the coordinates of each of the acquired feature points FP43 to FP48. Returning to FIG. 1, the explanation will be continued.

The calculation unit 12-4 acquires the array data created by the second detection unit 12-3. The calculation unit 12-4 acquires the coordinates of each of the multiple feature points corresponding to the left eye portion and the coordinates of each of the multiple feature points corresponding to the right eye portion included in the acquired array data. The calculation unit 12-4 calculates a feature amount representing the shape of the left eye and a feature amount representing the shape of the right eye based on the coordinates of each of the multiple feature points corresponding to the left eye portion and the coordinates of each of the multiple feature points corresponding to the right eye portion acquired. In this embodiment, the aspect ratio represented by the ratio of the length (number of pixels) of the left eye image (hereinafter referred to as the "left eye aspect ratio") is used as an example of a feature amount representing the shape of the left eye, and the aspect ratio represented by the ratio of the length (number of pixels) of the right eye image (hereinafter referred to as the "right eye aspect ratio") is used as an example of a feature amount representing the shape of the right eye.

3A and 3B are diagrams showing an example of the process of the eye fatigue evaluation device according to the present embodiment, in which characteristic points P1 to P6 corresponding to the right eye are shown as an example.
As shown in FIG. 3A , an example of feature point P1 (corresponding to FP37 in FIG. 2 ) corresponds to the outer corner of the right eye, an example of feature point P2 (corresponding to FP38 in FIG. 2 ) corresponds to the intersection of the right boundary between the white and pupil of the eye and the upper eyelid in the right eye, and examples of feature point P2 (corresponding to FP38 in FIG. 2 ) and feature point P3 (corresponding to FP39 in FIG. 2 ) correspond to points in the right eye between the outer corner at feature point P1 and the inner corner at feature point P4, which divide the boundary line between the upper eyelid and the eye into thirds at equal intervals. As shown in FIG. 3A , an example of feature point P4 (corresponding to FP40 in FIG. 2 ) corresponds to the inner corner of the right eye, and examples of feature point P5 (corresponding to FP41 in FIG. 2 ) and feature point P6 (corresponding to FP42 in FIG. 2 ) correspond to points that divide the boundary between the lower eyelid and the eye into thirds at equal intervals between the outer corner of the eye at feature point P1 and the inner corner of the eye at feature point P4 in the right eye.

The calculation unit 12-4 calculates the right eye aspect ratio REAR using equation (1) based on the coordinates of each of the characteristic points P1 to P6 corresponding to the right eye.
REAR=(||P2-P6||+||P3-P5||)/2||P1-P4|| (1)
The calculation unit 12-4 calculates the left eye aspect ratio LEAR based on the coordinates of each of the six feature points corresponding to the left eye. Details of the process by which the calculation unit 12-4 calculates the left eye aspect ratio LEAR can be applied to the process of calculating the right eye aspect ratio REAR described above, so that the explanation will be omitted here.
The calculation unit 12-4 estimates the movement (orientation) of the subject S's head with respect to each of the X-axis, Y-axis, and Z-axis using the coordinates of a predetermined feature point among the feature points FP01 to FP68 of a person's face in order to correct an error caused by a difference in the positional relationship between the subject S and the imaging unit 1 in each of a plurality of images captured of the subject S. The calculation unit 12-4 performs error correction for each of the left eye aspect ratio and the right eye aspect ratio based on the estimation result of the movement of the subject S's head with respect to each of the X-axis, Y-axis, and Z-axis. The calculation unit 12-4 records the result of performing error correction for each of the left eye aspect ratio and the right eye aspect ratio in the recording device 14. Hereinafter, the corrected left eye aspect ratio and right eye aspect ratio are used.

カーネル関数ｋには、色々な種類があるが、今回は、式（３）によって表されるガウスカーネルを設定して、算出した。

式（２）に式（３）を代入すると、式（４）が得られる。

ｘの確率密度ｐ（ｘ）は、平均ｘ、標準偏差ｈの正規分布から求めたサンプル数Ｎ個の確率密度の平均値である。ｈに関しては、トライ＆エラーで、そのヒストグラムに対する当てはまりの良さで、決める必要がある。
図４は、一例として、画像表示端末(VDT: Visual Display Terminals)作業を開始する前の左目アスペクト比の確率密度分布を示す。
左目アスペクト比の統計的データを取得するにあたり、被検者Ｓの撮像を６０秒間行った。６０秒間の撮像で得られた画像データのうち、目の動き、頭の動きが比較的安定していると想定される撮像が開始されてから２０秒以後の１０００点のデータを利用して、確率密度関数を算出した。 4 is a diagram showing an example 1 of statistical data of the aspect ratio calculated by the eye fatigue evaluation device according to the present embodiment. In Fig. 4, the calculated aspect ratio data is represented by a histogram that visualizes the relative frequency. The same applies to the histograms shown below.
In reality, the calculated aspect ratio data is considered to vary infinitely and continuously within a certain interval, and can be expressed as a graph of the probability density distribution of the aspect ratio, as shown by the solid line.
Methods for estimating the probability density distribution of the aspect ratio include histogram density estimation and kernel density estimation, and the estimation method in FIG. 4 was performed using kernel density estimation.
The mathematical formula for kernel density estimation is given by Equation (2).

There are various types of kernel function k, but in this example, the Gaussian kernel represented by the formula (3) was set and used for calculation.

By substituting equation (3) into equation (2), equation (4) is obtained.

The probability density p(x) of x is the average value of the probability density of N samples obtained from a normal distribution with mean x and standard deviation h. h must be determined by trial and error based on how well it fits the histogram.
FIG. 4 shows, as an example, a probability density distribution of the left eye aspect ratio before starting work on a visual display terminal (VDT).
To obtain statistical data on the left eye aspect ratio, subject S was imaged for 60 seconds. Of the image data obtained during the 60 seconds of imaging, 1000 points of data from 20 seconds after the start of imaging, when eye and head movements are assumed to be relatively stable, were used to calculate a probability density function.

4, (1) shows an example of the probability density distribution of the left eye aspect ratio. According to this figure, the area surrounded by the probability density distribution of the left eye aspect ratio and the X-axis is 1.
In FIG. 4, (2) shows another example of the probability density distribution of the left eye aspect ratio.
The determination unit 12-5 acquires information specifying each of the left eye aspect ratios calculated by the calculation unit 12-4. The determination unit 12-5 calculates an index indicating eye fatigue of the left eye (hereinafter referred to as a "left eye eye fatigue index") based on the acquired information specifying each of the left eye aspect ratios.
Specifically, the determination unit 12-5 calculates the ratio between the peak and the range of aspect ratios corresponding to a predetermined ratio in the probability density distribution of the left-eye aspect ratio. In this embodiment, the explanation will continue for the case where 95% (probability distribution from 2.5% to 97.5%) is applied as an example of the predetermined ratio.

When 95% is applied as the specified rate, as shown in equation (2), the judgment unit 12-5 defines the range of the left eye aspect ratio where the probability distribution is 2.5% to 97.5% as xl, the range of the maximum value in the probability density function of the left eye aspect ratio from 2.5% to 97.5%, and the maximum value (peak) of the probability density function of the left eye aspect ratio as yl, and determines the ratio of xl to yl as an index of left eye fatigue.
Specifically, the peak of the probability density distribution of the left eye aspect ratio is 18.4195, and the range of aspect ratios corresponding to 95% is 0.0789, so the determination unit 12-5 calculates 233.3 as the left eye fatigue index.
Left eye fatigue index=(value at which the probability density function of the aspect ratio of the left eye is maximized (yl value))/(range of 2.5% or more in the probability distribution of the aspect ratio (xl value)) (2)

Fig. 5 is a diagram showing Example 2 of statistical data of aspect ratios calculated by the eye fatigue evaluation device according to the present embodiment. Fig. 5 shows, as an example, a probability density distribution of the right eye aspect ratio before the start of VDT work. In Fig. 5, the horizontal axis represents the random variable, and the vertical axis represents the probability density at that time.
To obtain statistical data on the right eye aspect ratio, subject S was imaged for 60 seconds. Of the image data obtained during the 60 seconds of imaging, 1000 points of data from 20 seconds after the start of imaging, when eye and head movements are assumed to be relatively stable, were used to calculate a probability density function.

5, (1) shows an example of the probability density distribution of the right-eye aspect ratio. According to this figure, the area surrounded by the probability density distribution of the right-eye aspect ratio and the X-axis is 1.
In FIG. 5, (2) shows another example of the probability density distribution of the right-eye aspect ratio.
The determination unit 12-5 acquires information specifying each of the multiple right eye aspect ratios calculated by the calculation unit 12-4. The determination unit 12-5 calculates an index indicating right eye fatigue (hereinafter referred to as a "right eye fatigue index") based on the acquired information specifying each of the multiple right eye aspect ratios.
Specifically, the determination unit 12-5 calculates the ratio between the peak and the range of aspect ratios corresponding to a predetermined ratio in the probability density distribution of the right-eye aspect ratio. In this embodiment, the explanation will continue for the case where 95% (probability distribution from 2.5% to 97.5%) is applied as an example of the predetermined ratio.

When 95% is applied as the specified rate, as shown in equation (3), the judgment unit 12-5 defines the range of the right eye aspect ratio where the probability distribution is 2.5% to 97.5% as xr, the range of the maximum value in the probability density function of the right eye aspect ratio from 2.5% to 97.5%, and the maximum value (peak) of the probability density function of the right eye aspect ratio as yr, and determines the ratio of xr to yr as an index of right eye fatigue.
Specifically, the peak of the probability density distribution of the right eye aspect ratio is 19.0934, and the range of aspect ratios corresponding to 95% is 0.0807, so the determination unit 12-5 calculates 236.5 as the right eye fatigue index.
Right eye eye fatigue index=(value at which the probability density function of the aspect ratio of the right eye is maximized (y value))/(range of 2.5% or more in the probability distribution of the aspect ratio (x value)) (3)
Returning to FIG.

The determination unit 12-5 performs an evaluation by determining the eye fatigue of the subject S based on the calculated right eye eye fatigue index and left eye eye fatigue index.
FIG. 6 is a diagram showing an example of a change over time in the probability density distribution of the right eye aspect ratio calculated by the eye fatigue evaluation device according to the present embodiment.
As an example, Fig. 6 shows the probability density distribution of the right eye aspect ratio acquired every hour from before the start of VDT work until four hours have passed since the start of VDT work. In Fig. 6, the horizontal axis represents the random variable, and the vertical axis represents the probability density at that time.
Furthermore, FIG. 6 also shows the change over time in the right eye eye fatigue index calculated based on the probability density distribution of the right eye aspect ratio acquired every hour.

Figure 6 shows that the maximum value (peak) (yr value) of the probability density function for the right eye aspect ratio is stable and large at a certain aspect ratio before VDT work begins (when eye fatigue is low) compared to after VDT work begins. In other words, as time passes during VDT work, the maximum value of the probability density function for the right eye aspect ratio decreases, resulting in a broader probability density distribution (probability density at the most frequent value decreases). In other words, the degree of right eye fatigue can be determined based on the maximum value of the probability density function for the right eye aspect ratio.

An example of the change over time in the probability density distribution of the left eye aspect ratio is omitted from the illustration, as it shows the same tendency as the right eye aspect ratio described above. As with the maximum value (peak) of the probability density function for the right eye aspect ratio, the maximum value (peak) (yl value) of the probability density function for the left eye aspect ratio also takes a stable and large value at a certain aspect ratio before VDT work begins (when eye fatigue is small) compared to after VDT work begins. In other words, it can be seen that as time passes during VDT work, the maximum value of the probability density function for the left eye aspect ratio becomes lower, resulting in a broader probability density distribution (probability density at the most frequent value decreases). In other words, the degree of fatigue of the left eye can be known based on the maximum value of the probability density function for the left eye aspect ratio.

FIG. 7 is a diagram showing an example of aspect ratio intervals, probability densities, and eye fatigue indexes calculated by the eye fatigue evaluation device according to the present embodiment.
7, the right eye fatigue index is 236.6 before the start of VDT work, 171.6 one hour after the start of VDT work, 141.3 two hours after the start of VDT work, 122.8 three hours after the start of VDT work, and 90.6 four hours after the start of VDT work. It can be seen that the right eye fatigue index decreases according to the time that has passed since the start of VDT work.

The left eye fatigue index is 233.5 before the start of VDT work, 183.1 one hour after the start of VDT work, 138.4 two hours after the start of VDT work, 125.5 three hours after the start of VDT work, and 74.4 four hours after the start of VDT work. It can be seen that the left eye fatigue index decreases according to the time that has elapsed since the start of VDT work.

The decrease in the right eye fatigue index and left eye fatigue index is assumed to be due to eye fatigue caused by long-term use of VDT devices such as smartphones and personal computers. When concentrating on staring at a screen and the eyeballs are not moving for a long time, the seven muscles that move the eyeballs and the muscles that adjust the thickness of the "crystalline lens" to focus become fatigued. In addition, the number of times the eyes blink decreases during VDT work.
Furthermore, the decrease in the right eye fatigue index and the left eye fatigue index is also thought to be due to dry eyes, which make tears evaporate more easily when wearing contact lenses or in an environment exposed to air conditioning, etc. Due to such causes, the probability of the aspect ratio becoming smaller increases, and the xr value in the denominator of the right eye fatigue index and the xl value in the denominator of the left eye fatigue index increase, causing a decrease in each of the right eye fatigue index and the left eye fatigue index, which indicate an increase in eye fatigue.

The determination unit 12-5 also performs statistical processing based on the right eye fatigue index and the left eye fatigue index to calculate an eye fatigue index that is an index of eye fatigue for both the right eye and the left eye combined. In this embodiment, the explanation will continue with the case where an average value is used as an example of statistical processing.
7, the eye fatigue index (average of left and right eyes) is 235.0 before the start of VDT work, 177.3 one hour after the start of VDT work, 139.8 two hours after the start of VDT work, 124.1 three hours after the start of VDT work, and 82.4 four hours after the start of VDT work. It can be seen that the fatigue index decreases with the time elapsed since the start of VDT work, just like the right eye fatigue index and the left eye fatigue index.

8 is a diagram showing an example of a change over time in the eye fatigue index calculated by the eye fatigue evaluation device according to the present embodiment, in which the horizontal axis represents the elapsed time from the start of the VDT work, and the vertical axis represents the eye fatigue index.
According to FIG. 8, it can be seen that the eye fatigue index decreases by about 25% after one hour has elapsed since the start of the VDT work. Therefore, when the eye fatigue index decreases by 25% or more from the initial value, it is preferable that the eye fatigue evaluation device 10 has the subject S stop the VDT work and take a rest for a certain period of time. Before resuming the work, the eye fatigue evaluation device 10 calculates the right eye fatigue index and the left eye fatigue index of the subject S again, and judges whether the calculated fatigue index has decreased by 25% or more based on the calculated right eye fatigue index and the left eye fatigue index. The eye fatigue evaluation device 10 performs the same process based on the calculated fatigue index after that, when the fatigue index has decreased by 25% or more. By configuring in this way, it is assumed that the subject S can prevent symptoms such as eye pain, decreased eyesight, stiff shoulders, and headaches. Returning to FIG. 1, the explanation will be continued.

The warning unit 12-6 acquires the eye fatigue index calculated by the determination unit 12-5. The warning unit 12-6 judges whether the acquired eye fatigue index is less than the eye fatigue index threshold. An example of the eye fatigue index threshold is a value that is 25% lower than the initial value of the eye fatigue index. The warning unit 12-6 issues a warning when the eye fatigue index is less than the eye fatigue index threshold. Specifically, the warning unit 12-6 issues a warning to the subject S or the administrator of the subject S.

(Operation of eye fatigue evaluation system 100)
FIG. 9 is a flow diagram showing an example of the operation of the eye fatigue evaluation system according to the present embodiment.
(S1-1)
The eye fatigue assessment device 10 displays an image captured by the imaging unit 1 on the display device 2. The eye fatigue assessment device 10 displays a gaze point on the display device 2 and has the subject S gaze at the gaze point.
(S2-1)
The imaging unit 1 creates image data of the subject S by imaging the subject S for a predetermined period of time. The data reading unit 15 acquires the image data of the subject S from the imaging unit 1 via the bus 16, and records the acquired image data in the recording device 14 in association with identification information of the subject S.
(S3-1)
In the eye fatigue evaluation device 10, the processing unit 12-1 calls up image data from the recording device 14 to the memory 13. The processing unit 12-1 acquires an image of the subject S by processing the image data called up to the memory 13. The first detection unit 12-2 acquires the image of the subject S from the processing unit 12-1. The first detection unit 12-2 detects the face of the subject S from the acquired image of the subject S.

(S4-1)
In the eye fatigue evaluation device 10, the first detection unit 12-2 changes the size of the image of the subject S.
(S5-1)
In the eye fatigue evaluation device 10, the second detection unit 12-3 acquires an image and coordinates VC of the subject S from the first detection unit 12-2. The second detection unit 12-3 detects the face of the subject S based on the acquired image and coordinates VC of the subject S, and detects feature points of the subject S's face from the detected image of the subject S's face.
(S6-1)
In the eye fatigue evaluation device 10, the second detection unit 12-3 acquires the coordinates of the feature points corresponding to the left eye portion and the coordinates of the feature points corresponding to the right eye portion from the coordinates VC of each of the acquired feature points. The calculation unit 12-4 acquires the coordinates of the feature points corresponding to the left eye portion and the coordinates of the feature points corresponding to the right eye portion acquired by the second detection unit 12-3. The calculation unit 12-4 calculates the left eye aspect ratio and the right eye aspect ratio of the subject S based on the acquired coordinates of the feature points corresponding to the left eye portion and the coordinates of the feature points corresponding to the right eye portion.

(S7-1)
In the eye fatigue evaluation device 10, a calculation unit 12-4 estimates the movement (orientation) of the head of a subject S with respect to each of the X-axis, Y-axis, and Z-axis by using the coordinates of predetermined feature points among feature points FP01 to FP68 of a human face.
(S8-1)
In the eye fatigue evaluation device 10, a calculation unit 12-4 performs error correction for each of the left eye aspect ratio and the right eye aspect ratio based on the estimated results of the movement (orientation) of the subject S's head along each of the X-axis, Y-axis, and Z-axis.
(S9-1)
In the eye fatigue evaluation device 10, the determination unit 12-5 calculates a right eye eye fatigue index based on the probability density distribution of the right eye aspect ratio. The calculation unit 12-4 calculates a left eye eye fatigue index based on the probability density distribution of the left eye aspect ratio. The determination unit 12-5 calculates the eye fatigue index by performing statistical processing based on the calculated right eye eye fatigue index and left eye eye fatigue index.
(S10-1)
In the eye fatigue evaluation device 10, the judgment unit 12-5 judges the degree of eye fatigue on the basis of the calculated eye fatigue index.
9, the order of steps S6-1 and S7-1 may be interchanged, i.e., steps S1-1 to S5-1, S7-1, S6-1, S8-1 to S10-1 may be executed in this order.

In the above embodiment, the first detection unit 12-2 has been described as using OpenCV Haar Cascades when detecting the face of the subject S from the image of the subject S, but is not limited to this example. For example, the first detection unit 12-2 may use an Active appearance model to detect the face of the subject S from the image of the subject S. The Active appearance model performs object detection based on a statistical model learned from the shape and appearance of the object. For example, the first detection unit 12-2 may use Local Binary Features to detect the face of the subject S from the image of the subject S. The Local Binary Features detect features very quickly by regression learning.
In the above-described embodiment, the first detection unit 12-2 detects the face of one subject S from the image of the subject S, but the present invention is not limited to this example. For example, the first detection unit 12-2 may detect the faces of each of multiple subjects S from the image of the subject S. In this case, the eye fatigue assessment device 10 may determine eye fatigue for each of the multiple subjects S.

In the above-described embodiment, the determination unit 12-5 calculates a right-eye eye fatigue index based on the probability density distribution of the right eye aspect ratio, and calculates a left-eye eye fatigue index based on the probability density distribution of the left eye aspect ratio. The determination unit 12-5 has calculated the eye fatigue index by performing statistical processing based on the calculated right-eye eye fatigue index and left-eye eye fatigue index, but this example is not limited to the above. For example, the determination unit 12-5 may calculate a right-eye eye fatigue index based on the probability density distribution of the right eye aspect ratio, and calculate the eye fatigue index by performing statistical processing based on the calculated right-eye eye fatigue index. For example, the determination unit 12-5 may calculate a left-eye eye fatigue index based on the probability density distribution of the left eye aspect ratio, and calculate the eye fatigue index by performing statistical processing based on the calculated left-eye eye fatigue index.

In the above embodiment, the warning unit 12-6 acquires the eye fatigue index calculated by the determination unit 12-5, and issues a warning if the acquired eye fatigue index is less than the eye fatigue index threshold value, but this is not limited to the example. For example, the warning unit 12-6 may determine whether the acquired eye fatigue index corresponds to any of the attention levels based on multiple eye fatigue index threshold values. Furthermore, the warning unit 12-6 may display the determination result of whether the acquired eye fatigue index corresponds to any of the attention levels on the display device 2.

10 is a diagram showing an example of a display of the evaluation result of the eye fatigue evaluation system according to the present embodiment. In the example shown in Fig. 10, the judgment result of the eye fatigue index is displayed using a pie chart.
Specifically, 177, 140, 124, and 82 are set as examples of eye fatigue index thresholds. In the pie chart, the following ranges are set: a level corresponding to no caution where the eye fatigue index is 177 or more, a caution level 1 where the eye fatigue index is 140 or more and less than 177, a caution level 2 where the eye fatigue index is 124 or more and less than 140, a caution level 3 where the eye fatigue index is 82 or more and less than 140, and a caution level 4 where the eye fatigue index is less than 82, and each range is displayed in a different color. The warning unit 12-6 displays the eye fatigue index with an arrow AR.
With this configuration, the subject S can refer to the result of the eye fatigue index assessment and know the degree of his or her eye fatigue. Here, as an example, a case where the number of eye fatigue index thresholds is four has been described, but the number of eye fatigue index thresholds may be two or three, or five or more.

The eye fatigue assessment device 10 may also predict how long the state of eye fatigue will continue under the same environment based on the calculated eye fatigue index. By configuring in this way, the timing when the subject S should take a break can be derived based on the prediction result.
Also, the measurement program may be automatically started on a terminal device such as a smartphone, tablet, or notebook personal computer by setting the device to automatically calculate the eye fatigue index for a user performing VDT work. The terminal device may capture an image of the user with an imaging device mounted thereon, and calculate the eye fatigue index by performing the above-mentioned processing based on image data obtained by capturing the image. The terminal device judges the eye fatigue of the user based on the eye fatigue index, and displays the judgment result based on the eye fatigue threshold of the user on the monitor of the device. By configuring in this way, the eye fatigue of the user can be constantly judged and the judgment result of the eye fatigue can be displayed, so that information regarding eye fatigue can be shown to the user.

According to the eye fatigue assessment system 100 of this embodiment, the eye fatigue assessment device 10 assesses the eye fatigue of a subject. The eye fatigue assessment device 10 includes a calculation unit 12-4 that calculates a feature amount representing an eye shape based on information identifying feature points of the eyes obtained from an image of the face of the subject, and a determination unit 12-5 that determines the eye fatigue of the subject based on the feature amount calculated by the calculation unit 12-4.
With this configuration, the eye fatigue assessment device 10 can calculate features representing the shape of the eyes based on information identifying eye feature points obtained from an image of the face of the subject, and can therefore determine the eye fatigue of the subject based on the calculation results of the features representing the shape of the eyes.

Since eye fatigue in the early stages is not noticeable, images are captured periodically using an imaging device implemented in a terminal device such as a smartphone, tablet, or laptop computer, and the eye fatigue evaluation device 10 calculates an eye fatigue index by carrying out the above-mentioned processing based on the image data created by capturing images using the imaging device. The eye fatigue evaluation device 10 judges whether or not the subject S is suffering from eye fatigue based on the calculated eye fatigue index. If the eye fatigue evaluation device 10 judges that the subject S is suffering from eye fatigue, the subject S may stop VDT work and resume work after improving the eye fatigue index. In this way, it is possible to prevent symptoms such as eye pain, decreased eyesight, stiff shoulders, and headaches in advance.

The "Guidelines for Occupational Health Management in Information Technology Equipment Work" created by the Ministry of Health, Labor and Welfare states that continuous work time and work rest times should not exceed one hour, and that a 10-15 minute work rest period should be provided between successive tasks. However, currently many people other than VDT workers use devices such as smartphones and tablet terminals for hours every day. It is possible to determine whether or not eyes are fatigued in a short time based on images captured by imaging devices implemented in smartphones, tablets, and notebook personal computers, and further increase the frequency of such determinations, making it possible to encourage such people to prevent symptoms such as eye pain, decreased eyesight, stiff shoulders, and headaches in advance.

In addition, in the eye fatigue evaluation device 10, the calculation unit 12-4 calculates a feature amount representing the shape of the eyes based on any one of information specifying feature points corresponding to the right eye, information specifying feature points corresponding to the left eye, and information specifying feature points corresponding to both eyes. By configuring in this way, the calculation unit 12-4 can calculate any one of a feature amount representing the shape of the right eye, a feature amount representing the shape of the left eye, and a feature amount representing the shapes of the right eye and the left eye.

Furthermore, in the eye fatigue evaluation device 10, the judgment unit 12-5 judges the eye fatigue of the subject S based on a statistic derived based on the distribution of the feature amounts representing the eye shape calculated by the calculation unit 12-4 at a predetermined time. By configuring in this way, the judgment unit 12-5 can derive a statistic as an eye fatigue index based on the distribution of the feature amounts representing the eye shape calculated by the calculation unit 12-4 at a predetermined time, and judge the eye fatigue of the subject S based on the derived statistic.
Furthermore, in the eye fatigue evaluation device 10, the judgment unit 12-5 judges the eye fatigue of the subject S from the probability density distribution of a predetermined interval based on the probability density distribution of the feature amount representing the eye shape. With this configuration, it is possible to derive a statistic as an eye fatigue index from the probability density distribution of a predetermined interval based on the probability density distribution of the feature amount representing the eye shape, and therefore it is possible to judge the eye fatigue of the subject S based on the derived statistic.

Furthermore, in the eye fatigue evaluation device 10, the calculation unit 12-4 calculates the aspect ratio as a feature representing the eye shape based on information identifying the feature points of the eye. With this configuration, the aspect ratio can be calculated as a feature representing the eye shape, and therefore eye fatigue of the subject S can be determined based on the calculated aspect ratio.
The eye fatigue evaluation device 10 further includes a warning unit 12-6 that issues a warning about eye fatigue to the subject S or the manager of the subject S when the degree of eye fatigue determined by the determination unit 12-5 exceeds a predetermined threshold. With this configuration, a warning about the eye fatigue of the subject S can be issued to the subject S or the manager of the subject S.

(Modification of the embodiment)
Hereinafter, an eye fatigue evaluation system according to a modified embodiment will be described with reference to the drawings. The eye fatigue evaluation system 100a according to the modified embodiment can be applied to the system shown in FIG. 1. However, it is different in that an eye fatigue evaluation device 10a is provided instead of the eye fatigue evaluation device 10. The imaging unit 1 and the display device 2 are connected to the eye fatigue evaluation device 10a.
The imaging unit 1 captures an image of a subject S.
The display device 2 displays the image captured by the imaging unit 1 .
The eye fatigue assessment device 10a displays an image captured by the imaging unit 1 on the display device 2. The eye fatigue assessment device 10a displays a gaze point on the display device 2 and has the subject S gaze at the gaze point. This configuration makes it possible to keep the line of sight of the subject S constant. The eye fatigue assessment device 10a evaluates the eye fatigue of the subject S based on the image captured by the imaging unit 1.

In one example of the eye fatigue evaluation system 100a, the imaging unit 1 is fixed on a desk. The subject S sits on a chair and gazes at the gaze point displayed on the display device 2.
The imaging unit 1 creates multiple image data of the subject S by capturing an image of the face of the subject S for a predetermined time. In this embodiment, the explanation will continue with a case where the predetermined time is 60 seconds as an example. The eye fatigue evaluation device 10a acquires multiple image data of the subject S created by the imaging unit 1. If the imaging unit 1 can capture 30 frames per second, 1800 frames can be captured in 60 seconds, so the eye fatigue evaluation device 10a acquires approximately 1800 pieces of image data. The eye fatigue evaluation device 10a records the multiple acquired image data of the subject S.

The eye fatigue evaluation device 10a calls up the recorded image data into a memory and processes the image data called up into the memory to detect a face from each of the images of the subject S. For example, the eye fatigue evaluation device 10a obtains information specifying a rectangle (quadrangle) enclosing the face of the subject S as a result of detecting the face from the image of the subject S.
The eye fatigue evaluation device 10a detects facial feature points of the subject S from each of multiple images of the subject S based on information identifying a rectangle enclosing the face of the subject S obtained from each of multiple images of the subject S.
The eye fatigue evaluation device 10a detects eye feature points of the subject S from each of multiple images of the subject S based on the detection results of the facial feature points of the subject S obtained from each of multiple images of the subject S.
The eye fatigue evaluation device 10a calculates features representing the shape of the subject S's eyes for each of multiple images of the subject S based on the detection results of the feature points of the subject S's eyes obtained from each of multiple images of the subject S.
The eye fatigue evaluation device 10a judges the eye fatigue of the subject S based on feature amounts representing the shape of the eyes of the subject S calculated for each of a plurality of images of the subject S.

The eye fatigue evaluation device 10a will be described in detail below.
The eye fatigue evaluation device 10a includes an input/output I/F 11, a CPU 12, a memory 13, a recording device 14, a data reading unit 15, and a bus 16. The bus 16 connects the input/output I/F 11, the CPU 12, the memory 13, the recording device 14, and the data reading unit 15 to one another.
The input/output I/F 11 is an interface for inputting and outputting information between the image capturing unit 1 and the display device 2 and the eye fatigue evaluation device 10a.

The CPU 12 executes a computer program (software) stored in the recording device 14 to function as a processing unit 12-1, a first detection unit 12-2, a second detection unit 12-3, a calculation unit 12a-4, a judgment unit 12a-5, and a warning unit 12a-6.

The calculation unit 12a-4 acquires the array data created by the second detection unit 12-3. The calculation unit 12a-4 acquires the coordinates of each of the multiple feature points corresponding to the left eye portion and the coordinates of each of the multiple feature points corresponding to the right eye portion included in the acquired array data. The calculation unit 12a-4 calculates a feature amount representing the shape of the left eye and a feature amount representing the shape of the right eye based on the coordinates of each of the multiple feature points corresponding to the left eye portion and the coordinates of each of the multiple feature points corresponding to the right eye portion acquired.
In a modified example of the embodiment, the explanation will continue with the case where the angle of the opening at the inner corner of the eye in the image of the left eye (hereinafter referred to as the "left eye inner corner opening angle") and the angle of the opening at the outer corner of the eye (hereinafter referred to as the "left eye outer corner opening angle") are used as an example of a feature representing the shape of the left eye, and the angle of the opening at the inner corner of the eye in the image of the right eye (hereinafter referred to as the "right eye inner corner opening angle") and the angle of the opening at the outer corner of the eye (hereinafter referred to as the "right eye outer corner opening angle") are used as an example of a feature representing the shape of the right eye.

11 is a diagram showing an example of the process of the eye fatigue assessment device according to the modified embodiment, which shows, as an example, characteristic points P1 to P6 corresponding to the right eye.
As shown in Fig. 11, an example of the characteristic point P1 (corresponding to FP37 in Fig. 2) corresponds to the outer corner of the right eye, an example of the characteristic point P2 (corresponding to FP38 in Fig. 2) corresponds to the intersection of the right boundary between the white and black of the eye and the upper eyelid in the right eye, and an example of the characteristic point P3 (corresponding to FP39 in Fig. 2) corresponds to the intersection of the left boundary between the white and black of the eye and the upper eyelid in the right eye. As shown in Fig. 11, an example of the characteristic point P4 (corresponding to FP40 in Fig. 2) corresponds to the inner corner of the right eye, an example of the characteristic point P5 (corresponding to FP41 in Fig. 2) corresponds to the intersection of the left boundary between the white and black of the eye and the lower eyelid in the right eye, and an example of the characteristic point P6 (corresponding to FP42 in Fig. 2) corresponds to the intersection of the right boundary between the white and black of the eye and the lower eyelid in the right eye.

The calculation unit 12a-4 calculates the right eye inner corner opening angle and the right eye outer corner opening angle based on the coordinates of each of the characteristic points P1 to P6 corresponding to the right eye.
Specifically, the calculation unit 12a-4 calculates the right eye outer corner opening angle (the angle between the line connecting the feature points P1 and P2 and the line connecting the feature points P1 and P6) using the law of cosines for a triangle formed by connecting the feature points P1, P2, and P6 with lines. The calculation unit 12a-4 calculates the right eye inner corner opening angle (the angle between the line connecting the feature points P4 and P3 and the line connecting the feature points P4 and P5) using the law of cosines for a triangle formed by connecting the feature points P3, P4, and P5 with lines. The calculation unit 12a-4 calculates the right opening angle, which is the opening angle of both the right eye inner corner and the right eye outer corner, by performing statistical processing based on the calculated right eye inner corner opening angle and right eye outer corner opening angle. In this embodiment, the explanation will continue with the case where an average value is used as an example of statistical processing.

The calculation unit 12a-4 calculates the left eye inner corner opening angle and the left eye outer corner opening angle based on the coordinates of each of the six feature points corresponding to the left eye. Details of the process in which the calculation unit 12a-4 calculates the left eye inner corner opening angle and the left eye outer corner opening angle are omitted here because the process of calculating the right eye inner corner opening angle and the right eye outer corner opening angle described above can be applied. The calculation unit 12a-4 calculates the left opening angle, which is the opening angle of both the left eye inner corner and the left eye outer corner, by performing statistical processing based on the calculated left eye inner corner opening angle and left eye outer corner opening angle. In this embodiment, the explanation will continue with the case where the average value is used as an example of statistical processing.
The calculation unit 12a-4 estimates the movement (orientation) of the subject S's head with respect to each of the X-axis, Y-axis, and Z-axis using the coordinates of predetermined feature points among the feature points FP01 to FP68 of the human face in order to correct an error caused by a difference in the positional relationship between the subject S and the imaging unit 1 in each of a plurality of images captured of the subject S. The calculation unit 12a-4 records the result of error correction performed on the calculated right opening angle and left opening angle in the recording device 14 based on the estimation result of the movement of the subject S's head with respect to each of the X-axis, Y-axis, and Z-axis.

Fig. 12 is a diagram showing an example 1 of statistical data of the right opening angle calculated by the eye fatigue evaluation device according to the modified embodiment. Fig. 12 shows, as an example, a probability density distribution of the right opening angle acquired every hour from before the start of the VDT work until four hours have passed since the start of the VDT work. In Fig. 12, the horizontal axis represents rad, and the vertical axis represents the probability density at that time.
To obtain statistical data on the right opening angle, the subject S was imaged for 60 seconds. Of the image data obtained during the 60 seconds of imaging, 1000 points of data from 20 seconds after the start of imaging, when eye and head movements are assumed to be relatively stable, were used to calculate a probability density function.

The determination unit 12a-5 acquires information specifying each of the multiple right opening angles calculated by the calculation unit 12a-4. The determination unit 12a-5 calculates an index indicating eye fatigue of the right eye (hereinafter referred to as a "right eye eye fatigue index") based on the acquired information specifying each of the multiple right opening angles.
Specifically, the determination unit 12a-5 calculates the ratio between the peak and the range of the right opening angle corresponding to a predetermined ratio in the probability density distribution of the right opening angle. In this embodiment, the explanation will continue for the case where 95% (probability distribution is from 2.5% to 97.5%) is applied as an example of the predetermined ratio.

When 95% is applied as the specified rate, as shown in equation (4), the judgment unit 12a-5 defines the range of the right opening angle where the probability distribution is 2.5% to 97.5% as xra, the range of the maximum value in the probability density function of the right opening angle from 2.5% to 97.5%, and the maximum value (peak) of the probability density function of the right opening angle of the right eye as yra, and determines the ratio of xra to yra as an index of right eye fatigue.
Specifically, before starting VDT work, the peak of the probability density distribution of the right opening angle is 7.1951, and the range of the right opening angle that corresponds to 95% is 0.29, so the judgment unit 12-5 calculates the right eye fatigue index to be 24.8.
Right eye eye fatigue index=(value at which the probability density function of the right eye's right opening angle is maximized (yra value))/(range of 2.5% or more in the probability distribution of the right opening angle (xra value)) (4)
FIG. 12 also shows the change over time in the right eye fatigue index calculated based on the probability density distribution of the right opening angle acquired every hour.

According to FIG. 12, it can be seen that the maximum value (peak) (yra value) of the probability density function at the right opening angle is stable and large at a certain right opening angle before the start of VDT work (when eye fatigue is small) compared to after the start of VDT work. In other words, it can be seen that as time passes during VDT work, the maximum value of the probability density function at the right opening angle becomes lower, resulting in a broader probability density distribution (probability density at the mode decreases). In other words, the degree of fatigue of the right eye can be determined based on the maximum value of the probability density function at the right opening angle.

Fig. 13 is a diagram showing an example 2 of statistical data of the left opening angle calculated by the eye fatigue evaluation device according to the modified embodiment. Fig. 13 shows, as an example, a probability density distribution of the left opening angle acquired every hour from before the start of VDT work until four hours have passed since the start of VDT work. In Fig. 13, the horizontal axis represents rad, and the vertical axis represents the probability density at that time.
To obtain statistical data on the left opening angle, the subject S was imaged for 60 seconds. Of the image data obtained during the 60 seconds of imaging, 1000 points of data from 20 seconds after the start of imaging, when eye and head movements are assumed to be relatively stable, were used to calculate a probability density function.

The determination unit 12a-5 acquires information specifying each of the plurality of left opening angles calculated by the calculation unit 12a-4. The determination unit 12a-5 calculates an index indicating eye fatigue of the left eye (hereinafter referred to as a "left eye eye fatigue index") based on the acquired information specifying each of the plurality of left opening angles.
Specifically, the determination unit 12a-5 calculates the ratio between the peak and the range of the left opening angle corresponding to a predetermined ratio in the probability density distribution of the left opening angle. In this embodiment, the explanation will continue for the case where 95% (probability distribution is from 2.5% to 97.5%) is applied as an example of the predetermined ratio.

When 95% is applied as the specified rate, as shown in equation (5), the judgment unit 12a-5 defines the range of the left opening angle where the probability distribution is 2.5% to 97.5% as xla, the range of the maximum value in the probability density function of the left opening angle from 2.5% to 97.5%, and the maximum value (peak) of the probability density function of the left opening angle of the left eye as yla, and determines the ratio of xla to yla as an index of left eye fatigue.
Specifically, before starting VDT work, the peak of the probability density distribution of the left opening angle is 6.9738, and the range of the left opening angle that corresponds to 95% is 0.31, so the judgment unit 12a-5 calculates the left eye fatigue index to be 22.5.
Left eye fatigue index=(value at which the probability density function of the left eye's left opening angle is maximized (yla value))/(range of 2.5% or more in the probability distribution of the left opening angle (xla value)) (5)
FIG. 13 also shows the change over time in the left eye fatigue index calculated based on the probability density distribution of the left opening angle acquired every hour.

According to FIG. 13, it can be seen that the maximum value (peak) (yla value) of the probability density function at the left opening angle is stable and large at a certain left opening angle before the start of VDT work (when eye fatigue is small) compared to after the start of VDT work. In other words, it can be seen that as time passes during VDT work, the maximum value of the probability density function at the left opening angle decreases, resulting in a broader probability density distribution (probability density at the mode decreases). In other words, the degree of fatigue of the left eye can be determined based on the maximum value of the probability density function at the left opening angle.

FIG. 14 is a diagram showing an example of intervals, probability densities, and eye fatigue indexes of opening angles calculated by the eye fatigue evaluation device according to the modified example of the embodiment.
14, the right eye fatigue index is 24.83 before the start of VDT work, 16.22 one hour after the start of VDT work, 13.38 two hours after the start of VDT work, 11.18 three hours after the start of VDT work, and 9.38 four hours after the start of VDT work. It can be seen that the right eye fatigue index decreases exponentially according to the time that has passed since the start of VDT work.

The left eye fatigue index was 22.58 before the start of VDT work, 16.94 one hour after the start of VDT work, 11.74 two hours after the start of VDT work, 8.35 three hours after the start of VDT work, and 7.95 four hours after the start of VDT work. It can be seen that the left eye fatigue index decreases according to the time that has elapsed since the start of VDT work.

The decrease in the right eye fatigue index and the left eye fatigue index is assumed to be due to eye fatigue caused by long-term use of VDT devices such as smartphones and computers. When concentrating on staring at a screen and the eyeballs are not moving for a long time, muscle fatigue occurs in the seven muscles that move the eyeballs and the muscles that adjust the thickness of the "crystalline lens" to focus. In addition, the number of times the eyes blink is often reduced during VDT work. Furthermore, the decrease in the right eye fatigue index and the left eye fatigue index is also thought to be due to dry eyes, which make tears more likely to evaporate when wearing contact lenses or in an environment exposed to air conditioning. Due to these causes, the probability of the opening angle becoming smaller increases, and the xra value in the denominator of the right eye fatigue index and the xla value in the denominator of the left eye fatigue index increase, causing the right eye fatigue index and the left eye fatigue index, which indicate increased eye fatigue, to decrease.

The determination unit 12a-5 also performs statistical processing based on the right eye fatigue index and the left eye fatigue index to calculate an eye fatigue index that is an index of eye fatigue for both the right eye and the left eye combined. In this embodiment, the explanation will continue with the case where an average value is used as an example of statistical processing.
14, the fatigue index (average of left and right eyes) is 23.7 before the start of VDT work, 16.58 one hour after the start of VDT work, 12.56 two hours after the start of VDT work, 9.76 three hours after the start of VDT work, and 8.67 four hours after the start of VDT work. It can be seen that the fatigue index decreases with the time elapsed since the start of VDT work, just like the right eye fatigue index and left eye fatigue index.

15 is a diagram showing an example of a change over time in the eye fatigue index calculated by the eye fatigue assessment device according to the modified example of the embodiment, in which the horizontal axis represents the elapsed time from the start of the VDT work, and the vertical axis represents the eye fatigue index.
According to FIG. 15, it can be seen that the eye fatigue index decreases by about 25% after one hour has elapsed since the start of the VDT work. Therefore, when the eye fatigue index decreases by 25% or more from the initial value, it is preferable that the eye fatigue evaluation device 10a has the subject S stop the VDT work and take a rest for a certain period of time. Before resuming the work, the eye fatigue evaluation device 10a calculates the right eye fatigue index and the left eye fatigue index of the subject S again, and judges whether the fatigue index calculated thereafter has decreased by 25% or more based on the calculated right eye fatigue index and the left eye fatigue index. The eye fatigue evaluation device 10a performs the same process when the fatigue index calculated thereafter has decreased by 25% or more. It is assumed that by configuring in this way, the subject S can prevent symptoms such as eye pain, decreased eyesight, stiff shoulders, and headaches.

The warning unit 12a-6 acquires the eye fatigue index calculated by the determination unit 12a-5. The warning unit 12a-6 judges whether the acquired eye fatigue index is less than the eye fatigue index threshold. An example of the eye fatigue index threshold is a value that is 25% lower than the initial value of the eye fatigue index. The warning unit 12a-6 issues a warning when the eye fatigue index is less than the eye fatigue index threshold. Specifically, the warning unit 12a-6 issues a warning to the subject S or the administrator of the subject S.

(Operation of eye fatigue evaluation system 100a)
FIG. 16 is a flowchart showing an example of the operation of the eye fatigue evaluation system according to the modified example of the embodiment.
Steps S1-2 to S5-2 can be implemented by steps S1-1 to S5-1 in FIG. 9, so a description thereof will be omitted here.
(S6-2)
In the eye fatigue evaluation device 10a, a calculation unit 12a-4 estimates the movement (orientation) of the head of a subject S with respect to each of the X-axis, Y-axis, and Z-axis by using the coordinates of predetermined feature points among feature points FP01 to FP68 on a human face.
(S7-2)
In the eye fatigue evaluation device 10a, a calculation unit 12a-4 calculates the right eye inner corner opening angle and the right eye outer corner opening angle based on the coordinates of each of characteristic points P1 to P6 corresponding to the right eye.

(S8-2)
In the eye fatigue evaluation device 10a, the calculation unit 12a-4 estimates the movement (orientation) of the subject S's head along each of the X-axis, Y-axis, and Z-axis by using the coordinates of predetermined feature points among the feature points FP01 to FP68 of a person's face. The calculation unit 12a-4 records the results of error correction performed on the calculated right opening angle and left opening angle based on the estimated results of the movement of the subject S's head along each of the X-axis, Y-axis, and Z-axis in the recording device 14.
(S9-2)
In the eye fatigue evaluation device 10a, the determination unit 12a-5 calculates a right eye eye fatigue index based on the probability density distribution of the right eye inner corner opening angle. The calculation unit 12a-4 calculates a left eye eye fatigue index based on the probability density distribution of the left eye inner corner opening angle. The determination unit 12a-5 calculates the eye fatigue index by performing statistical processing based on the calculated right eye eye fatigue index and left eye eye fatigue index.
(S10-2)
In the eye fatigue evaluation device 10a, a judgment unit 12a-5 judges the degree of eye fatigue on the basis of the calculated eye fatigue index.
16, the order of steps S6-2 and S7-2 may be interchanged, i.e., steps S1-2 to S5-2, S7-2, S6-2, S8-2 to S10-2 may be executed in this order.

In a modified example of the embodiment described above, the determination unit 12a-5 calculates a right eye eye fatigue index based on the probability density distribution of the right eye opening angle, and calculates a left eye eye fatigue index based on the probability density distribution of the left eye opening angle. The determination unit 12a-5 has calculated the eye fatigue index by performing statistical processing based on the calculated right eye eye fatigue index and left eye eye fatigue index, but this example is not limited to the above. For example, the determination unit 12a-5 may calculate a right eye eye fatigue index based on the probability density distribution of the right eye opening angle, and calculate the eye fatigue index by performing statistical processing based on the calculated right eye eye fatigue index. For example, the determination unit 12a-5 may calculate a left eye eye fatigue index based on the probability density distribution of the left eye opening angle, and calculate the eye fatigue index by performing statistical processing based on the calculated left eye eye fatigue index.

In the modified example of the embodiment described above, the calculation unit 12a-4 calculates the right eye outer corner opening angle and the right eye inner corner opening angle, and calculates the right opening angle based on the calculated right eye outer corner opening angle and right eye inner corner opening angle, but this is not limited to the example. For example, the calculation unit 12a-4 may calculate the right eye outer corner opening angle and calculate the right opening angle based on the calculated right eye outer corner opening angle, or may calculate the right eye inner corner opening angle and calculate the right opening angle based on the calculated right eye inner corner opening angle.
In the modified example of the embodiment described above, the calculation unit 12a-4 calculates the left eye outer corner opening angle and the left eye inner corner opening angle, and calculates the left opening angle based on the calculated left eye outer corner opening angle and left eye inner corner opening angle, but this is not limited to the example. For example, the calculation unit 12a-4 may calculate the left eye outer corner opening angle and calculate the left opening angle based on the calculated left eye outer corner opening angle, or may calculate the left eye inner corner opening angle and calculate the left opening angle based on the calculated left eye inner corner opening angle.

In the modified example of the embodiment described above, the warning unit 12a-6 acquires the eye fatigue index calculated by the determination unit 12a-5, and issues a warning if the acquired eye fatigue index is less than the eye fatigue index threshold value, but this is not limited to the example. For example, the warning unit 12a-6 may determine whether the acquired eye fatigue index corresponds to any of the attention levels based on a plurality of eye fatigue index threshold values. Furthermore, the warning unit 12a-6 may display the determination result of whether the acquired eye fatigue index corresponds to any of the attention levels on the display device 2.

Figure 17 is a diagram showing an example of the display of the evaluation result of the eye fatigue evaluation system according to the modified embodiment. In the example shown in Figure 17, the judgment result of the eye fatigue index is displayed using a bar chart. Specifically, 16.5, 12.6, 9.8, and 8.7 are set as examples of eye fatigue index thresholds. In the bar chart, the following ranges are set: a level corresponding to no attention where the eye fatigue index is 16.5 or more, a caution level 1 where the eye fatigue index is 12.6 or more and less than 16.5, a caution level 2 where the eye fatigue index is 9.8 or more and less than 12.6, a caution level 3 where the eye fatigue index is 8.7 or more and less than 9.8, and a caution level 4 where the eye fatigue index is less than 8.7, and each range is displayed in a different color. The warning unit 12a-6 displays the eye fatigue index as a character. By configuring in this way, the subject S who refers to the judgment result of the eye fatigue index can know the degree of his or her eye fatigue state. Here, as an example, a case where the number of eye fatigue index thresholds is four has been described, but the number of eye fatigue index thresholds may be two or three, or may be five or more.

According to the eye fatigue evaluation system 100a according to the modified embodiment, in the eye fatigue evaluation device 10 according to the embodiment described above, the calculation unit 12a-4 calculates either or both of the angle of the opening at the inner corner of the eye and the angle of the opening at the outer corner of the eye as a feature representing the shape of the eye based on information identifying the feature points of the eye. By configuring in this way, either or both of the angle of the opening at the inner corner of the eye and the angle of the opening at the outer corner of the eye can be calculated as a feature representing the shape of the eye, so that the eye fatigue of the subject S can be determined based on either or both of the calculated angle of the opening at the inner corner of the eye and the angle of the opening at the outer corner of the eye.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the invention.

1...imaging unit, 2...display device, 10...eye fatigue evaluation device, 11...input/output I/F, 12...CPU, 12-1...processing unit, 12-2...first detection unit, 12-3...second detection unit, 12-4, 12a-4...calculation unit, 12-5, 12a-5...determination unit, 12-6, 12a-6...warning unit, 13...memory, 14...recording device, 15...data reading unit

Claims (8)

An eye fatigue evaluation device for evaluating eye fatigue of a subject, comprising:
a calculation unit that calculates a feature amount representing an eye shape based on information identifying eye feature points obtained from an image of the face of the subject;
a determination unit that determines eye fatigue of the subject based on the feature amount calculated by the calculation unit ,
The determination unit is an eye fatigue evaluation device that determines eye fatigue of the subject based on a statistical quantity derived based on the distribution of features representing the eye shape calculated by the calculation unit at a predetermined time . The eye fatigue assessment device according to claim 1, wherein the calculation unit calculates the feature quantity representing the shape of the eye based on any one of information specifying feature points corresponding to the right eye, information specifying feature points corresponding to the left eye, and information specifying feature points corresponding to both eyes. The eye fatigue evaluation device according to claim 1 , wherein the determining unit determines eye fatigue of the subject from a probability density distribution of a predetermined section based on a probability density distribution of a feature amount representing an eye shape. The eye fatigue evaluation device according to claim 1 , wherein the calculation unit calculates an aspect ratio as a feature quantity representing a shape of the eye based on the information identifying feature points of the eye. 4. The eye fatigue evaluation device according to claim 1, wherein the calculation unit calculates either or both of an angle of an opening at the inner corner of the eye and an angle of an opening at the outer corner of the eye as features representing an eye shape based on the information identifying feature points of the eye . The eye fatigue evaluation device according to claim 1 , further comprising: a warning unit that issues a warning regarding eye fatigue to the subject or a manager of the subject when the degree of eye fatigue determined by the determination unit exceeds a predetermined threshold. A computer-implemented method for assessing eye fatigue, comprising:
calculating a feature quantity representing an eye shape based on information identifying eye feature points obtained from an image of the subject's face;
and determining eye fatigue of the subject based on the feature amount calculated in the calculating step ,
In the determining step, the eye fatigue of the subject is determined based on a statistical quantity derived based on the distribution of features representing the eye shape calculated in the calculating step at a predetermined time . On the computer,
calculating a feature quantity representing an eye shape based on information identifying eye feature points obtained from an image of the subject's face;
and determining eye fatigue of the subject based on the feature amount calculated in the calculating step .
In the determining step, the program determines eye fatigue of the subject based on a statistical quantity derived based on the distribution of features representing the eye shape calculated in the calculating step at a predetermined time .

Images