JP7631780B2 - Eye movement measuring device, eye movement measuring method and program - Google Patents

Description

The present invention relates to an eye movement measuring device, an eye movement measuring method, and a program for detecting the eye movement of a subject.

It is generally known that humans observe an object space by constantly moving their eyes. That is, the human eye obtains visual information by repeating fixation, in which the gaze is fixed on the object being observed, and saccades, in which the gaze is moved quickly (approximately 100 degrees (deg)/500 degrees/second) for a short period of time (approximately 20 ms (milliseconds) to 60 ms).

However, even during fixation, the eyeballs actually make so-called fixational eye movements (hereinafter referred to as microsaccades).
This microsaccade is a type of fixational eye movement, and is a minute jump movement with an amplitude of 1 degree or less in terms of the rotation angle of the eyeball and a duration of several tens of milliseconds or less.

There are many research reports suggesting that visual attention is strongly reflected in the control system of microsaccades.
Many research reports have shown that the frequency of microsaccades increases with a shift in visual attention, whereas the frequency of microsaccades decreases with the concentration of visual attention.

Therefore, by analyzing the frequency of microsaccades (see, for example, Non-Patent Documents 1 and 2), it is possible to estimate a person's internal state, such as determining whether they are concentrating their visual attention on the object they are looking at, or whether they are simply looking absentmindedly (without concentrating their attention).

Masanori Takahashi, Hirohisa Isogai, Judy L. Van Raalte, "An Attempt to Detect Microsaccades from Eye Movements in Anticipatory Reaction Situations - Using an Anticipatory Reaction Task for Tennis Serves -", Sports Industry Research, Vol. 28, No. 1 (2018), pp. 13-29 Kazutaka Suzuki, Haruyoshi Toyoda, Ryohei Hanayama, Masahiro Ishii, "Development of a simultaneous binocular fixation micromovement measurement device using an intelligent vision sensor and evaluation of the left-right difference in microsaccades," Journal of Biomedical Engineering, 53(5), 247-254, 2015

As described above, by detecting microsaccades, it is possible to estimate the degree of a person's potential attention to an object to which the gaze is directed.
However, when trying to detect microsaccades, it may be necessary to detect eye movements with an eye rotation angle of 0.1 degree or less and a duration of 0.01 seconds or less, and gaze measurement devices designed to detect general saccades are not able to measure such minute and rapid eye movements.

Therefore, when attempting to detect microsaccades, it is necessary to improve the measurement accuracy and measurement speed of the eye rotation angle.
In conventional gaze measurement, the position of the pupil must be determined accurately to estimate the direction of gaze. However, when trying to improve measurement accuracy, the mechanical structure tends to become more complex and the measurement equipment tends to become larger and larger, assuming it is placed in a fixed location, in order to increase the accuracy.
Furthermore, in order to improve the measurement accuracy, it is necessary to fix the subject's head so that the eye position does not fluctuate significantly.

Therefore, the gaze measurement devices in Non-Patent Document 1 and Non-Patent Document 2 measure microsaccades by fixing the subject's head in a large measuring instrument, and therefore cannot measure microsaccades when the subject is walking freely indoors and outdoors in their daily lives.
For example, it is not possible to measure microsaccades in a situation where a subject is moving between shelves in a store and looking at products.

The present invention was made in consideration of these circumstances, and provides an eye movement measuring device, an eye movement measuring method, and a program that can be worn by a subject and can measure eye movement, including microsaccades, in the subject's everyday state, i.e., when moving around or moving the head in daily life.

In order to solve the above-mentioned problems, the eye movement measuring device of the present invention is an eye movement measuring device that detects a position change of a focus point of the eye in a moving image of the eye that is captured including the eye, and is equipped with a measurement range setting unit that sets an area of the moving image that satisfies predetermined conditions as a measurement range, a luminance gradient calculation unit that calculates a luminance gradient in the measurement range, a luminance change amount calculation unit that calculates an amount of luminance change between frame images of the measurement range, and a movement distance calculation unit that calculates the position change using a gradient method from each of the luminance change amount and the luminance gradient, and is characterized in that the measurement range setting unit selects pixels within the measurement range that have a luminance value less than a first threshold value for reducing the effect of shot noise in the moving image as the measurement range .
The eye movement measuring device of the present invention is characterized in that the measurement range setting unit sets an area including the boundary between the pupil and the iris of the eye as the measurement range.

The eye movement measuring device of the present invention is characterized in that the moving images are captured at a frame cycle in which the change in pupil position between the frame images is within a movement range that falls within the measurement range in which the luminance gradient has a constant gradient.

The eye movement measuring device of the present invention is characterized in that the measurement range setting unit compares the brightness value of each pixel in the frame image with a brightness threshold, extracts a pupil image area that is the image area of the pupil, and sets an area obtained by enlarging the pupil image area from the center point of the pupil image area in the outer circumferential direction as the measurement range.

The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects, as the measurement range, a group of pixels in an area within the measurement range where the luminance gradient is higher than other areas.

The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects pixels with lower luminance values in the luminance gradient as pixels in the measurement range.

The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects pixels in the measurement range based on both the luminance gradient and the luminance value of the pixel within the measurement range.

The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects pixels in the measurement range by comparing a numerical value obtained by dividing the brightness gradient by the amount of shot noise estimated from the brightness value of the pixel within the measurement range with a second threshold value.

The eye movement measuring device of the present invention is characterized in that within the measurement range, pixels for which the value obtained by dividing the luminance gradient by the square root of the luminance value of the pixel is greater than a second threshold value are defined as pixels of the measurement range.

The eye movement measuring device of the present invention is characterized in that the measurement range setting unit determines an x-axis luminance gradient as the luminance gradient in the x-axis direction and a y-axis luminance gradient as the luminance gradient in the y-axis direction corresponding to the measurement range, and determines an x-axis movement amount, which is the movement amount of the pupil in the x-axis direction, and a y-axis movement amount, which is the movement amount in the y-axis direction, by formulating simultaneous linear equations from the luminance change amount, the x-axis luminance gradient, and the y-axis luminance gradient.

In the eye movement measuring device of the present invention, the measurement range setting unit compares a first absolute value, which is the absolute value of the luminance gradient in the x-axis direction, of each pixel in the measurement range, with a second absolute value, which is the absolute value of the luminance gradient in the y-axis direction, and sets pixels in the measurement range in which the first absolute value is greater than the second absolute value as a first group, and pixels in the measurement range in which the second absolute value is greater than the first absolute value as a second group, and calculates a value obtained by adding the x-axis luminance gradients of each pixel in the first group and multiplying the result by a variable amount of movement in the x-axis direction, and a value obtained by adding the y-axis luminance gradients and multiplying the result by a variable amount of movement in the y-axis direction. one equation of the simultaneous linear equations is generated in which the sum of the x-axis luminance gradients of each of the pixels in the second group is an integrated value of the luminance change amount for each of the pixels, and the other equation of the simultaneous linear equations is generated in which the sum of the x-axis luminance gradients of each of the pixels in the second group is multiplied by the x-axis movement amount, which is a variable, and the sum of the y-axis luminance gradients is multiplied by the y-axis movement amount, which is a variable, is an integrated value of the luminance change amount for each of the measurement ranges, and the x-axis movement amount and the y-axis movement amount are each calculated by calculating the simultaneous linear equations.

The eye movement measuring device of the present invention is characterized in that it irradiates the eye with infrared light and captures the light reflected from the eye in multiple tones to obtain the moving image.

The eye movement measuring method of the present invention is an eye movement measuring method that detects a position change of a focus point of the eye in a moving image of the eye that is captured including the eye, and is characterized in including a measurement range setting process in which a measurement range setting unit sets an area of the moving image that satisfies predetermined conditions as a measurement range, and selects, within the measurement range, pixels having a luminance value less than a first threshold value for reducing the effect of shot noise in the moving image as the measurement range; a luminance gradient calculation process in which a luminance gradient is calculated in the measurement range by a luminance gradient calculation unit; a luminance change amount calculation process in which a luminance change amount calculation unit calculates an amount of luminance change between frame images in the measurement range; and a movement distance calculation process in which a gradient method is used to calculate the position change using each of the luminance change amount and the luminance gradient.

The program of the present invention is a program that causes a computer to execute the operation of an eye movement measuring device that detects a change in position of a focus point of an eyeball in a moving image of an eye that is captured including the eyeball, and causes the computer to function as a measurement range setting means that sets an area of the moving image that satisfies predetermined conditions as a measurement range, and selects, within the measurement range, pixels having a luminance value less than a first threshold value for reducing the effect of shot noise in the moving image as the measurement range , a luminance gradient calculation means that calculates a luminance gradient in the measurement range, a luminance change amount calculation means that calculates the amount of luminance change between frame images in the measurement range, and a movement distance calculation means that calculates the position change using a gradient method based on each of the luminance change amount and the luminance gradient.

As described above, the present invention provides an eye movement measuring device, an eye movement measuring method, and a program that can be worn by a subject and can measure eye microsaccades while the subject is in an everyday state, i.e., moving around and moving their head in daily life.

1 is a conceptual diagram showing an example of the configuration of a measurement system using an eye movement measurement device of the present embodiment. 10 is a diagram showing a frame image and luminance values of pixels in an imaging region in which each part of the eye is imaged in the frame image. FIG. 5 is a conceptual diagram of an eye for explaining a process of setting a measurement range from an imaging region of a pupil 501 in a frame image. FIG. 11A and 11B are diagrams illustrating calculation of a luminance gradient in an area where a candidate pixel exists; FIG. 13 is a diagram illustrating an example of calculation of a luminance gradient in an image frame. 6A and 6B are conceptual diagrams illustrating a process of calculating the amount of pupil movement by a gradient method in the present embodiment. 13 is a diagram showing the relationship between pixel positions and pixel luminance values between frame images when the amount of movement Δx is large. FIG. 1 is a conceptual diagram for explaining derivation of a movement amount in hopping movement of a pupil on an imaging surface. 11 is a diagram illustrating the linearity of the luminance gradient in the boundary region between the pupil and the iris of the eyeball in the frame image. FIG. 2B is a diagram showing the luminance gradient in the boundary region of each of the pupil 501 and the iris 502 in FIG. FIG. 7 is a diagram showing a frame image pattern obtained by two-dimensionally expanding the pattern shown in FIG. 6( a ). 11 is a conceptual diagram illustrating a process of generating simultaneous linear equations from a plurality of measurement pixels. FIG. 10 is a flowchart illustrating an example of the operation of a measurement process in the eye movement measurement device according to the present embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
1 is a conceptual diagram showing an example of the configuration of a measurement system using an eye movement measurement device according to the present embodiment. In FIG. 1, the measurement system 1 includes an eye movement measurement device 10, an infrared irradiation device 20, an imaging lens 30, an infrared filter 40, and an image sensor 50.
The eye movement measurement device 10 detects a change in the position of the pupil, which is a point of interest in the eye, from a moving image obtained by capturing input human eye movement.

The infrared irradiation device 20 includes an LED (Light Emitting Diode) that emits infrared rays to illuminate the eye area including the eyeball of the subject. In this embodiment, the LED is a device that emits invisible infrared light (infrared rays) with a wavelength of, for example, 770 nm.
Here, when visible light is used for illumination, the subject may react to the light of the illumination, i.e., the iris may close to reduce the amount of light entering the pupil, causing the size of the pupil to change, and the position of the pupil may change to avoid the light. Or, the subject may be drawn to the light of the illumination, making it impossible to measure eye movement that occurs in a natural state corresponding to the object being observed, which is used to estimate the psychological state of the subject when observing the object. For this reason, in this embodiment, infrared light with an invisible wavelength is used to illuminate the eye in order to measure eye movement without being affected by the irradiated light.

The imaging lens 30 is a small imaging lens for capturing an image of the eyeball from a close distance, and forms an image of reflected light (reflected infrared light) from the subject eyeball on the imaging surface of the image sensor 50, which will be described later.

The infrared filter 40 is an optical filter that transmits infrared light of invisible wavelengths and blocks light of visible wavelengths. In this embodiment, for example, an optical filter with a cutoff wavelength of 700 nm is used.

The image sensor 50 is a device that converts the brightness of light irradiated onto an imaging surface into an electrical signal and outputs it as image data, and is, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.
The image sensor 50 outputs frame images, which are images captured at a predetermined frame period (the frame period in this embodiment will be described in detail later), as a continuous moving image to the eye movement measurement device 10. Here, each of the frame images in the moving image is captured in multiple gradations.

The eye movement measuring device 10 includes a data input/output unit 11 , a measurement range setting unit 12 , a luminance gradient calculation unit 13 , a luminance change amount calculation unit 14 , a movement distance calculation unit 15 , a display unit 16 , and a memory unit 17 .
The data input/output unit 11 writes the captured image supplied from the image sensor 50 into the storage unit 17 for storage.
In addition, the data input/output unit 11 outputs the pupil movement amount (the movement amount when the eyeball rotates, which is the x-axis movement amount Δx and the y-axis movement amount Δy between frames in successive frame periods described later) measured by the eye movement measurement device 10 to an external device.

The measurement range setting unit 12 sets the measurement range as an image region for extracting measurement pixels used to calculate the movement amount of the pupil in a frame image that is a captured image. In this embodiment, a method is used for measuring the movement amount of the eyeball at the point of interest using a gradient method that measures the movement amount using a luminance gradient and a luminance change amount.
Therefore, in order to improve the accuracy of the measured amount of movement by using an area with a large luminance gradient to measure a large amount of luminance change even if the amount of movement is small, the boundary area of the pupil and iris, which has a higher luminance gradient than other parts of the eyeball, is used as the measurement target, as will be described later.

2A and 2B are diagrams showing frame images and luminance values of pixels in an image area in which each part of the eye is captured in the frame image. Fig. 2A shows a frame image including an image area of a pupil 501, an iris 502, and a conjunctiva 503 of an eyeball 500.
Moreover, a boundary region 505 indicates an image region near the boundary between the pupil 501 and the iris 502. A reflection region 506 is an image region in which a reflected image of the irradiated infrared light source is captured, and has a higher luminance value than the surrounding image regions.

Fig. 2(b) is a graph showing the luminance values of each pixel on the line segment A-A' in the frame image of Fig. 2(a). In Fig. 2(b), the horizontal axis shows the x-coordinate of the pixel (pixel position), and the vertical axis shows the luminance value of each pixel.
A dotted line B1 indicates the luminance value of the pixel in the imaging region of the sclera 503 of the eye.
A dotted line B2 indicates the luminance values of pixels in an imaging region of a boundary region 505 between the pupil 501 and the iris 502.

Each of the dotted lines B3 and B4 indicates the luminance value of a pixel in the imaging area of the reflective area 506.
A dotted line B5 indicates the luminance value of the pixel in the imaging region of the boundary between the iris 502 and the sclera 503 of the eye.
In this embodiment, since the boundary area between the pupil and the iris of the eye has a large luminance gradient compared to other areas of the eye, the gradient method is used to calculate the movement amount of the pupil. For this reason, the imaging area of the pupil is extracted from the frame image in which the eye is imaged.

Here, when detecting the imaging region in which the pupil 501 is imaged, the measurement range setting unit 12 compares the luminance value of each pixel in the frame image with the pupil detection threshold value.
Then, the measurement range setting unit 12 extracts a collection of pixels having a luminance value less than the pupil detection threshold (dotted line 10 in FIG. 2B) as an imaging area in which the pupil 501 is imaged. The imaging area of the pupil 501 has the lowest luminance value in the frame image, and therefore can be easily distinguished from the imaging areas of other parts.

In this embodiment, the pupil detection threshold value is an intermediate value between the luminance value of the pixel where the pupil is imaged and the luminance value of the pixel where the iris is imaged.
Furthermore, the pupil detection threshold may be determined by experimentally capturing images of a number of human eyes and detecting whether the pupil luminance value is distinguishable from other surrounding areas.
In this embodiment, pixels near a boundary region 505 between the pupil 501 and the iris 502, which is the range with a large luminance gradient (the region indicated by dotted line B2) shown in FIG. 2A, are used as measurement pixels (described later) used to measure (estimate) the amount of movement of the pupil 501 by the gradient method.

FIG. 3 is a conceptual diagram of an eye for explaining the process of setting the measurement range from the imaging region of the pupil 501 in the frame image.
After extracting the imaging area of the pupil 501 described in FIG. 2, i.e., the inner portion of the boundary area 505, the measurement range setting unit 12 extracts the central pixel (center O) of the pupil 501 area, and sets the number of pixels from the central pixel to the boundary area 505 as the radius R of the pupil 501 area.
Then, the measurement range setting unit 12 multiplies the radius R by a predetermined coefficient value p (p>1, for example, 1.5 in this embodiment) and sets the range of radius R′ as the measurement range (processing target area) 600 .

That is, the measurement range setting unit 12 determines a measurement range 600 that necessarily includes the boundary area 505 between the pupil 501 and the iris 502 as a region for extracting measurement pixels for measuring the amount of movement of the pupil (i.e., the amount of movement of the eyeball).
This prevents the frame image of the subject's eye from including images of unnecessary parts that could cause errors, such as glasses or eyebrows, other than the eyeball, and determines the measurement range 600 as the area where the boundary area of the pupil and iris is likely to be captured.

Returning to FIG. 1, the luminance gradient calculation unit 13 determines whether or not each pixel in the frame image is within the measurement range 600 described above.
Then, the luminance gradient calculation unit 13 determines whether or not the luminance value of each pixel within the measurement range 600 is less than a preset luminance threshold value (first threshold value).

As a result, in a frame image capturing an image of the subject's eye, unnecessary pixels that have large luminance values due to reflections of light sources in the captured portion of the eyeball and that may cause errors in detecting the amount of movement are excluded from candidate pixels to be extracted as measurement pixels.
Then, the luminance gradient calculation unit 13 determines pixels within the measurement range 600 whose luminance values are less than the above luminance value as candidate pixels, which are candidates for measurement pixels.
Next, the luminance gradient calculation unit 13 calculates the luminance gradient at each of the extracted candidate pixels using each of eight adjacent pixels adjacent to the measurement pixel for which the luminance gradient is to be calculated, as explained below with reference to FIG.

FIG. 4 is a diagram for explaining the calculation of the luminance gradient in an area where a candidate pixel exists.
The luminance gradient of the region 700 to which the candidate pixel C _i,j belongs is calculated by the luminance gradient calculation unit 13 as an x-axis direction luminance gradient sx and a y-axis direction luminance gradient sy.
In the following description, the luminance values of pixels C _i-1,j-1 to C _i+1,j+1 are respectively indicated as luminance values LC _i-1,j-1 to C _i+1,j+1 . Here, the x-axis luminance gradient sx and the y-axis luminance gradient sy of pixel C _i,j are calculated using eight adjacent pixels C _{i-1,j-1 , C i-1} ,j , C _i-1,j+ 1 , C _{i,j-1 , C i,j+1} , C i+ _{1 ,j-1} , C _i+1,j , C _i+1,j+1 (these pixels form a pixel area including the measurement pixel) in the upper, lower, left, right and diagonal directions.

The luminance gradient calculation unit 13 calculates the x-axis direction luminance gradient sx by the following formula.
sx=[(LC _i-1,j-1 -LC _i+1,j+1 )/2+(LC _i,j-1 -LC _i,j+1 )/2+(LC _i+1,j-1 -LC _i+1,j+1 )/2]/3
That is, the x-axis direction luminance gradient sx is calculated from the average value of a luminance gradient obtained by subtracting the luminance value LC i _{+1,j +1} of pixel C _i+1,j+1 from the luminance value LC _i-1,j-1 of pixel C _i-1,j-1 and dividing the result by two pixels, a luminance gradient obtained by subtracting the luminance value LC _{i,j +1 of pixel C i,j-1} from the luminance value LC _i,j-1 of pixel C _i,j+1 and dividing the result by two pixels, and a luminance gradient obtained by subtracting the luminance value LC _i+1,j+1 of pixel C _{i+1,j-1 from the luminance value LC i+1,j-1} of pixel C _i+1,j-1 and dividing the result by two pixels.

Similarly, the luminance gradient calculation unit 13 calculates the y-axis direction luminance gradient sy by the following formula.
sy=[(LC _i-1,j-1 -LC _i+1,j-1 )/2+(LC _i-1,j -LC _i+1,j )/2+(LC _i-1,j+1 -LC _i+1,j+1 )/2]/3
That is, the y-axis direction luminance gradient y is calculated as the average value of a luminance gradient obtained by subtracting the luminance value LC i _{+1,j -1} of pixel C _i+1,j-1 from the luminance value LC _i-1,j-1 of pixel C _i-1,j-1 and dividing the result by two pixels, a luminance gradient obtained by subtracting the luminance value LC _i+1,j of pixel C _i+1,j from the luminance value LC _i-1,j of pixel C _i-1,j and dividing the result by two pixels, and a luminance gradient obtained by subtracting the luminance value LC _i+1,j+1 of pixel C _{i-1,j+1 from the luminance value LC i-1,j+} 1 of pixel C _i-1,j+1 and dividing the result by two pixels.

5 is a diagram showing an example of calculation of the luminance gradient in an image frame, in which the horizontal axis indicates the coordinate position (pixel position) of a pixel, and the vertical axis indicates the luminance value of the pixel.
The luminance gradient s _n-1 of pixel position x _n-1 is 25 (=(125-75)/ ₂₌ 50/2), obtained by subtracting the luminance value 75 of the pixel at pixel position x _n-2 from the luminance value 125 of the pixel at pixel position x n-2, and dividing the result by the distance of two pixels.
Furthermore, the luminance gradient s _{n+ 1 at pixel position x n+1} is 25 (=(75-25)/ ₂ =50/2), obtained by subtracting the luminance value 25 of the pixel at pixel position x _n+2 from the luminance value 75 of the pixel at pixel position x n and dividing the result by two pixels as the distance.

Returning to FIG. 1, the luminance gradient calculation unit 13 judges whether or not the absolute value of the luminance gradient S of the candidate pixel (i.e., a composite vector of vector sx.X and vector sy.Y, where X is a unit vector in the x-axis direction and Y is a unit vector in the y-axis direction) divided by the square root of the luminance value I of the candidate pixel (=|S|/(I) ^1/2 ) is equal to or greater than a preset gradient luminance value ratio threshold (second threshold). Here, |S|=( ^sx2 + ^sy2 ) ^1/2 , and (I)1/2, which is the square root of the luminance value, is proportional to (N) ^1/2 , which is the square root of the number of ^photons . Also, sx is the x-axis luminance gradient indicating the amount of movement of the pupil in the x-axis direction, and sy is the y-axis luminance gradient indicating the amount of movement of the pupil in the y-axis direction (described later).
Then, the luminance gradient calculation unit 13 determines, as a measurement pixel, a candidate pixel for which |S|/(I) ^1/2 is equal to or greater than a gradient luminance value ratio threshold value.

At this time, the brightness gradient calculation unit 13 may be configured to display on the display unit 16 a confirmation image in which the measurement pixel is marked in the frame image, indicating which pixel in each of the consecutive frame images is the measurement pixel.
The user may then observe the confirmation image on the display unit 16 and adjust the gradient luminance value ratio threshold so that only pixels in the boundary region between the pupil and the iris become measurement pixels, thereby re-extracting the measurement pixels, or may detect pixels extracted as measurement pixels outside the boundary region between the pupil and the iris, and remove these pixels from the measurement pixels.

6 is a conceptual diagram for explaining a process for calculating the amount of pupil movement by the gradient method in this embodiment. In order to simplify the explanation of the gradient method, calculation of the amount of movement in one dimension (only in the x-axis direction) will be explained in FIG.
6A shows the distribution of luminance values in one dimension as a gradation, in which a pattern 201 indicates the n-th frame image, and a pattern 202 indicates the (n+1)-th frame image.
Then, the movement amount Δx (the movement amount Δx in the x-axis direction) is calculated using the same measurement pixel 205 in the pattern 201 and the pattern 202. Here, the measurement pixel 205 is a measurement pixel at the same pixel position in each of the frame images of the pattern 201 and the pattern 202.

6B is a graph for explaining how to obtain the movement amount Δx from the luminance gradient sx (x-axis luminance gradient sx) of the measurement pixel 205 and the luminance change amount ΔI by the gradient method. In FIG. 6, the horizontal axis indicates the coordinate position (pixel position) of the pixel, and the vertical axis indicates the luminance value of the pixel. A curve 203 (solid line) indicates the relationship between the pixel and the luminance value of the pattern 201, and a curve 204 (dotted line) indicates the relationship between the pixel and the luminance value of the pattern 202. The phase of the curve 204 is shifted in the positive direction of the x-axis by the movement amount Δx from the curve 203. This movement amount Δx indicates the movement amount of the pupil in the frame period (the imaging period of the image sensor 50) between frame images.
Here, the luminance change amount ΔI is the luminance value difference (I _n −I _n+1 ) obtained by subtracting the luminance value I _{n +1} of the measurement pixel 205 of the pattern 201 from the luminance value I n of the measurement pixel 205 of the pattern 201 .

As the pupil moves (the eyeball rotates and moves), the position of the pupil imaged at measurement pixel 205 of the frame image of pattern 202 moves by an amount Δx from the position of the pupil imaged at measurement pixel 205 of the frame image of pattern 201.
In this way, as the pupil position changes, the difference in luminance value I at a position corresponding to the amount of movement Δx in the pupil region is measured as the amount of luminance change ΔI.

Therefore, the movement amount Δx can be geometrically calculated from the luminance gradient sx of the measurement pixel 205 and the luminance change amount ΔI of the measurement pixel 205 in FIG.
6B, it can be seen that there is a relationship in the relational expression ΔI=-sxΔx. As a result, the amount of movement Δx can be calculated by the following first formula, which is a modification of the above relational expression.
Δx=-ΔI/sx ... Equation 1 In this Equation 1, in the x-axis direction, sx has a positive polarity when the gradient is one where the brightness increases in the positive direction, and a negative polarity when the gradient is one where the brightness increases in the negative direction. Also, the movement amount Δx has a positive polarity when moving in the positive direction, and a negative polarity when moving in the negative direction.

Furthermore, when calculating the amount of movement Δx from the above-mentioned first equation, it is assumed that the luminance gradient sx of the measurement pixel 205 is constant between the pattern 201 and the pattern 202 (between the frame images).
That is, the luminance gradient sx at the measurement pixel 205 of the curve 203 of the pattern 201 and the luminance gradient sx' at the measurement pixel 205 of the curve 204 of the pattern 202 must be considered to be constant within an allowable error.
Therefore, when the amount of pupil movement Δx between frame images becomes large, the luminance gradient sx of measurement pixel 205 of pattern 201 and the luminance gradient sx' of measurement pixel 205 of pattern 202 do not each have a constant value, and the above first equation cannot calculate the amount of movement Δx from the amount of luminance change ΔI using the frame images of pattern 201 and pattern 202.

FIG. 7 is a diagram showing the relationship between pixel positions and pixel luminance values between frame images when the amount of movement Δx is large.
Since the amount of movement Δx is large, the shift distance between the curve 203 of the pattern 201 and the curve 204 of the pattern 202 at the measurement pixel 205 becomes large.
Therefore, the luminance gradient sx at the measurement pixel 205 of the curve 203 of the pattern 201 and the luminance gradient sx' at the measurement pixel 205 of the curve 204 of the pattern 202 change midway and are not constant (non-linear).
Therefore, when calculating the amount of movement Δx using the luminance value ΔI according to the first equation, it is necessary to select a frame period that results in the amount of movement Δx that results in the same luminance gradient sx.

For this reason, the image sensor 50 in this embodiment uses an image sensor with a short frame period and capable of high-speed imaging, and for example, has an imaging surface with 512 x 512 pixels in a grid pattern, and can capture moving images at 500 frames per second.

In addition, the speed of saccades in normal hopping movements of the pupil (hereinafter referred to as saccade speed) is a maximum of 500 degrees (°) per second. Therefore, the rotation angle of the pupil in 1/500 seconds is a maximum of 1 degree. In addition, as the amount of movement of the pupil becomes smaller, the maximum speed in hopping movements also decreases.
On the other hand, the speed of a microsaccade (amplitude << 1 degree) is, for example, 100 degrees/second or less at most. Therefore, the pupil rotation angle in 1/500 seconds is a maximum of 0.2 degrees.

Here, as the movement amount of the image sensor 50 of the pupil, the maximum pixel movement amount on the imaging surface in one frame time (1/500 seconds) is obtained using FIG.
The conditions for calculating the maximum pixel movement amount in FIG. 8 are that the resolution of the captured image of the image sensor 50 is 128×128 pixels, and the size of the eyeball in the captured image is 50 pixels from the center of the pupil.

Fig. 8 is a conceptual diagram for explaining derivation of the amount of movement in the hopping movement of the pupil on the imaging plane. Fig. 8(a) shows a frame image in a moving image as a captured image, and has a resolution of 128 x 128 pixels. Fig. 8(a) also shows the imaging pixel range of the eyeball on the imaging plane of the image sensor 50. The pupil imaging area EA is the area in the captured image 50A where the pupil is imaged.
A circular area having a radius r (=50 pixels) centered on the center O of the pupil imaging area EA indicates an eyeball imaging area IA, which is an imaging area in which a portion of the eyeball is imaged.

FIG. 8B is a diagram for explaining calculation of the pixel movement amount from the rotation angle of the rotational movement of the eyeball during a microsaccade.
For example, when the rotation angle is 0.2 degrees, the eyeball movement amount dx (Δx) can be calculated as follows from the relationship dx/r=sin(0.2 deg)s=0.0035, assuming that the radius r of the pupil imaging area EA is 50 pixels.
dx = sin(0.2 deg) x 50 (pixels) = 0.17 (pixels)
Furthermore, when the rotation angle is 1.0 degree, the amount of movement dx (Δx) can be calculated as follows from the relationship dx/r=sin(1.0 deg)s=0.017, assuming that the radius r of the pupil imaging area EA is 50 pixels.
dx = sin(0.2 deg) x 50 (pixels) = 0.87 (pixels)

FIG. 9 is a diagram for explaining the linearity of the luminance gradient in the boundary region between the pupil and the iris of the eyeball in the frame image.
9A shows a region Z (near the boundary region of the pupil and the iris) in which the graph in FIG. 2B is enlarged. Region Z is a region of 30 pixels from pixel position 54 to pixel position 84.

FIG. 9B is a graph showing an enlarged range Z of 30 pixels from pixel position 54 to pixel position 84 in FIG.
Observing the luminance gradient at each pixel in Fig. 9(b) reveals that the luminance gradient is constant when the movement amount Δx calculated in Fig. 8 is in the range of 0.17 pixels, and that there is no difference in the luminance gradient even in the range of 0.87 pixels. Therefore, it is possible to measure the movement amount of the pupil within a movement range in which the luminance gradient falls within a similar pixel region (within a region of 9 adjacent pixels including the measurement pixel).

Therefore, as shown in Figure 8, by capturing images at a frame rate of 500 frames/second, the measurement of the movement amount Δx in microsaccades of 100 degrees/second or less is within the range of the pupil with a certain luminance gradient. As a result, if the luminance gradient sx between frame images is equal to calculate the movement amount Δx, the movement amount in the x direction will also be equal, and errors due to nonlinearity can be suppressed, making it possible to measure the movement amount Δx with a specified accuracy using the first formula.

In this embodiment, as described above, the amount of movement of the pupil in the x-axis direction (and the y-axis direction, which will be described later) is calculated using the gradient method from the amount of change in luminance ΔI and the luminance gradient sx of pixels in a pixel region in which an image of the boundary region between the pupil and the iris is captured.
However, since the change in pupil position accompanying eye movement in microsaccades is suppressed to a minute amount Δx as already described, the decrease in accuracy of detection of the amount of movement due to the influence of measurement noise has a large effect.

Generally, shot noise occurs due to statistical fluctuations in the number of photons (actually, the amount of charge corresponding to the number of photons) accumulated in each pixel on the imaging surface of an image sensor.
When the number of accumulated photons is N, the level of this shot noise is proportional to the square root of the number of photons, that is, (N) ^1/2 .

In addition, since the luminance signal level is proportional to the number of accumulated photons N and the shot noise level is proportional to (N) ^1/2 as described above, the S/N ratio (signal-to-noise ratio) is proportional to (N) ^1/2 (= N/(N) ^1/2 ).
Therefore, in the measurement of luminance, as the number N of accumulated photons increases, the signal-to-noise ratio improves and the effect of shot noise in the measurement of luminance is reduced.

On the other hand, since the luminance gradient and the amount of change in luminance are each proportional to the difference value of the number of accumulated photons N, unlike the luminance signal, the accuracy does not improve as the number of accumulated photons N increases. Conversely, as the number of accumulated photons N increases, the shot noise (N) ^1/2 included in the difference value increases.
For this reason, when the luminance gradient is the same, the measurement accuracy of the luminance gradient and the amount of change in luminance is improved by using pixels with smaller luminance values.
For this reason, as already described, the luminance gradient calculation unit 13 uses candidate pixels for which |S|/(I) ^1/2 is equal to or greater than the gradient luminance value ratio threshold as measurement pixels, where the square root of the luminance value (I) ^1/2 is proportional to the square root of the number of photons (N) ^1/2 .

FIG. 10 is a diagram showing the luminance gradient in the boundary region of each of the pupil 501 and the iris 502 in FIG.
In FIG. 10, a luminance difference ΔI between a frame image N and a frame image N+1 one frame period after this frame image N when the pupil moves by Δx is obtained.
When pixels are selected as measurement pixels, if the pixels have the same luminance gradient, the luminance difference ΔI when moved will be the same for all the pixels.

However, pixel position 601 has 200 photons and a noise amount of 14.1 (=(200) ^1/2 ). Pixel position 602 has 100 photons and a noise amount of 10 (=(100) ^1/2 ). Pixel position 603 has 20 photons and a noise amount of 4.7 (=(20) ^1/2 ).
Therefore, the amount of movement Δx ₆₀₂ calculated at pixel position 601 has a noise level 3 (=14.1/4.7) times higher than the amount of movement Δx ₆₀₃ calculated at pixel position 603, resulting in a decrease in measurement accuracy.

In this embodiment, the measurement pixels in which the boundary region between the pupil and the iris is imaged are used to measure the amount of movement Δx (as well as the amount of movement Δy in the y-axis direction, described below), since this is an area where the luminance gradient is higher than other regions. As already mentioned, the boundary region between the pupil and the iris has a large luminance gradient, and even a small amount of movement Δx results in a large amount of luminance change ΔI, making it suitable for detecting the amount of movement Δx.

In order to reduce the effect of the above-mentioned shot noise, when the same luminance gradient is present, a measurement pixel having a lower luminance value (a smaller number N of accumulated photons) is used.
Therefore, in this embodiment, as already described, the luminance gradient calculation unit 13 determines whether or not |S|/(I) ^½ of each candidate pixel is equal to or greater than a preset gradient luminance value ratio threshold, and extracts candidate pixels that are equal to or greater than the gradient luminance value ratio threshold as measurement pixels.

In this embodiment, in order to obtain the movement amount of the pupil in the eyeball in two dimensions, the movement amount Δx parallel to the x-axis is defined as the x-axis direction movement amount Δx, the movement amount Δy parallel to the y-axis is defined as the y-axis direction movement amount, the luminance gradient parallel to the x-axis direction is defined as the x-axis direction luminance gradient, and the luminance gradient parallel to the y-axis direction is defined as the y-axis direction luminance gradient, and the first equation, which is a one-dimensional equation already described above, is expanded to the second equation, which is a linear equation shown below.
ΔI=(-s _x )・Δx+(-s _y )・Δy... Second formula

In this second equation, the x-axis luminance gradient _sx has a positive polarity when the luminance increases in the positive direction along the x-axis, and a negative polarity when the luminance increases in the negative direction along the x-axis. Also, the x-axis direction movement amount Δx has a positive polarity when moving in the positive direction along the x-axis, and a negative polarity when moving in the negative direction along the x-axis.

In the second equation, the luminance gradient s _y has a positive polarity when the luminance increases in the positive direction along the y axis, and the luminance gradient s _y has a negative polarity when the luminance increases in the negative direction along the y axis. Also, the y-axis direction movement amount Δy has a positive polarity when moving in the positive direction along the y axis, and has a negative polarity when moving in the negative direction along the y axis.

In the second equation, the values obtained by measurement are the luminance gradients sx and sy and the luminance change amount ΔI, so the x-axis direction movement amount Δx and the y-axis direction movement amount Δy are both unknowns.
Therefore, the x-axis direction movement amount Δx and the y-axis direction movement amount Δy are each calculated by solving the following simultaneous linear equations generated by the above-mentioned second equation.
ΔI1=(-s _x1 )・Δx+(-s _y1 )・Δy
ΔI2=(-s _x2 )・Δx+(-s _y2 )・Δy

FIG. 11 is a diagram showing a frame image pattern obtained by two-dimensionally expanding the pattern shown in FIG.
In Figure 11, a specified area 801 of the eyeball is imaged, and a frame image (801) of the area 801 is obtained. One frame period later, an area 802 that has moved from the above area 801 of the eyeball by an x-axis movement amount Δx and a y-axis movement amount Δy is imaged, and a frame image (802) of the area 802 is obtained.

In the measurement pixel 611, the x-axis direction luminance gradient sx is _sx1 , the y-axis direction luminance gradient sy is _sy1 , and the luminance change amount between the luminance value of the region 801 and the luminance value of the region 802 is ΔI1.
In addition, in the measurement pixel 812, the x-axis direction luminance gradient sx is _sx2 , the y-axis direction luminance gradient sy is _sy2 , and the amount of luminance change between the luminance value of the region 801 and the luminance value of the region 802 is ΔI2.

In addition, when extracting each of measurement pixel 811 and measurement pixel 812 from the candidate pixels, the brightness gradient calculation unit 13 determines that each of |S1|/(I1) ^1/2 = (sx ₁ ² + sy ₁ ² ) ^1/2 /(I1) ^1/2 and |S1|/(I2) ^1/2 = (sx ₂ ² + sy ₂ ² ) ^1/2 /(I2) ^1/2 is greater than or equal to the gradient brightness value ratio threshold.
Then, the movement distance calculation unit 15 generates the simultaneous linear equations described above using the measurement values of the measurement pixel 811 and the measurement pixel 812 (described later).

The brightness change amount calculation unit 14 calculates the difference in brightness value of a specified measurement pixel between frame images by subtracting the brightness value of the specified measurement pixel in a frame image from the brightness value of the same measurement pixel in a frame image one period later.
The movement distance calculation unit 15 calculates the x-axis direction movement amount Δx and the y-axis direction movement amount Δy by solving the above simultaneous linear equations using the x-axis direction luminance gradient sx, the y-axis direction luminance gradient sy, and the luminance change amount ΔI, which are the measurement values of the measurement pixel 811 and the measurement pixel 812, respectively.

As described above, the simultaneous linear equations may be generated and solved using two measurement pixels. However, in order to further improve accuracy, in this embodiment, the simultaneous linear equations are generated from a plurality of measurement pixels as shown below.
12 is a conceptual diagram for explaining a process of generating simultaneous linear equations from a plurality of measurement pixels. In FIG. 12, as an example of the plurality of measurement pixels, six measurement pixels p1 to p6 (1≦n≦6) are selected as measurement pixels for calculating a shift vector.

Here, the luminance gradient calculation unit 13 compares the absolute value |sx _{n |} of the x-axis direction luminance gradient sx _n (first absolute value) with the absolute value |sy _n | of the y-axis direction luminance gradient (second absolute value).
Then, the luminance gradient calculation unit 13 groups the measurement pixels where |sx _n |>|sy _n | (measurement pixels p2, p4, and p6 in FIG. 12) as a group of advantageous points (first group), and sets these as sx _n1 and sy _n1 , respectively.
Also, sx _n1 and sy _n1 of each measurement pixel in the x-significant point group are integrated to obtain Σsx _n1 and Σsy _n1 .
Here, in the x-significant point group, when the x-axis direction gradient is negative, the sign is inverted in order to make the sign of the luminance gradient consistent when accumulating. In FIG. 12, the measurement pixel p4 in the x-significant point group is the measurement pixel for which polarity inversion is performed.
ΔIn1=(-)(-sx _n1 )Δx+(-)(-sy _n1 )Δy
=sx _n1 Δx+sy _n1 Δy

Then, the luminance variation calculation unit 14 accumulates the luminance variation ΔIn1 of each of the measurement pixels in the x dominant point group to obtain ΣΔIn1.
As a result, the movement distance calculation unit 15 generates the following first linear equation as one of the linear equations based on the x advantageous point group.
ΣΔIn1=Σ(-sx _n1 )Δx + Σ(-sy _n1 )Δy

On the other hand, the luminance gradient calculation unit 13 groups the measurement pixels where |sx _n |≦|sy _n | (measurement pixels p1, p3, and p5 in FIG. 12) as a y-dominant point group (second group), and defines these as sx _n2 and sy _n2 , respectively.
Here, in the y-dominant point group, when the y-axis direction gradient is negative, the sign is inverted in order to make the sign of the brightness gradient consistent when performing integration. In FIG. 12, the measurement pixel p1 in the y-dominant point group is the measurement pixel for which polarity inversion is performed.
ΔIn2=(-)(-sx _n2 )Δx+(-)(-sy _n2 )Δy
=sx _n2 Δx+sy _n2 Δy

Furthermore, the luminance variation calculation unit 14 accumulates the luminance variation ΔIn2 of each of the measurement pixels in the y dominant point group to obtain _ΣΔIn2 .
As a result, the following second linear equation is generated as the other linear equation based on the y advantageous point group.
ΣΔIn2=Σ(-sx _n2 )Δx + Σ(-sy _n2 )Δy

Then, the movement distance calculation unit 15 solves the simultaneous linear equations consisting of the following first and second linear equations to obtain the x-axis direction movement amount Δx and the y-axis direction movement amount Δy.
ΣΔIn1=Σ(-sx _n1 )Δx + Σ(-sy _n1 )Δy
ΣΔIn2=Σ(-sx _n2 )Δx + Σ(-sy _n2 )Δy
Furthermore, the data input/output unit 11 outputs the amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction calculated by the movement distance calculation unit 15 to an external device.
In addition, the data input/output unit 11 may sequentially display the x-axis direction movement amount Δx and the y-axis direction movement amount Δy calculated by the movement distance calculation unit 15 on the display unit 16 and write and store them in the memory unit 17.

As described above, according to this embodiment, the subject's eye is captured as a moving image, and the amount of pupil movement is calculated using the gradient method based on the amount of change in luminance ΔI of the measurement pixel between successive frame images of the moving image, and each of the x-axis luminance gradient and the y-axis luminance gradient. Therefore, there is no need to determine the absolute position of the pupil with high precision; it is sufficient to detect eye movement indicating microsaccades, so the device can be made smaller than conventional devices, and since there is no need to fix the subject's head, it is possible to create a device that can be worn on the subject's head. Microsaccades in the subject's everyday state, that is, when moving and moving their head in daily life, can be accurately measured by determining the amount of pupil movement.
This makes it possible to measure microsaccades under conditions that are similar to those in real life, i.e., to measure and analyze human responses to objects under a variety of conditions using a wearable measuring device.

In addition, according to this embodiment, in order to detect eye movement, the boundary area between the pupil and the iris in the frame image is used as the area with a large brightness gradient, so that even with a small movement, a large amount of brightness change can be detected, and the amount of pupil movement can be obtained with high accuracy.

Furthermore, according to this embodiment, pixels in which the ratio of the luminance gradient (S) to the square root of the luminance value (I ^1/2 ) is equal to or greater than a preset gradient luminance value ratio threshold are used as measurement pixels. Therefore, the pupil movement amount is calculated using measurement pixels (so-called dark pixels with low luminance value for the same luminance gradient) that are less affected by shot noise that increases in proportion to the square root of the number of photons (luminance value), and the measurement accuracy can be improved.

In addition, according to the present invention, the amount of pupil movement is measured using a gradient method that uses the luminance gradient and the amount of change in luminance, so as long as the image capture device has a sufficiently high frame rate, it is possible to obtain high measurement accuracy for the amount of pupil movement of the detected eyeball even if the resolution itself is not that high. And, even if there is no clear feature, as long as there is a detectable luminance gradient, it is possible to measure the amount of eyeball movement.

FIG. 13 is a flowchart for explaining an example of the operation of the measurement process in the eye movement measurement device according to this embodiment.
Step S101:
When a frame image is supplied from an image sensor 50 that is an external device, the data input/output unit 11 writes and stores the frame image data in the storage unit 17 .
The measurement range setting unit 12 erases each of the integrated values Σsx1 and Σsx2 of the x-axis direction luminance gradient sx, the integrated values Σsy1 and Σsy2 of the y-axis direction luminance gradient sy, and the integrated values ΣΔI1 and ΣΔI2 of the luminance change amount ΔI obtained from the immediately previous frame image stored in the memory unit 17 (clearing the data).

Step S102:
The measurement range setting unit 12 reads the luminance value data of each pixel of the frame image from the storage unit 17 .

Step S103:
Then, the measurement range setting unit 12 determines whether or not the luminance value of each pixel in the read frame image is less than the pupil detection threshold value.
At this time, the measurement range setting unit 12 determines pixel values whose luminance values are less than the pupil detection threshold as pixels in the pupil region where the pupil is captured, while the measurement range setting unit 12 determines pixel values whose luminance values are equal to or greater than the pupil detection threshold as pixels in the region where a part other than the pupil is captured.

The measurement range setting unit 12 extracts an image area that is a collection of pixels in the pupil area and pixels surrounded by the pixels in the pupil area, and sets this image area as an area in which the pupil is imaged (pupil imaged area).
Then, the measurement range setting unit 12 sets the distance from the center O of the measurement range 600 to a pixel on the outer periphery of the measurement range 600 as the radius R.
The measurement range setting unit 12 calculates a radius R′ by multiplying the radius R by a predetermined coefficient value p, and sets the range from the center O to the radius R′ as the measurement range 600 .

Step S104:
The luminance gradient calculation unit 13 refers to the storage unit 107, selects a pixel in the frame image, and reads the pixel position of that pixel.
Here, the luminance gradient calculation unit 13 selects pixels in the order of columns, for example, starting from the top row in the reference frame image, while sequentially switching the row to be selected.

Step S105:
The luminance gradient calculation unit 13 determines whether or not the pixel position of the selected pixel is included in the measurement range 600 having a radius R' from the center O.
At this time, if the pixel position of the selected pixel is included in the measurement range 600 having a radius R' from the center O, the luminance gradient calculation unit 13 advances the process to step S106.
On the other hand, if the pixel position of the selected pixel is not included in the measurement range 600 of radius R' from the center O, the luminance gradient calculation unit 13 advances the process to step S112.

Step S106:
The luminance gradient calculation unit 13 refers to the storage unit 107 and reads the luminance value of the pixel selected in the frame image.
Then, the luminance gradient calculation unit 13 determines whether or not the pixel value of the read pixel is less than a luminance threshold value (first threshold value).

At this time, if the pixel value of the read pixel is less than the luminance threshold value (first threshold value), the luminance gradient calculation unit 13 extracts this pixel as a candidate pixel and proceeds to step S107.
On the other hand, if the pixel value of the read pixel is not less than the luminance threshold value (first threshold value), the luminance gradient calculation unit 13 advances the process to step S112.

Step S107:
The luminance gradient calculation unit 13 calculates each of the x-axis direction luminance gradient sx and the y-axis direction luminance gradient sy of the candidate pixel from the eight adjacent pixels by the process described with reference to FIG.
Then, the luminance gradient calculation unit 13 calculates the luminance gradient S by calculating (sx ² +sy ² ) ^1/2 from each of the x-axis direction luminance gradient sx and the y-axis direction luminance gradient sy.

Step S108:
The luminance gradient calculation unit 13 calculates |S|/(I) ^1/2 from each of the calculated luminance gradient S and luminance value I, and determines whether the calculated |S|/(I) ^1/2 is equal to or greater than a gradient luminance ratio threshold value.
At this time, if the calculated |S|/(I) ^1/2 is equal to or greater than the gradient luminance ratio threshold, the luminance gradient calculation unit 13 extracts this candidate pixel as a measurement pixel, and advances the process to step S109.
On the other hand, if the calculated |S|/(I) ^1/2 is less than the gradient luminance ratio threshold, the luminance gradient calculation unit 13 advances the process to step S112.

Step S109:
The luminance gradient calculation unit 13 compares the absolute value |sx _n | of the x-axis luminance gradient sx _n with the absolute value |sy _n | of the y-axis luminance gradient sy _n for each measurement pixel.
Then, the luminance gradient calculation unit 13 groups the measurement pixels where |sx _n |>|sy _n | as an x-dominant point group (first group), and groups the measurement pixels where |sx _n |≦|sy _n | as a y-dominant point group (second group).
Then, when the measurement pixel is a measurement pixel of the x-advantageous point group, the luminance gradient calculation unit 13 accumulates the x-axis luminance gradient sx _n1 and the y-axis luminance gradient sy _n1 of the measurement pixel of the x-advantageous point group to calculate the accumulated values Σsx _n1 and Σsy _n1 , and writes and stores them in the memory unit 17.
On the other hand, when the measurement pixel is a measurement pixel of the y-dominant point group, the luminance gradient calculation unit 13 accumulates each of the x-axis luminance gradient sx _n2 and the y-axis luminance gradient sy _n2 of the measurement pixel of the y-dominant point group to calculate the accumulated values Σsx _n2 and Σsy _n2 , and writes and stores them in the memory unit 17.

Step S110:
The luminance change amount calculation unit 14 refers to the frame image data in the storage unit 17, and subtracts the luminance value of the corresponding measurement pixel in the frame image from the luminance value of the measurement pixel in the frame image immediately before to calculate the luminance change amount ΔI.

Step S111:
When the measurement pixel is a measurement pixel of the x advantageous point group, the brightness change amount calculation unit 14 accumulates the brightness change amount ΔI1 (=ΔI) of the measurement pixel of the x advantageous point group to calculate the accumulated value ΣΔI1 and writes and stores it in the memory unit 17.
On the other hand, when the measurement pixel is a measurement pixel of the y advantageous point group, the brightness change amount calculation unit 14 accumulates the brightness change amount ΔI2 (= ΔI) of the measurement pixel of the y advantageous point group to calculate the accumulated value ΣΔI2 and writes and stores it in the memory unit 17.

Step S112:
The luminance gradient calculation unit 13 determines whether or not the processing has been completed for all pixels in the frame image.
Here, the luminance gradient calculation unit 13 makes the determination based on, for example, whether or not the selected pixel is the pixel position of the end point in the frame image.
At this time, if the processing for all pixels in the frame image has been completed, the luminance gradient calculation unit 13 advances the process to step S113.
On the other hand, if the processing has not been completed for all pixels in the frame image, the luminance gradient calculation unit 13 advances the process to step S104.

Step S113:
The movement distance calculation unit 15 reads out the integrated values Σsx _n1 , Σsy _n1 , and ΣΔI1, and the integrated values Σsy _n2 , Σsx _n2 , and ΣΔI2, and creates the following simultaneous linear equations.
ΣΔIn1=Σ(-sx _n1 )Δx + Σ(-sy _n1 )Δy
ΣΔIn2=Σ(-sx _n2 )Δx + Σ(-sy _n2 )Δy
Then, the movement distance calculation unit 15 solves the simultaneous linear equations, calculates the amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction as the amount of movement of the pupil, and ends the process.
At this time, the movement distance calculation unit 15 displays the calculated amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction on the display screen of the display unit 16 .
Furthermore, the data input/output unit 11 may output each of the calculated x-axis direction movement amount Δx and y-axis direction movement amount Δy.

It should be noted that a program for implementing the functions of the eye movement measuring device 10 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to measure the amount of pupil movement. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.
Additionally, "computer system" includes a WWW system equipped with a homepage providing environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that hold a program for a certain period of time, such as volatile memory (RAM) within a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication circuit such as a telephone circuit.

The above program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication circuit (communication line) such as a telephone circuit. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

REFERENCE SIGNS LIST 1 measurement system 10 eye movement measuring device 11 data input/output section 12 measurement range setting section 13 luminance gradient calculation section 14 luminance change amount calculation section 15 movement distance calculation section 16 display section 17 memory section 20 infrared irradiation device 30 imaging lens 40 infrared filter 50 image sensor 300 eyeball 301 pupil 302 iris 600 measurement range

Claims (14)

An eye movement measuring device for detecting a change in a position of a focus point of an eye in a moving image of an eye that includes the eye,
a measurement range setting unit that sets an area of the moving image that satisfies a predetermined condition as a measurement range;
a luminance gradient calculation unit that calculates a luminance gradient in the measurement range;
a luminance change amount calculation unit for calculating a luminance change amount between frame images in the measurement range;
a moving distance calculation unit that calculates the position change by using a gradient method based on each of the luminance change amount and the luminance gradient ;
Equipped with
The measurement range setting unit selects, within the measurement range, pixels having a luminance value less than a first threshold for reducing an effect of shot noise in the moving image as the measurement range.
An eye movement measuring device comprising: 2. The eye movement measuring device according to claim 1, wherein the measurement range setting unit sets an area including a boundary between the pupil and the iris of the eye as the measurement range. The moving image is
The eye movement measuring device according to claim 1 or claim 2, characterized in that the position change of the pupil between the frame images is captured at a frame period within a movement range that falls within the measurement range in which the luminance gradient has a constant gradient. 4. The eye movement measuring device according to claim 2, wherein the measurement range setting unit compares the brightness value of each pixel in the frame image with a brightness threshold value, extracts a pupil image area which is an image area of the pupil, and sets an area obtained by enlarging the pupil image area from a center point of the pupil image area in an outer circumferential direction as the measurement range. 5. The eye movement measuring device according to claim 1, wherein the measurement range setting unit selects, as the measurement range, a group of pixels in an area within the measurement range in which the luminance gradient is higher than other areas . The eye movement measuring device according to claim 1 , wherein the measurement range setting unit selects pixels having lower luminance values in the luminance gradient as pixels in the measurement range. The eye movement measuring device according to claim 1 , wherein the measurement range setting unit selects pixels in the measurement range based on both the luminance gradient and the luminance value of a pixel within the measurement range. The eye movement measuring device according to any one of claims 1 to 7, characterized in that the measurement range setting unit selects pixels in the measurement range by comparing a numerical value obtained by dividing the brightness gradient by an amount of shot noise estimated from the brightness values of pixels within the measurement range with a second threshold value. The eye movement measuring device according to any one of claims 1 to 8, characterized in that the measurement range setting unit sets, within the measurement range, pixels for which a value obtained by dividing the luminance gradient by the square root of the luminance value of the pixel is greater than a second threshold value as pixels of the measurement range. The measurement range setting unit,
10. The eye movement measuring device according to claim 1 , further comprising: determining an x-axis luminance gradient as a luminance gradient in the x-axis direction and a y-axis luminance gradient as a luminance gradient in the y-axis direction corresponding to the measurement range; and formulating simultaneous linear equations from the luminance change amount, the x-axis luminance gradient, and the y-axis luminance gradient to determine an x-axis movement amount that is a movement amount of the pupil in the x-axis direction and a y-axis movement amount that is a movement amount in the y- axis direction. The measurement range setting unit,
comparing a first absolute value, which is an absolute value of the luminance gradient in the x-axis direction, with a second absolute value, which is an absolute value of the luminance gradient in the y-axis direction, in each of the measurement ranges;
a first group of pixels in a measurement range in which the first absolute value is greater than the second absolute value, and a second group of pixels in a measurement range in which the second absolute value is greater than the first absolute value;
generating one of the simultaneous linear equations, in which a result of adding up the x-axis luminance gradients of each of the pixels in the first group and multiplying the result by an amount of movement in the x-axis direction, which is a variable, and adding up the y-axis luminance gradients and multiplying the result by an amount of movement in the y-axis direction, which is a variable, is an integrated value of the amount of change in luminance of each of the pixels;
generating the other equation of the simultaneous linear equations, in which a result of adding up the x-axis luminance gradients of each of the pixels in the second group and multiplying the result by an amount of movement in the x-axis direction, which is a variable, and a result of adding up the y-axis luminance gradients and multiplying the result by an amount of movement in the y-axis direction, which is a variable, is an integrated value of the amount of change in luminance in each of the measurement ranges;
The eye movement measuring device according to claim 10 , wherein the x-axis movement amount and the y-axis movement amount are each obtained by calculating the simultaneous linear equations. The eye movement measuring device according to claim 1 , further comprising: a step of irradiating the eye with infrared light and capturing a multi-tone image of reflected light from the eye to obtain the moving image. An eye movement measuring method for detecting a change in position of a focus point of an eye in a moving image of an eye that includes the eye, the method comprising:
a measurement range setting step in which a measurement range setting unit sets an area of the moving image that satisfies a predetermined condition as a measurement range, and selects, within the measurement range, pixels having a luminance value less than a first threshold value for reducing an effect of shot noise in the moving image as the measurement range ;
a luminance gradient calculation step in which a luminance gradient calculation unit calculates a luminance gradient in the measurement range;
a luminance change amount calculation step in which a luminance change amount calculation unit calculates a luminance change amount between frame images in the measurement range;
a movement distance calculation step in which a movement distance calculation unit calculates the position change by using a gradient method based on each of the luminance change amount and the luminance gradient;
13. An eye movement measuring method comprising: A program for causing a computer to execute an operation of an eye movement measuring device for detecting a change in a position of a focus point of an eye in a moving image of an eye that includes the eye,
The computer,
a measurement range setting means for setting an area of the moving image that satisfies a predetermined condition as a measurement range, and selecting, within the measurement range, pixels having a luminance value less than a first threshold value for reducing an effect of shot noise in the moving image as the measurement range;
a luminance gradient calculation means for calculating a luminance gradient in the measurement range;
a luminance change amount calculation means for calculating an amount of luminance change between frame images in the measurement range;
a program for causing the computer to function as a moving distance calculation means for calculating the position change by using a gradient method based on each of the amount of luminance change and the luminance gradient.

Images