JP6311461B2 - Gaze analysis system and gaze analysis apparatus - Google Patents

Description

  The present invention relates to a technique for estimating and analyzing a person's line of sight.

  Conventionally, it is known to estimate and track the line of sight of a person looking at the printed material to be evaluated, and to measure the line of sight of the posted product, analyze the degree of gaze and interest, and evaluate and analyze consumer purchasing trends. ing. In order to provide appropriate purchase information to consumers, it is important to know how to view the printed information on the printed material to be evaluated.

  In order to analyze and evaluate the degree of attention of the posted information of the printed material to be evaluated, the applicant simply estimates and tracks the subject's gaze with high accuracy without causing unnecessary gaze movement in the subject. The technique which can do is proposed (refer patent document 1). Using this technique, it is possible to create a heat map on the reference image after converting the gazing point into a reference image that is in a state where a printed matter or the like is captured from the front. A heat map is a type of graph that expresses the magnitude of a numerical value based on the result of clustering or the like by color shading. With the heat map, it is possible to confirm which part on the reference image the subject was viewing.

JP 2013-81762 A JP 2007-319187 A

DavidG. Low, “Distinctive image features from scalevariant keypoints”, Int. Journal of Computer Vision, Vol. 60, no. 2, PP. 91-110, 2004. H. Bay, T .; Tuyterrars, and L. VanGool, “SURF: Speeded Up Robust. Features”, InECCV, pp. 404-417, 2006. Koichi Hashimoto, “Visual Servo-II—Basics of Computer Vision”, Transactions of the Virtual Reality Society of Japan, Vol. 4, 1999. Toby Technology Japan Co., Ltd., Tobii Glass Eye Tracker, [online] [searched on September 8, 2011], Internet <URL: http: // www. tobii. com / ja-JP / eye-tracking-research / japan / products / hardware / tobii-glasses-eye-tracker /> OpenCV, SIFT / SURF, [online] [searched on September 8, 2011], Internet <URL: http: // opencv. jp / opencv-2.2 / cpp / features2d_feature_detection_and_description. html> V. Lepetit and P.M. Fua, “Keypoint Recognition using Randomized Trees”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28, no. 9, pp. 1465-1479, 2006. M.M. Ozuyal, P.M. Fua, V.A. Lepetit, -Fast Keypoint Recognition in Ten Lines of Code. , Proc. of Int. Conf. on Computer Vision and Pattern Recognition (2007)

  By using a heat map created by using the technique described in Patent Document 1, it is possible to confirm how much and what part of the reference image has been viewed. However, if the target object is three-dimensional, has many curved surfaces, or has many areas with few features, the features of the target are correctly reflected in the reference image if the reference image is obtained from a certain angle. There is a problem that it is not. Also, when the object is imaged in a dim place, the scene image captured by the scene camera for tracking the line of sight is darker than it is actually seen, and it is difficult to recognize and match the object. There's a problem.

  Therefore, the present invention provides a line-of-sight analysis system and a line-of-sight analysis apparatus that can identify a spot on a target object that is viewed by a viewer even when the target object is present in an environment that is not suitable for imaging. This is the issue.

  In order to solve the above problems, the present invention includes a camera that captures an object, a distance sensor that measures a distance from the object, a line-of-sight tracking apparatus that acquires a gaze point of the subject, the line-of-sight tracking apparatus, and data A system having a gaze analysis device capable of communication, wherein the gaze analysis device is measured by a scene image captured by the camera, a scene image acquisition means for acquiring a gazing point on the scene image, and the distance sensor. A scene distance image acquiring unit for acquiring a scene distance image based on the determined distance, a scene image feature point extracting unit for extracting a plurality of scene image feature points from the scene image, and a plurality of scene distance image features from the scene distance image. Scene distance image feature point extraction means for extracting points, reference image feature points in a reference image prepared in advance, and the scene image feature points A scene image associating means for associating; a scene distance image associating means for associating a reference distance image feature point with a scene distance image feature point in a reference distance image prepared in advance; and the scene image and scene A coordinate transformation matrix that transforms the coordinates of both of the distance images based on the correspondence between the feature points of the selected image and the reference image or the reference distance image, with the image that has been associated more frequently as the selected image. For each selected image, and using the generated coordinate transformation matrix as a reference for the coordinates of the gazing point associated with the scene image for each of the plurality of scene images. There is provided a line-of-sight analysis system characterized by comprising corresponding point calculation means for converting to coordinates on an image and calculating corresponding points.

  According to the line-of-sight analysis system according to the present invention, a camera that captures an object and a distance sensor that measures the distance from the object are used, and the image is captured by the camera using a line-of-sight tracking device that acquires the gaze point of the subject. A scene image and a gazing point on the scene image are acquired, a scene distance image based on the distance measured by the distance sensor is acquired, a plurality of scene image feature points are extracted from the scene image, and a plurality of scenes are extracted from the scene distance image. Extracts distance image feature points, associates scene image feature points with reference image feature points, associates scene distance image feature points with reference distance image feature points, and supports correspondence between scene image and scene distance image Based on the correspondence between the feature point of the selected image and the corresponding reference image or the selected feature point, the coordinate of both is converted using the image with more attachments as the selected image A target transformation matrix is created for each selected image, and using the created coordinate transformation matrix, for each of the plurality of scene images, the coordinates of the gazing point associated with the scene image are used as the coordinates on the reference image. Since conversion is performed and corresponding points are calculated, it is possible to specify a spot on the target object that is viewed by the viewer even if the target object exists in an environment that is not suitable for imaging.

  In the present invention, the line-of-sight tracking device of the line-of-sight analysis system performs imaging by the camera and measurement by the distance sensor in synchronization. According to the present invention, since the imaging by the camera and the measurement by the distance sensor are performed in synchronization, the gazing point acquired together with the scene image can be acquired as the gazing point corresponding to the scene distance image.

  The present invention further includes a camera that captures an object and a distance sensor that measures the distance to the object, and a scene image captured by the camera and a scene based on the distance measured by the distance sensor. A line-of-sight analysis apparatus connected to a line-of-sight tracking apparatus for acquiring a distance image and a gaze point of a subject on a scene image so as to be capable of data communication, the scene image captured by the camera, and the scene Scene image acquisition means for acquiring a gazing point on an image, scene distance image acquisition means for acquiring a scene distance image based on a distance measured by the distance sensor, and extracting a plurality of scene image feature points from the scene image Scene image feature point extracting means; scene distance image feature point extracting means for extracting a plurality of scene distance image feature points from the scene distance image; Reference image feature point acquisition means for acquiring a plurality of reference image feature points in the image, reference distance image feature point acquisition means for acquiring a plurality of reference distance image feature points in the reference distance image, the scene image feature points and the reference image feature points Of the scene image and the scene distance image, the scene image association means for obtaining the correspondence relationship, the scene distance image association means for obtaining the correspondence relationship between the scene distance image feature point and the reference distance image feature point, and the scene image and the scene distance image. A coordinate transformation matrix that converts the coordinates of both of the selected images based on the correspondence between the feature point of the selected image and the reference image or the selected feature point corresponding to the feature point of the selected image is selected for each selected image. Using the transformation matrix creation means to create and the created coordinate transformation matrix, each of the plurality of scene images is associated with the scene image. Was the coordinate of the gazing point is converted into coordinates on the reference image, the corresponding point calculation means for calculating a corresponding point, to provide a visual line analysis apparatus characterized by having a.

  According to the line-of-sight analysis apparatus according to the present invention, a scene image captured by the camera of the line-of-sight tracking apparatus and a scene distance based on a distance measured by a distance sensor of the line-of-sight tracking apparatus while acquiring a gaze point on the scene image Acquire images, extract multiple scene image feature points from the scene image, extract multiple scene distance image feature points from the scene distance image, associate the scene image feature points with the reference image feature points, and The image feature point is associated with the reference distance image feature point, and the reference image or the selection corresponding to the feature point of the selected image is selected as the selected image of the scene image and the scene distance image, which has been associated with a large amount. Based on the correspondence between feature points, a coordinate transformation matrix that transforms the coordinates of both is created for each selected image, and multiple coordinates are created using the created coordinate transformation matrix. For each scene image, the coordinates of the gazing point associated with the scene image are converted to the coordinates on the reference image, and the corresponding points are calculated. Therefore, the scene that includes the camera and the distance sensor and is captured by the camera Combined with an image, a scene distance image based on the distance measured by the distance sensor, and a gaze tracking device that acquires the gaze point of the corresponding subject, even if the target is in an environment that is not suitable for imaging It is possible to specify a spot on the object that is viewed by the viewer.

  According to the present invention, even if the target object is present in an environment that is not suitable for imaging, it has an effect that it is possible to specify a spot on the target object that is viewed by the viewer.

1 is an external view showing an outline of a line-of-sight analysis system according to an embodiment of the present invention. It is a figure which shows the hardware constitutions of the gaze analysis system which concerns on one Embodiment of this invention. It is a flowchart which shows the processing operation of the gaze analysis system which concerns on one Embodiment of this invention. It is a figure which shows the scene image and gaze point which are acquired by the gaze tracking apparatus. It is a figure which shows the gazing point and feature point on a scene image. It is a figure which shows the correspondence of a reference image feature point and a scene image feature point. It is a figure which shows the conversion type | formula from the gaze point of a scene image to the corresponding point of a reference | standard image. It is a figure which shows the example of the coordinate transformation from the gaze point of a scene image to the corresponding point of a reference | standard image. It is a figure which shows the relationship between the reference | standard image and the corresponding point obtained by converting a gaze point. It is a figure which shows an example of the heat map produced by the visual line analysis system which concerns on this embodiment.

<1. System configuration>
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the drawings. FIG. 1 is an external view showing an outline of a line-of-sight analysis system according to an embodiment of the present invention. In FIG. 1, 100 is a line-of-sight analysis apparatus, 200 is a line-of-sight tracking apparatus, and 300 is an object. In FIG. 1, the example which used the leaflet printed matter as an object is shown.

  The line-of-sight tracking apparatus 200 obtains a scene image by capturing the object 300 from a position close to the viewpoint of the person wearing the glasses-type line-of-sight tracking apparatus 200 with an attached camera, and from the person applied with the attached distance sensor. A scene distance image based on the distance is acquired. In synchronization with the timing of imaging and the timing of distance measurement, the viewpoint position is acquired by the infrared sensor, and the gazing point on the scene image is acquired. The gazing point and scene distance image corresponding to the scene image and each scene image are passed to the line-of-sight analysis apparatus 100 as data by transmission or other means. The line-of-sight analysis apparatus 100 stores the reference image and the reference distance image of the object 300, associates the feature points of each scene image acquired from the line-of-sight tracking apparatus 200 with the feature points of the reference image, and tracks the line of sight. The feature points of each scene distance image acquired from the apparatus 200 are associated with the feature points of the reference distance image, and the image with the larger number of associations is selected as the selected image, and the selected image and the reference image or the reference distance image A coordinate transformation matrix is calculated, and the gaze point of each scene image is transformed into a corresponding point on the reference image using the coordinate transformation matrix.

  FIG. 2A is a hardware configuration diagram of the line-of-sight analysis apparatus 100. The line-of-sight analysis apparatus 100 can be realized by incorporating a dedicated program into a general-purpose computer. As shown in FIG. 2, the line-of-sight analysis apparatus 100 includes a CPU (Central Processing Unit) 10a, a main memory RAM (Random Access Memory) 10b, a hard disk and a flash for storing programs and data executed by the CPU 10a. A storage device 10c such as a memory; an instruction input unit 10d such as a keyboard / mouse; a data input / output I / F (interface) 10e for data communication with an external device such as a line-of-sight tracking device 200 or a data storage medium; A display unit 10f, which is a display device such as a display, is provided and connected to each other via a bus.

  In the storage device 10c, a reference image, a reference image feature point that is a feature point extracted from the reference image by the methods described in Patent Document 1 and Non-Patent Documents 1 to 7, and a feature point extracted from a reference distance image Are stored in association with the respective images. The reference image is acquired by imaging the image data of the object itself or the object from the front. The reference distance image is acquired as a data array in which a distance sensor is installed in front of an object, the distance is measured, and coordinates and distances are associated with each other based on the measured distance. Since this data array can be expressed as an image by displaying the value corresponding to each coordinate as a gradation on a display device, it will be referred to as a reference distance image. The pixel sizes (vertical × horizontal number of pixels) of the reference image and the reference distance image are arranged to be the same.

  FIG. 2B is a hardware configuration diagram of the eye tracking device 200. The infrared light source 201 irradiates the eyeball 301 (the cornea (pupil) or sclera (white eye)) of the subject with the infrared ray 207. The infrared sensor 202 receives the infrared ray 208 (reflected light) reflected from the eyeball 301. The scene camera 203 is a two-dimensional image sensor. The scene camera 203 captures, as a scene image, a scene (an image including a flyer centered on a gazing point) that is substantially equivalent to the subject's visual field. The distance sensor 206 is a sensor that measures the distance to the object. The line-of-sight tracking apparatus 200 shown in FIG. 2B is realized by attaching a known distance sensor to the known line-of-sight tracking apparatus shown in Patent Document 1.

  Various known types of distance sensors can be used as the distance sensor 206, but in this embodiment, a TOF (Time-of-Time) that measures the distance from the time it takes for the projected laser to reciprocate to the object. The Flight method is used. Therefore, the distance sensor 206 shown in FIG. 2 has a configuration including a laser light source, a time measurement circuit, and a light receiving unit.

  The processing unit 204 includes an arithmetic processing circuit, a storage circuit, a connector, and a secondary battery. For example, the processing unit 204 uses the reflected light 208 received based on the sclera reflection method (the robust tracking method, according to Patent Document 2) using the difference in reflectance between the cornea and the sclera of the eyeball 301. To estimate the direction of the line of sight. Note that the processing unit 204 can use a pupil-corneal reflection method, a corneal reflection method, a double pre-Kinnie method, or the like instead of the sclera reflection method. And the coordinate of the gaze point on the scene image which is a two-dimensional plane is calculated | required using the direction of the gaze estimated as shown in patent document 1, and the position of the printed matter in three-dimensional space. In addition, the processing unit 204 transmits the obtained gazing point and the captured scene image to the line-of-sight analysis device 100 connected in communication. Further, the processing unit 204 transmits the acquired scene distance image to the line-of-sight analysis device 100 connected for communication. The connector is communicatively connected to the line-of-sight tracking apparatus 200 and the line-of-sight analysis apparatus 100. The secondary battery supplies power to each circuit of the processing unit 204 itself, the infrared light source 201, the infrared sensor 202, and the scene camera 203. In the processing unit 204, the arithmetic processing circuit executes each of the above processes according to a program stored in the storage circuit.

<2. Processing action>
Next, the processing operation of the line-of-sight analysis system shown in FIGS. 1 and 2 will be described using the flowchart shown in FIG. First, the operator activates the eyeglass-type line-of-sight tracking device 200 and puts the eyeglass-type eye-gaze tracking device 200 like glasses to start browsing the object 300. When the eye tracking device 200 is activated, the scene camera 203 starts imaging, the distance sensor 206 starts measuring the distance, the infrared light source 201 starts emitting light, and the infrared sensor 202 also starts receiving light. A series of processes from step S11 to step S14 shown in FIG. 3 and a series of processes from step S15 to step S18 are performed in parallel.

  First, a series of processing from step S11 to step S14 will be described. First, the line-of-sight tracking apparatus 200 acquires a scene image and a gazing point (step S11). Specifically, the scene camera 203 captures images at predetermined time intervals to acquire a scene image, and the infrared sensor 202 acquires a gazing point. The scene camera 203, the infrared light source 201, and the infrared sensor 202 are synchronized with each other, and a gazing point is acquired at the timing when a scene image is captured. The predetermined time interval can be set as appropriate. For example, it can be set at an interval of 1 second. Also, the number of shots can be set arbitrarily. In the present embodiment, description will be made assuming that 10 scene images are taken at intervals of 1 second. The processing unit 204 of the line-of-sight tracking apparatus 200 sequentially transmits the acquired scene image and gazing point to the line-of-sight analysis apparatus 100 via a data communication unit (not shown). The gazing point is transmitted in a format including at least the coordinates in the corresponding scene image. The line-of-sight analysis apparatus 100 receives a scene image and a gazing point via the data input / output I / F 10e.

  FIG. 4 is a diagram illustrating a relationship between the scene image C and the gazing point E. As illustrated in FIG. As described above, when 10 scene images are captured, the line-of-sight analysis apparatus 100 receives 10 scene images C1 to C10. Then, the line-of-sight analysis apparatus 100 extracts gazing points E1 to E10 corresponding to the scene images C1 to C10, respectively. In FIG. 4, “• (black circles, dots)” in the scene images C1 to C10 indicated by rectangular frames indicate the positions of the gazing points E1 to E10. In the example of FIG. 4, only the rectangular frames are shown as the scene images C1 to C10, but actually, as shown in FIG. 5, at least a part (preferably all) of the object 300 is captured. To be recorded. FIG. 5A shows an example of one scene image, and FIG. 5B shows a state in which a gazing point is mapped on the scene image.

  Next, the line-of-sight analysis apparatus 100 extracts scene image feature points (step S12). Specifically, the scene image is analyzed, and a scene image feature point that is a point expressing the feature of the scene image is extracted. Extraction of scene image feature points from a scene image can be performed by various known methods. For example, the SIFT algorithm described in Non-Patent Document 1, the SURF (SpeededUpRobustFeatures) algorithm described in Non-Patent Document 2, the Randomized Tree algorithm described in Non-Patent Document 6, and the Ferns algorithm described in Non-Patent Document 7 may be adopted. it can. That is, a scene image feature point is extracted from a scene image by the CPU 10a executing a program for realizing the above algorithm. In this embodiment, when the CPU 10a executes a program corresponding to the SIFT algorithm described in Non-Patent Document 1, scene image feature points are extracted from the scene images C1 to C10, respectively. FIG. 5C shows one feature point Q extracted from the scene image C.

  Next, the line-of-sight analysis apparatus 100 acquires a reference image feature point (step S13). Specifically, the reference image feature points extracted and stored in advance from the reference image are acquired from the storage device 10c. Subsequently, the line-of-sight analysis apparatus 100 associates the scene image feature point extracted from the scene image with the reference image feature point (step S14). Specifically, using the technique described in Patent Document 1, a predetermined number of feature points corresponding to the reference image and each scene image are extracted, and feature points having high SIFT feature value similarity are associated with each other. .

  FIG. 6 shows the result of associating the reference image feature point P with the scene image feature point Q. By using the method described in Patent Document 1, the feature points are associated with each other as shown in FIG. Usually, in order to determine whether or not the SIFT feature value similarity is high by comparing the SIFT feature value similarity with a predetermined threshold, each scene image corresponds to the reference image feature point of the reference image. The number of scene image feature points attached is different. Since the feature point association in step S14 is for identifying the corresponding points between the images of the same object, various known methods other than the method described in Patent Document 1 are used. It is possible to use.

  Next, a series of processing from step S15 to step S18 will be described. First, the line-of-sight tracking apparatus 200 acquires a scene distance image (step S15). Specifically, the distance sensor 206 measures the distance at predetermined time intervals and acquires a scene distance image. The predetermined time interval can be set as appropriate. For example, it can be set at an interval of 1 second. The number of distance measurements can also be set arbitrarily. In the present embodiment, description will be made assuming that the distance is measured 10 times at intervals of 1 second. The processing unit 204 of the line-of-sight tracking apparatus 200 creates a data array in which coordinates and distances are associated with each other based on the distance measured by the distance sensor 206. Since this data array can be expressed as an image by displaying the value corresponding to each coordinate as a gradation on a display device, it is called a scene distance image. The processing unit 204 of the line-of-sight tracking apparatus 200 sequentially transmits the acquired scene distance images to the line-of-sight analysis apparatus 100 via a data communication unit (not shown). The line-of-sight analysis apparatus 100 receives a scene image via the data input / output I / F 10e.

  Next, the line-of-sight analysis apparatus 100 extracts scene distance image feature points (step S16). Specifically, the scene distance image is analyzed, and a scene image distance image feature point that is a point expressing the feature of the scene distance image is extracted. Extraction of scene distance image feature points from a scene image can be performed by various known methods. For example, as with scene images, the SIFT algorithm described in Non-Patent Document 1, the SURF (SpeededUpRobustFeatures) algorithm described in Non-Patent Document 2, the Randomized Tree algorithm described in Non-Patent Document 6, and the Ferns algorithm described in Non-Patent Document 7. Can be adopted. That is, a scene distance image feature point is extracted from a scene distance image by the CPU 10a executing a program for realizing the above algorithm. In the present embodiment, when the CPU 10a executes a program corresponding to the SIFT algorithm described in Non-Patent Document 1, scene distance image feature points are extracted from the scene distance images CK1 to CK10, respectively.

  Next, the line-of-sight analysis apparatus 100 acquires a reference distance image feature point (step S17). Specifically, the reference distance image feature points extracted in advance from the reference distance image and stored are acquired from the storage device 10c. Subsequently, the scene distance image feature point extracted from the scene distance image is associated with the reference distance image feature point (step S18). Specifically, similar to step S14, the technique described in Patent Document 1 is used to extract a predetermined number of feature points corresponding to the reference distance image and each scene distance image, and the similarity of SIFT feature values is high. The feature points are associated with each other. Similar to the case of step S13, the feature points are associated with each other using the image as shown in FIG.

  As described above, the series of processes in steps S11 to S14 and the series of processes in steps S15 to S18 are performed in parallel. Therefore, as described above, the series of processing from step S11 to step S14 may be executed first, but the series of processing from step S15 to step S18 may be executed first. Further, processing may be performed alternately for each scene image and each scene distance image. In any case, it is necessary to finish both processes before proceeding to the next process.

  Preferably, step S11 and step S15 are preferably performed simultaneously. Specifically, the imaging by the camera 203 and the measurement by the distance sensor 206 are performed in synchronization. In this case, the synchronization means that the synchronization is performed within a predetermined relatively short time range (for example, 0.1 second).

  When the association between the reference image and the feature points of each scene image in step S14 is completed for all scene images, and the association of the reference distance image and the feature points of each scene distance image in step S18 is completed for all scene distance images, Then, an image to be adopted is selected (step S20). Specifically, as a result of the association in step S14 and step S18, an image with more feature points associated is selected. This is because the matching accuracy is higher when more associations are performed.

  The image to be adopted is selected for each scene image and each scene distance image. That is, the scene image C1 and the scene distance image CK1 are compared, and the one that has many feature points associated therewith is selected. Accordingly, when 10 scene images and 10 scene distance images are used as in the present embodiment, one of the images with the same number is selected, and a total of 10 images are selected. For example, C1, C2, C5, and C8 are selected from the scene image, and CK3, CK4, CK6, CK7, CK9, and CK10 are selected from the scene distance image.

  Next, a coordinate transformation matrix is created using the selected image and the feature points of the corresponding reference image or reference distance image (step S30). When a scene image is selected, the reference image feature point (X, Y) and the scene image feature point (x, y) are substituted into a predetermined conversion formula shown in FIG. 7A to obtain a coordinate conversion matrix. In this embodiment, a homography matrix H for projecting from one plane of a three-dimensional plane to another plane is used as the coordinate transformation matrix. FIG. 7 is a diagram illustrating a conversion formula from the gazing point of the scene image to the corresponding point of the reference image when the homography matrix H is used. Of the homography matrix H shown as the coordinate transformation matrix in FIG. 7B, the element h33 is a scale factor. Therefore, when normalized with h33 = 1, the remaining eight elements are obtained to obtain the coordinate transformation matrix. Is obtained. Therefore, elements h11, h12, h13, h21, h22, h23, h31, h32 are assigned by substituting a predetermined number or more as corresponding reference image feature points (X, Y) and scene image feature points (x, y). Can be obtained.

  When the scene distance image is selected in step S20, the reference distance image feature point (X, Y) and the scene distance image feature point (x, y) are substituted into the predetermined conversion formula shown in FIG. A coordinate transformation matrix is obtained in the same manner as when a scene distance image is selected.

  Next, using the generated coordinate conversion matrix, the coordinates of the gazing point corresponding to each scene image are converted to the coordinates on the reference image, and the corresponding points are calculated (step S40). Specifically, the gaze point on the scene image is coordinate-transformed to the corresponding point on the reference image using the coordinate transformation matrix created in step S30. FIG. 8 is a diagram illustrating a state in which the gazing point E of one scene image C is coordinate-converted to a corresponding point T on the reference image K. Since the coordinates of the reference image K and the reference distance image KK are matched from the beginning, the corresponding point TK on the reference distance image KK can be processed as it is on the reference image K as it is. A plurality of corresponding points can be obtained on the reference image by performing coordinate conversion of the gazing point of each scene image on the reference image.

  FIG. 9 is a diagram illustrating the relationship between the reference image K, the corresponding point T, and the corresponding point TK. As described above, the reference image K is an image in which the state from the front of the object 300 is recorded, and is only one image. When the scene images C1, C2, C5, C8 and the scene distance images CK3, CK4, CK6, CK7, CK9, CK10 are selected as the selection images, the line-of-sight analysis apparatus 100 displays the gazing points E1, E2, E5, E8, Coordinate conversion is performed using the coordinate conversion matrix corresponding to each of the scene images C1, C2, C5, and C8, and the converted coordinates are set as corresponding points T1, T2, T5, and T8. Also, the gazing points E3, E4, E6, E7, E9, and E10 are coordinate-transformed using the coordinate transformation matrix corresponding to each scene distance image CK3, CK4, CK6, CK7, CK9, and CK10, respectively, and the transformed coordinates Are the corresponding points TK3, TK4, TK6, TK7, TK9, TK10. In FIG. 9, “• (black circle, dot)” in the reference image K indicated by a rectangular frame indicates the positions of the corresponding points T1 to TK10. In the example of FIG. 9, only a rectangular frame is shown as the reference image K, but actually, an image from the front of the object 300 is recorded.

  In the example of FIG. 9, a total of ten corresponding points T and TK corresponding to each scene image and each scene distance image are obtained on one reference image. In the example of FIG. 9, a case where the corresponding points are all different from each other in coordinates (pixels), that is, a case where a plurality of corresponding points do not overlap one coordinate (pixel) is shown. When a plurality of corresponding points overlap the same coordinate (pixel), the number of corresponding points corresponding to one coordinate (pixel) is counted and recorded.

  Subsequently, a heat map is created using the reference image and the corresponding points T and TK. Specifically, by executing clustering by dividing the coordinates of each reference image K into predetermined areas, or by performing processing such as expanding a colored area other than the corresponding points using a Gaussian filter or the like, the heat map is create. As a method for creating a heat map using the coordinates (pixels) of a specific point such as a corresponding point, various known methods can be used. The created heat map is based on the image captured by the camera and the distance measured by the distance sensor, so that the more suitable gazing point is expressed, so the object is three-dimensional and has many curved surfaces. Even when there are many areas with few features, or even when the object is imaged in a dim place, it is possible to identify the spot on the object that the viewer has noticed.

  An example of a heat map created by the line-of-sight analysis system according to this embodiment is shown in FIG. When a lot of corresponding points are included in a certain region, it becomes a red color representing heat, and changes to yellow and green as the number of corresponding attention points decreases. In the example of FIG. 10, a color corresponding to heat (the number of corresponding points) is overwritten on a leaflet that is an object captured from the front as a reference image.

  In the above embodiment, the case where the object is a printed flyer has been described. In particular, even on a flat surface such as a leaflet printed matter, the effect of the present invention can be obtained when there are many areas with few features or when browsing in a dim place. Further, as described above, the present invention is also effective when a three-dimensional object with many curved surfaces is imaged in a dim place. An example of this is the interior of a seat space of an automobile.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above embodiments, and various modifications can be made. For example, in the above embodiment, in the eyeglass-type line-of-sight tracking apparatus 200, the scene camera 203 and the distance sensor 206 are installed adjacent to each other in the left-right direction at a position corresponding to one lens, but are installed adjacent to each other in the vertical direction. Or may be set apart from the left and right ends. If possible, it is preferable to install them adjacent to each other.

10a: CPU (Central Processing Unit)
10b ... RAM (Random Access Memory)
10c: Storage device 10d: Instruction input unit 10e: Data input / output I / F
DESCRIPTION OF SYMBOLS 10f ... Display part 100 ... Gaze analysis apparatus 200 ... Gaze tracking apparatus 201 ... Infrared light source 202 ... Infrared sensor 203 ... Scene camera 204 ... Processing part 206 ... Distance sensor 207 ... Infrared 208 ... Infrared (reflected light)
300 ... Object

Claims (4)

The system includes a camera that captures an object, a distance sensor that measures a distance from the object, and a gaze tracking device that acquires a gaze point of a subject, and a gaze analysis device that can perform data communication with the gaze tracking device. And
The line of sight analyzer is
A scene image captured by the camera, a scene image acquisition means for acquiring a gazing point on the scene image,
Scene distance image acquisition means for acquiring a scene distance image based on the distance measured by the distance sensor;
Scene image feature point extracting means for extracting a plurality of scene image feature points from the scene image;
Scene distance image feature point extracting means for extracting a plurality of scene distance image feature points from the scene distance image;
A scene image associating means for associating a reference image feature point in the reference image prepared in advance with the scene image feature point;
A scene distance image associating means for associating a reference distance image feature point with the scene distance image feature point in a reference distance image prepared in advance;
Of the scene image and the scene distance image, the image with the larger number of correspondences is selected as the selected image, and the coordinates of both are calculated based on the correspondence between the feature point of the selected image and the reference image or the reference distance image. A transformation matrix creation means for creating a coordinate transformation matrix to be transformed for each selected image;
Using the generated coordinate transformation matrix, for each of the plurality of scene images, corresponding points for converting corresponding points of the gazing point associated with the scene image into coordinates on a reference image and calculating corresponding points A calculation means;
A line-of-sight analysis system characterized by comprising:   The line-of-sight analysis system according to claim 1, wherein the line-of-sight tracking apparatus synchronously performs imaging by the camera and measurement by the distance sensor. A camera that captures an object; and a distance sensor that measures a distance from the object; a scene image captured by the camera; a scene distance image based on the distance measured by the distance sensor; and a scene image A gaze analyzer connected to the gaze tracking device for acquiring the gaze point of the subject above and capable of data communication,
A scene image captured by the camera, a scene image acquisition means for acquiring a gazing point on the scene image,
Scene distance image acquisition means for acquiring a scene distance image based on the distance measured by the distance sensor;
Scene image feature point extracting means for extracting a plurality of scene image feature points from the scene image;
Scene distance image feature point extracting means for extracting a plurality of scene distance image feature points from the scene distance image;
Reference image feature point acquisition means for acquiring a plurality of reference image feature points in the reference image;
Reference distance image feature point acquisition means for acquiring a plurality of reference distance image feature points in the reference distance image;
Scene image associating means for obtaining a correspondence relationship between the scene image feature points and the reference image feature points;
Scene distance image associating means for obtaining a correspondence relationship between the scene distance image feature point and a reference distance image feature point;
Of the scene image and the scene distance image, an image having a larger correspondence relationship is selected as a selected image, and based on the correspondence relationship between the feature point of the selected image and the reference image or the selected feature point, the coordinates of both are selected. A transformation matrix creation means for creating a coordinate transformation matrix for transforming each selected image;
Using the generated coordinate transformation matrix, for each of the plurality of scene images, corresponding points for converting corresponding points of the gazing point associated with the scene image into coordinates on a reference image and calculating corresponding points A calculation means;
A line-of-sight analysis apparatus comprising:   A program for causing a computer to function as the line-of-sight analysis apparatus according to claim 3.