JP7640427B2 - Gaze position analysis system and method - Google Patents

Description

The present invention relates to a gaze position analysis system and a gaze position analysis method.

When analyzing a user's interests, work status, etc. from gaze information acquired using a gaze measurement device worn by the user, the user moves freely in a three-dimensional space in which various objects are placed. For this reason, it is desirable to be able to confirm where in the three-dimensional space the user is gazing, and how that transition occurs.

Patent Documents 1 and 2 disclose technology that uses a gaze measurement device worn by a user to analyze gaze information acquired while the user moves freely in three-dimensional space as the gaze position in the three-dimensional space.

Patent document 1 discloses a technology that generates a virtual three-dimensional space using images captured from multiple different shooting positions, calculates the shooting positions in the virtual three-dimensional space, and calculates the user's gaze position and gaze time in the virtual three-dimensional space from the user's line of sight obtained at the time the images were captured.

On the other hand, Patent Document 2 discloses a technology that identifies an object that a user is focusing on based on the position of the display device in three-dimensional space, the user's line of sight, and the position of the object in three-dimensional space.

JP 2020-135737 A JP 2018-195319 A

In Patent Document 1, the virtual three-dimensional space generated from multiple images may contain sparse data or may only be partially generated, making it difficult to determine what the subject was looking at.

On the other hand, in Patent Document 2, it is assumed that the user's line of sight in three-dimensional space and the positions of objects in the three-dimensional space can always be obtained, which poses the problem that the equipment required is large-scale and the scope of application is limited. In addition, the positions of all objects to be analyzed must be clearly defined in advance, which requires a lot of effort in advance preparation.

When analyzing gaze position in three-dimensional space, it is desirable to be able to view how the gaze position has transitioned across the entire three-dimensional space, as well as to focus on a specific object and easily view how the gaze position has transitioned over that object.

If it were possible to associate the gaze position with a three-dimensional model of each object that exists in three-dimensional space, it would be possible to perform a more detailed analysis. Furthermore, if each object model could be automatically positioned in three-dimensional space, it would be possible to simplify the pre-settings required for analysis.

The object of the present invention is to automatically associate a user's gaze position with a three-dimensional model corresponding to each object existing in a three-dimensional space in a gaze position analysis system.

The gaze position analysis system of one aspect of the present invention has a spatial model creation unit, an object position and orientation estimation unit, and a gaze position calculation unit, and is a gaze position analysis system that automatically matches an object model gaze position, which is a user's gaze position on an object model, on an object model that is a three-dimensional model of an object existing in space. The spatial model creation unit acquires a first-person video that is an image similar to the user's field of vision, acquires a first-person video gaze position that is the gaze position of the user on the first-person video, creates a spatial model from the first-person video that is a three-dimensional model of the space in the range in which the user directs his or her gaze, and calculates the gaze position of the first-person video in the spatial model. The object position and orientation estimation unit estimates the position and orientation of the object in the spatial model by matching the object model with the spatial model, places the object model in the spatial model using the position and orientation of the object in the spatial model, calculates the gaze direction in the spatial model using the shooting position and orientation of the first-person video in the spatial model and the first-person video gaze position, and the gaze position calculation unit calculates the object model gaze position by determining the intersection of the gaze direction and the object model in the spatial model.

The gaze position analysis system according to one aspect of the present invention is a gaze position analysis system that automatically matches a user's gaze position on an object model, which is a three-dimensional model of an object existing in space, and is characterized by having a spatial model creation unit that creates a spatial model, which is a three-dimensional model of the surrounding space, from a plurality of captured images, an object position and orientation estimation unit that matches the spatial model with the object model and places the object model on the spatial model based on the position and orientation obtained by the matching, and a gaze position calculation unit that calculates the gaze position on the object model based on the placed object model and the gaze direction in the spatial model.

According to one aspect of the present invention, a gaze position analysis system can automatically map a user's gaze position onto a three-dimensional model that corresponds to each object that exists in a three-dimensional space.

FIG. 2 is a diagram showing the configuration of a general computer when the gaze position analysis system according to the embodiment of the present invention is executed by the general computer. 1 is a diagram illustrating a basic configuration of a gaze detection device according to the present invention. FIG. 1 is a diagram illustrating an example of a format of first-person video data. FIG. 11 is a diagram illustrating an example of a format of gaze position data. FIG. 2 is a diagram illustrating an example of a format of object model data. FIG. 2 is a diagram illustrating an example of a format of spatial model data. FIG. 11 is a diagram illustrating an example of a format of object placement data. FIG. 2 is a diagram illustrating the contents of a process executed by a spatial model creation program. FIG. 11 is a diagram illustrating an example of a flowchart of a process executed by a gaze position calculation program. 11A to 11C are diagrams illustrating the contents of a process executed by a gaze position calculation program. FIG. 1 is a diagram showing an example of an AR marker placed on an actual object. FIG. 1 is a diagram showing an example of an object model whose shape and size change. FIG. 2 is a diagram showing an example of an object model and a gaze position on the object model. FIG. 13 is a diagram showing an example of a case where the gaze position is always displayed facing the user. 13 is a diagram showing an example of a case where the gaze position is displayed by restricting the adjustment method for the position and orientation of an object model. FIG.

The following describes an embodiment of the present invention with reference to the drawings.

FIG. 1 is a diagram showing the configuration of a general computer that executes a gaze position analysis system according to an embodiment of the present invention.
The gaze measurement device 101 in Figure 1 is an input device that measures a first-person image, which is an image similar to the user's field of vision, and the gaze position on the first-person image, and records them in a database that stores first-person image data 110 and gaze position data 111, respectively, and can use a device that is commonly used and known by names such as an "eye tracker."

In particular, the gaze measurement device 101 of the present invention is assumed to be a portable device that can be worn by the user. This makes it possible to measure the gaze direction of the user while they are able to move freely in space.

FIG. 2 shows a basic configuration of the gaze detection device 101 that is worn by a user.
2, 201 is a photographing device for acquiring a first-person video of a user, and a device equivalent to a camera generally used in a personal computer or the like can be used. 202 is a glasses-type device equipped with a sensor for detecting the movement of the user's eyes and measuring the gaze direction. 203 is a terminal for recording the acquired first-person video data 110 and gaze position data 111 in a database. The terminal 203 may transmit data to the information processing device 102.

The gaze measurement device 101 is not limited to a portable type, but may be a stationary type device as long as it is capable of measuring the gaze position while the user is able to move freely through the space and is capable of acquiring the gaze position on a first-person image.

The information processing device 102 in FIG. 1 is an information processing device for executing each program in the gaze position analysis system.

The input device 103 includes typical computer input devices such as a keyboard, buttons, a mouse, or a touch panel for controlling the start and end of the system.

The output device 104 is a means for displaying the results of gaze position analysis and the system's operating status to the user, and may include the screen of a smartphone or tablet device, or a display device for a general computer.

105 is a storage device for storing each program in the gaze position analysis system. The storage device 105 includes a space model creation program 106, an object position and orientation estimation program 107, a gaze position calculation program 108, and a gaze position display program 109.

Here, the information processing device 102 functions as a space model unit by executing processing according to a space model creation program 106. The information processing device 102 also functions as an object position/orientation estimation unit by executing processing according to an object position/orientation estimation program 107. The information processing device 102 also functions as a gaze position calculation unit by executing processing according to a gaze position calculation program 108. The information processing device 102 also functions as a gaze position display unit by executing processing according to a gaze position display program 109.

The database of first-person video data 110 stores data on first-person video, which is video similar to the user's field of vision. It is assumed that the first-person video data 110 is acquired by the gaze measurement device 101 as described above.

FIG. 3 shows an example of the format of the first-person video data 110.
The data name 301 in Fig. 3 is a name given to the first-person video data, and can be any string of characters and symbols. The first-person video data 110 includes a plurality of images captured at predetermined time intervals or at any timing in the order in which they were captured, and the number of data 302 in Fig. 3 indicates the number of images included in the data.

Time 1 in 303 indicates the time when the first image was acquired, image 1 in 304 indicates the first image acquired, and shooting position and orientation 1 in 305 indicates the position and orientation of the camera that captured the first image. Time n in 306 indicates the time when the nth image was acquired, image n in 307 indicates the nth image, and shooting position and orientation n in 308 indicates the position and orientation of the camera that captured the nth image. Note that shooting positions and orientations 305 and 308 are data calculated by the spatial model creation program 106, as described below, and are blank when each image is acquired.

The format of the first-person video data 110 may be any format, such as a commonly used video format, as long as it is a format that allows easy acquisition of images at each time in the data. The gaze position data 111 database stores the gaze position data acquired by the gaze measurement device 101.

FIG. 4 shows an example of the format of the gaze position data 111. As shown in FIG.
As described above, since it is assumed that the gaze position is represented as position coordinates on the first-person video data 110, the gaze position data 111 has corresponding first-person video data 110. Therefore, the correspondence between the gaze position data 111 and the first-person video data 110 is expressed by describing the same name as the data name 301 in the corresponding first-person video data in the data name 401.

The gaze position data 111 also includes multiple position coordinates acquired at predetermined time intervals or at arbitrary timing in the order in which they were acquired. The number of data 402 in FIG. 4 indicates the number of position coordinates included in the data. 403 is the time when the gaze position coordinates were first acquired, 404 is the position coordinates of the first acquired gaze position, and 405 describes the name of the object when the gaze position data is associated with an object model by the gaze position calculation program 108.

406 is the time when the position coordinates of the gaze position were acquired for the nth time, 407 is the position coordinates of the gaze position acquired for the nth time, and 408 describes the name of the object when the gaze position data is associated with an object model by the gaze position calculation program 108. At the time when the gaze position data is acquired, there is no associated object model, so 405 and 408 are left blank.

In addition, when the gaze position data 111 is acquired, the gaze position is a position coordinate on the first-person video, i.e., two-dimensional coordinate data, but when the gaze position data 111 is associated with an object model, it becomes a position coordinate on the object model and is rewritten into three-dimensional coordinate data.

The gaze position data 111 described above is based on the premise that the gaze position data acquired by the gaze measurement device 101 and the gaze position data associated with the object model are managed as the same data. However, the gaze position data acquired by the gaze measurement device 101 and the gaze position data associated with the object model may be managed as separate data.

The database of object model data 112 stores three-dimensional models of objects that exist in the space within the user's line of sight. In this invention, it is assumed that the three-dimensional model of each object is represented as a collection of points that represent the shape of the object, i.e., point cloud data.

Object models created using general 3D CAD and the like are often represented as a collection of polygons, but object models represented as polygons can be easily converted into point cloud models. For example, first, for each polygon, divide the polygon into triangles with lines connecting a vertex to all vertices other than the one adjacent to that vertex, and then repeat the process of dividing each triangle with a line connecting a vertex to the midpoint of the opposite side, and finally select the vertices of all triangles, allowing the object model represented as polygons to be represented as point cloud data.

FIG. 5 shows an example of the format of an object model stored in the database of the object model data 112. In FIG.
5, 501 is a name given to an object, and can be any string of characters and symbols. 502 indicates the number of points included in the object model, 503 indicates the position coordinates of the first point, and 504 indicates the position coordinates of the nth point. The format shown in FIG. 5 includes only the position coordinates of the points, but it is also possible to include information associated with each point, such as information about color.

The spatial model data 113 database stores a spatial model, which is a three-dimensional model of the space within the user's line of sight. The spatial model is data created by the spatial model creation program 106 using the first-person video data 110, and is assumed to be represented as point cloud data in the same way as the object model described above.

FIG. 6 shows an example of the format of a spatial model stored in the database of the spatial model data 113.
The model name 601 in Fig. 6 is a name given to the spatial model, and can be any string of characters and symbols. 602 is the name of the first-person video data used to create the spatial model, 603 is the number of points included in the spatial model, 604 is the position coordinate of the first point, and 605 is the position coordinate of the nth point. The format shown in Fig. 6 includes only the position coordinates of the points, but information associated with each point, such as information about color, may also be included.

The object placement data 114 database stores data on object models that are matched to a spatial model and placed on the spatial model by the object position and orientation estimation program 107.

FIG. 7 shows an example of the format of the object placement data 114 stored in the database of the object placement data 114. In FIG.
Assuming that the object placement data 114 is stored for each spatial model, the name of the spatial model corresponding to the model name 701 in Fig. 7 is described. The number of objects 702 indicates the number of object models placed in the target spatial model.

In FIG. 7, 703 indicates the name of the first object placed on the spatial model, 704 indicates the position and orientation of the first object on the spatial model, and 705 indicates the time when the first object was placed on the spatial model. 706 indicates the name of the nth object placed on the spatial model, 707 indicates the position and orientation of the nth object on the spatial model, and 708 indicates the time when the nth object was placed on the spatial model.

The present invention performs analysis processing of the gaze position on each object using a spatial model creation program 106, an object position and orientation estimation program 107, and a gaze position calculation program 108 from first-person video data 110 and gaze position data 111 acquired from the gaze measurement device 101. In particular, this embodiment assumes that processing is performed while first-person video data 110 and gaze position data 111 are acquired in real time from the gaze measurement device 101.

For this reason, the spatial model creation program 106 first performs a process of constantly reading newly saved data from the data acquired from the gaze measurement device 101 and stored in the database of first-person video data 110, and creating a spatial model.

A technique that can be used to create a spatial model represented by point cloud data from video data is a combination of the well-known SLAM (Simultaneous Localization and Mapping) and MVS (Multi-View Stereo) methods.

SLAM is a technology that creates coarse point cloud data by analyzing the correspondence between images using multiple consecutive images that correspond to the movement of the camera. On the other hand, the MVS method is a technology that creates more detailed, dense point cloud data by using the analysis results of SLAM.

Figure 8 shows an example of how a spatial model is created using point cloud data with the SLAM and MVS methods.

In Figure 8, 801 represents an object that exists within the range where the user's line of sight moves, and for simplicity, it is assumed that there are no other objects in the vicinity. 802 and 804 represent the camera positions when the first-person video was captured, and 803 and 805 illustrate the range of the camera's field of view and the camera's attitude when the first-person video was captured from each camera position.

As shown in 802 and 804, by using images of the same object or the same location taken from multiple locations, a spatial model represented by point cloud data as shown in 806 is created.

In FIG. 8, only two camera positions are shown, but more images are generally used to create a more accurate spatial model 806. By using the above-mentioned techniques, the spatial model 806 can be created in real time in response to changes in the user's gaze position. In addition to the above-mentioned techniques, any other technique may be used in the spatial model creation program 106 as long as it can create a spatial model similarly represented by point cloud data.

Furthermore, by using SLAM technology, the position and orientation of the camera that captured the first-person video on the created spatial model can also be calculated at the same time. As described below, the position and orientation of the camera is information required to calculate the gaze position on the object model. For this reason, when using a spatial model creation technology that cannot calculate the position and orientation of the camera, it is necessary to use a separate means of acquiring the position and orientation of the camera on the spatial model. For example, any technology that acquires the position and orientation in space can be used, such as using a sensor that acquires the position and orientation.

The object position and orientation estimation program 107 is a program for matching the point cloud data of the object model to a portion of the point cloud data in the spatial model created by the spatial model creation program 106, i.e., for adjusting the position and orientation so that they match well, and for determining the position and orientation of the object model in the spatial model.

The object position and orientation estimation program 107 may use well-known techniques such as the ICP (Iterative Closest Point) algorithm or the NDT (Normal Distribution Transform) algorithm. Alternatively, any technique may be used as long as it can match point cloud data and determine the position and orientation of an object model in a spatial model. In addition, the object position and orientation estimation program 107 controls the timing of processing by the gaze position calculation program 108, which will be described later.

The gaze position calculation program 108 is a program that calculates the gaze position on the object model using gaze position data 111 acquired by the gaze measurement device 101, object model data 112, and spatial model data 113 created by the spatial model creation program 106.

The flow of processing of the gaze position calculation program 108 will be described with reference to the flowchart of FIG.
9, new gaze position data 111 is obtained from the gaze position data 111 stored in the database of the gaze position data 111. Alternatively, new gaze position data may be obtained directly from the gaze measurement device 101.

In step 902, the gaze direction on the spatial model is calculated using the newly acquired gaze position data 111 and the shooting positions and attitudes 305 and 308 calculated when creating the spatial model by the spatial model creation program 106 and recorded in the first-person video data 110.

The gaze direction on the spatial model is expressed by a vector indicating the starting point of gaze and the line of sight from that position. To calculate the gaze direction on the spatial model, first, the shooting position and orientation for the first-person video data at the same time as the newly acquired gaze position data 111 is acquired from the first-person video data 110.

If there is no shooting position and orientation at the exact same time, a method may be used in which, for example, a shooting position and orientation for the first-person video data 110 corresponding to a time before or after the time of the gaze position data 111 is acquired, and the shooting position and orientation obtained by performing interpolation based on the relationship with the time of the gaze position data 111 is used. Next, the acquired shooting position and orientation are used to perform coordinate conversion of the gaze position on the first-person video to the gaze position on the spatial model.

As described above, the gaze position data 111 is represented by the gaze position on the first-person video, but by applying information such as the camera's viewing angle and focal length to a well-known pinhole camera model, the positional relationship between the shooting position, first-person video, and gaze position can be obtained at the same size as in the actual space. Furthermore, by performing coordinate conversion on the obtained positional relationship using the shooting position and orientation obtained from the first-person video data 110, the positional relationship between the camera's shooting position, first-person video, and gaze position on the spatial model can be expressed as shown in FIG. 10.

In FIG. 10, 1001 indicates the spatial model, 1002 indicates the shooting position of the camera that captured the first-person video on the spatial model 1001, and 1003 indicates the shooting range of the first-person video on the spatial model 1001, which corresponds to the shooting posture of the camera.

1004 is the gaze position on the first-person image that is associated with the spatial model 1001. The gaze direction on the spatial model 1001 can be obtained as vector 1005 in FIG. 10 that starts at 1002 and passes through gaze position 1004 on the first-person image.

In step 903 in FIG. 9, it is determined whether an object model placed on the spatial model 1001 exists in the gaze direction on the spatial model 1001 determined in step 902. If it is determined that an object model placed on the spatial model 1001 exists in the gaze direction on the spatial model 1001, the process proceeds to step 906, and if it is determined that an object model does not exist, the process proceeds to step 904.

The method of making this determination is as follows: first, information on the object model placed in the target spatial model 1001 is obtained from the object placement data 114, and the point cloud data of the corresponding object model is read from the object model data 112 and placed on the spatial model. Next, the distance between a vector representing the gaze direction on the spatial model 1001 (hereinafter, gaze direction vector), for example, vector 1005 in FIG. 10, and each point of the object model placed on the spatial model 1001 is calculated, and the distance closest to the gaze direction vector is selected.

If the selected distance is equal to or less than a predetermined threshold, it can be determined that the target object model is in the gaze direction. As the threshold for distance, for example, the largest distance or half the largest distance between points in the point cloud data constituting the object model can be selected. In addition, as a method of determination, it may be determined that the target object model is in the gaze direction when there are a number of points on the spatial model whose distance from the gaze direction vector is equal to or less than a predetermined threshold equal to or more than a predetermined threshold.

In addition, in addition to the point cloud data, polygon data, which is data in a format represented by a collection of polygons, may also be stored as object models to be stored in the database of object model data 112, and the polygon data for the corresponding object model may also be arranged on the spatial model 1001. If there is a point where each polygon arranged on the spatial model 1001 intersects with a vector representing the line of sight arranged on the spatial model, it may be determined that the target object model is in the gaze direction.

Furthermore, if there are multiple objects placed on the spatial model 1001, the point closest to the gaze direction vector 1005 or the position on the gaze direction vector of the polygon that intersects with the gaze direction vector 1005 can be obtained, and the starting point of the gaze direction vector 1005, i.e., the point or object model including the polygon closest to the shooting position, can be selected as the object model existing in the gaze direction.

Furthermore, a virtual space that can express the same positional relationship as the spatial model 1001 may be prepared, and each time an object is placed on the spatial model 1001, an object model may be placed in the virtual space, and the above-mentioned determination process may be executed in the virtual space.

In step 904, the object position and orientation estimation program 107 is used to match the object model 1004 to the point cloud data on the space model 1001 that exists in the gaze direction on the space model 1001. This process is a process for making it possible to replace a part of the space model 1001 with the target object model 1006 by adjusting the position and orientation of the object model 1006 in various ways relative to the space model 1001 in FIG. 10.

As point cloud data on the spatial model 1001 existing in the gaze direction, for example, point cloud data on the spatial model whose distance from the gaze direction vector 1005 is equal to or less than a predetermined distance can be selected. The predetermined distance can be the maximum size of the object model to be matched or half the maximum size, etc.

Alternatively, the point cloud data may be present within a range of a specific shape and a predetermined size with the gaze direction vector 1005 as the central axis. Also, the position of each point in the point cloud data on the gaze direction vector may be obtained, and points within a predetermined range may be selected centered on the point where the point cloud data is most concentrated on the gaze direction vector. In addition to the above, any method may be used as long as it is capable of selecting point cloud data to be matched with the gaze direction vector as the center.

The object models to be matched to the point cloud data on the spatial model can be all object models stored in the database of object model data 112, or well-known QR codes (registered trademark), AR markers, symbols or character strings representing the names of objects, etc. can be placed on the actual objects, and these can be read using well-known image recognition technology or character recognition technology, and the corresponding object models can be selected from the object model data 112 for matching.

1101 in FIG. 11 shows an example of an AR marker placed on an actual object. Alternatively, well-known point cloud features (three-dimensional features) may be extracted from the point cloud data on the spatial model and the point cloud data of the object model, and an object model having point cloud features similar to the point cloud data on the spatial model may be selected from the object model data 112 for matching.

In addition, when performing assembly work, the shape and size of an object present at a specific location change. In such a case, as shown in 1201, 1202, and 1203 in FIG. 12, the object model showing the state of the object for each work step and its sequential relationship are stored in the object model data 112, and matching with the spatial model is performed in sequence, making it possible to match the object model according to the work step with the spatial model.

Specifically, for example, if no object model is matched to the point cloud data on the spatial model that exists in the gaze direction, the object model corresponding to the first step is selected and matching is performed.

On the other hand, if an object model corresponding to one of the processes has already been matched to the point cloud data on the spatial model present in the gaze direction, the object model corresponding to the next process is selected and matched with the point cloud data on the spatial model. If the object model corresponding to the next process is matched to the spatial model with a higher degree of match than the already matched object model and spatial model, the already matched object model is replaced with the object model corresponding to the next process.

The degree of matching may be the same as that used in the processing in step 905 described below. Furthermore, matching of the object model corresponding to the next step to the spatial model may be performed when the gaze position moves onto an already matched object model or when a change in the spatial model is detected in the spatial model creation program 106.

To detect a change in the spatial model, the spatial model creation program 106, for example, compares a spatial model generated using a newly acquired first-person video image and a predetermined number of first-person video images immediately preceding that image with the spatial model generated previously, and if the difference between the two is equal to or greater than a predetermined threshold, it is determined that the spatial model has changed.

The difference between spatial models can be calculated, for example, by searching for the closest point in the other spatial model for each point in one spatial model, calculating the distance to the searched point, and averaging the distances for all points in one spatial model. Alternatively, the number of points whose calculated distance is equal to or greater than a predetermined threshold value may be used. Alternatively, any method may be used as long as it can calculate the difference between point groups.

In step 905, it is determined whether the object model has been correctly matched to the point cloud data on the spatial model that exists in the gaze direction on the spatial model. If the match is correct, the matching result of the object model is stored in the database of object placement data 114 and then the process proceeds to step 906; if not, the process proceeds to step 908.

A method for determining whether an object model has been correctly matched to point cloud data on a spatial model is to calculate the degree of match between the point cloud data on the spatial model and the object model placed on the spatial model based on the results of the matching process, and if the degree of match is greater than a predetermined value, it is determined that the object model has been correctly matched.

If there are multiple object models that can be matched to the point cloud data on the spatial model, the highest degree of matching is selected, and if the selected degree of matching is greater than a predetermined value, it is determined that the object models have been matched correctly, and that the object model corresponding to the selected degree of matching has been matched to the point cloud data on the spatial model.

The degree of agreement can be determined, for example, by searching for the point in the point cloud data on the spatial model that is the shortest distance from each point in the object model placed on the spatial model using the matching results, determining the number of points whose distance is smaller than a predetermined threshold, and calculating the ratio of the number of points determined to the number of points in the object model. In addition to the above, any index can be used as the degree of agreement between the point cloud data on the spatial model and the object model, as long as it can determine whether the matching result between the point cloud data is good or bad.

In step 906, the matching result in step 904 is used to calculate the gaze position of the object model placed on the spatial model. The object model placed on the spatial model and the gaze direction vector on the spatial model are 1005 in FIG. 10, so a process similar to the process in step 903 described above, that is, the process of determining whether an object model placed on the spatial model exists in the gaze direction on the spatial model, can be used.

However, in step 903, when the object model is point cloud data, attention is paid to whether there is a point whose distance from the gaze direction vector 1005 is equal to or less than a predetermined threshold. However, in calculating the gaze position, a point whose distance from the gaze direction vector is equal to or less than a predetermined threshold and that is closest to the shooting position 1002 is selected and used as the gaze position.

Alternatively, if the distance from the gaze direction vector 1005 is equal to or less than a predetermined threshold and there are multiple points within a predetermined range from the point closest to the shooting position 1002, the gaze position may be the average of the point closest to the shooting position 1002 and the points within the predetermined range from that point.

Alternatively, if the object model is represented as a collection of polygons, the intersections between the gaze direction vector and each polygon in the object model can be found, and the intersection closest to the shooting position 1002 can be found as the gaze position.

Since the gaze position obtained as described above is the gaze position on the spatial model, in step 906, the determined gaze position is subjected to coordinate conversion using the position and orientation of the object model on the spatial model obtained by the matching result in step 904. As a result, the gaze position 1004 on the spatial model 1001 is converted to the gaze position 1007 on the object model 1006, i.e., in the coordinate system of the object model.

In addition, if an object model is placed on the spatial model in step 905, in step 906, the gaze position on the placed object model may be calculated based on past gaze position data. This is because, in the initial stages of processing of the gaze position calculation program 108, there may be a small amount of point cloud data for each object on the spatial model, and it may not be possible to place the object model on the spatial model.

In step 907, the gaze position on the object model determined in step 906 is stored in gaze position data 111. When storing the gaze position of the object model, the shooting position and orientation of the first-person video on the spatial model, the gaze position in the spatial model, the position and orientation of the object model on the spatial model, etc. may also be stored together.

In step 908, if an instruction to end the process is received from the input device 103 or the like, the process ends; if not, the process returns to step 901.

In the gaze position calculation program 108 described above, the first-person video data 110 and gaze position data 111 are acquired in real time, and the object model data 112 is placed on the spatial model while generating the spatial model data 113, and the gaze position on the object model is calculated. However, after the acquisition of the target first-person video data 110 and gaze position data 111 has been completed, the processing of the gaze position calculation program 108 may be executed while sequentially reading the data stored in the first-person video data 110 and gaze position data 111.

In this case, the gaze position on the object model may be calculated after creating a spatial model, or after creating a spatial model and placing an object model on the spatial model. In addition, in a location on the spatial model where no object model is placed, the object model may be manually placed, and then the above-mentioned processing may be performed.

Alternatively, after executing the processing of the gaze position calculation program 108, the gaze position calculation program 108 may be executed again using the spatial model data 113 and object placement data 114 generated after executing the gaze position calculation program 108. This is assumed to be the case in the early stages of processing of the gaze position calculation program 108, when there is little point cloud data for each object on the spatial model and it is not possible to place the object model on the spatial model.

In such a case, among the gaze positions stored in the gaze position data 111, there may be gaze positions on the object model that remain uncalculated. In this case, the gaze position calculation program 108 is executed after making the following changes.

The first change is that the gaze position data is acquired in step 901 by reading the data from gaze position data 111 in the order in which it was saved. In addition, a process is added to determine whether the gaze position on the object model has been calculated for the acquired gaze position data, and if so, to proceed to step 908.

The second change is that in step 903, if an object model is not located in the gaze direction on the spatial model, the process proceeds to step 908 instead of step 904.

The third change is to delete steps 904 and 905. By making the above three changes to the gaze position calculation program 108, it is possible to calculate the gaze position on the object model for gaze positions where the gaze position on the object model has not been calculated. In this case, the above-mentioned processing may be performed after manually placing an object model in a location where no object model is placed on the spatial model.

The gaze position display program 109 is a program that displays gaze position data in response to instructions from the user. The gaze position is displayed using methods well known in gaze analysis, such as a heat map method that shows the frequency distribution of gaze positions, or a method that visually shows the length of time that the gaze position remained within a certain range using the size of a circle, etc.

In addition to displaying the gaze position on the first-person image, the gaze position on the object model is also displayed. A method for displaying the gaze position on the object model can be to prepare a virtual space model, which is a virtual space corresponding to the spatial model, place an object model in the virtual space model based on the data stored in the object placement data 114, and display the gaze position on the object model placed in the virtual space model. Alternatively, a method can be used in which only specific object models are selected and the gaze position is displayed only on the selected object models.

Furthermore, when displaying the gaze position on an object model, the virtual space model or object model may be adjusted and displayed so that the gaze position always faces the user directly.

In FIG. 13, 1301 shows an example of an object model, and 1302, 1303, 1304, and 1305 show examples of gaze positions on the object model. Also, FIG. 14 shows an example of the gaze position shown in FIG. 13 always displayed facing the user.

In FIG. 14, 1401 is an object model in which the gaze point 1302 is displayed facing the user, and 1402 is the gaze point displayed on 1401. 1403 is an object model in which the gaze point 1303 is displayed facing the user, and 1404 is the gaze point displayed on 1403. 1405 is an object model in which the gaze point 1304 is displayed facing the user, and 1406 is the gaze point displayed on 1405. 1407 is an object model in which the gaze point 1305 is displayed facing the user, and 1408 is the gaze point displayed on 1407.

In the display method shown in FIG. 14, the gaze positions are displayed by adjusting the object model so that each gaze position faces completely directly at the user. To do this, for example, if the object model is point cloud data, a point in a predetermined range including the gaze position is selected, the direction of the normal of a plane including the gaze position is found from the selected point, and the position and orientation of the object model are adjusted so that the found normal faces the user. If the object model is represented as a collection of polygons, a polygon including the gaze position is selected from the object model, and the position and orientation of the object model are adjusted so that its normal faces the user.

In addition, when displaying the gaze position, restrictions may be placed on the method of adjusting the position and orientation of the displayed object model. Furthermore, by adjusting the degree to which the gaze position faces the user, large changes in the orientation of the object model when displaying the gaze position may be suppressed.

For example, the first gaze position is displayed so that it faces the user directly, and the position and orientation of the object model are adjusted for subsequent gaze positions so that the degree to which the gaze position faces the user does not exceed a predetermined range. The degree to which the gaze position faces the user can be determined by the angle or dot product between the normal direction of the plane including the gaze position and the direction facing the user, i.e., the direction perpendicular to the screen.

FIG. 15 shows an example of a display in which the gaze position is displayed using the above-mentioned method.
15 shows a case where an object model is displayed with its vertical coordinate axis tilted toward the user, and only rotation around the vertical coordinate axis of the object model is permitted as a method of adjusting the position and orientation when displaying the gaze position. The position and orientation of an object model 1501 is adjusted so that an initial gaze position 1502 faces the user directly.

However, in FIG. 15, the coordinate axis of the object model in the vertical direction is tilted, and the gaze position cannot be perfectly faced to the user, so the position and orientation are adjusted so that it faces as directly as possible. Specifically, the normal of the plane including the gaze position is found as described above, and the position and orientation of the object model is adjusted so that the difference between the found normal and the direction facing the user, i.e., the direction perpendicular to the screen, is minimized. In object model 1503, gaze position 1504 is displayed in a state where it faces the user less directly than gaze position 1502, i.e., the gaze position can be confirmed, but is not facing the user. Gaze position 1506 on object model 1505 and gaze position 1508 on object model 1507 are also displayed in the same way as gaze position 1504.

As described above, the gaze position analysis system in this embodiment of the present invention generates a spatial model, which is a three-dimensional model of the space in the range in which the user directs his or her gaze from a first-person video, which is an image similar to the user's field of vision, and calculates the shooting position and orientation of the first-person image in the spatial model, calculates the gaze direction on the spatial model using the shooting position and orientation of the first-person video on the spatial model and the gaze position on the first-person video, places an object model in the spatial model based on the position and orientation of the object on the spatial model estimated by matching the object model, which is a three-dimensional model of the object, with the spatial model, and calculates the gaze position on the object model by finding the intersection between the gaze direction on the spatial model and the object model.

According to an embodiment of the present invention, a gaze measurement device that is worn by a user is used, and the gaze information of the user measured while moving freely within the space being measured can be used to automatically map the gaze position of the user onto a three-dimensional model of various objects that exist within the space.

101 Gaze measurement device 102 Information processing device 103 Input device 104 Output device 105 Storage device 106 Space model creation program 107 Object position and orientation estimation program 108 Gaze position calculation program 109 Gaze position display program 110 First-person video data 111 Gaze position data 112 Object model data 113 Space model data 114 Object arrangement data

Claims (15)

A gaze position analysis system having a space model creation unit, an object position and orientation estimation unit, and a gaze position calculation unit, which automatically associates an object model gaze position, which is a gaze position of a user on an object model, with an object model, which is a three-dimensional model of an object existing in a space,
The spatial model creation unit includes:
Acquire a first-person image that is an image similar to the user's visual field;
A first-person image gaze position, which is a gaze position of the user on the first-person image, is acquired;
creating a spatial model, which is a three-dimensional model of the space in the range in which the user is looking, from the first-person image;
Calculating a shooting position and orientation of the first-person video in the spatial model;
The object position and orientation estimation unit
estimating a position and orientation of the object in the spatial model by matching the object model with the spatial model;
placing the object model in the spatial model using a position and orientation of the object in the spatial model;
Calculating a gaze direction in the spatial model using the shooting position and orientation of the first-person image and the first-person image gaze position in the spatial model;
The gaze position calculation unit
A gaze position analysis system, comprising: a gaze position analysis unit that calculates a gaze position of the object model by determining an intersection between the gaze direction and the object model in the spatial model. The object position and orientation estimation unit
The gaze position analysis system of claim 1, characterized in that if there is data in the spatial model to which the object model is not matched in the gaze direction, the position and orientation of the object in the spatial model is estimated by matching the object model with the data in the spatial model to which the object model is not matched. The object position and orientation estimation unit
selecting data in the spatial model that exists within a predetermined range based on the gaze direction from the data in the spatial model;
2. The gaze position analysis system according to claim 1, wherein the position and orientation of the object in the selected spatial model are estimated by matching the object model with data in the selected spatial model. The object position and orientation estimation unit
Reading a preset image pattern, character string, or symbol string from the first-person video;
selecting the object model corresponding to the image pattern, the character string, or the symbol string;
calculating a position of the image pattern, the character string or the symbol string in the spatial model from the spatial model, a shooting position and orientation of the first-person video in the spatial model, and a position of the image pattern, the character string or the symbol string in the first-person video;
selecting data in the spatial model that exists within the predetermined range based on a position in the spatial model of the image pattern, the character string, or the symbol string;
4. The gaze position analysis system according to claim 3, wherein the position and orientation of the object in the selected spatial model are estimated by matching the selected object model with data in the selected spatial model. The object position and orientation estimation unit
Calculating a difference between data representing a spatial model created from the first-person image at a certain time and data of a range of the spatial model already created that corresponds to the first-person image at the same time;
If the calculated difference is greater than a predetermined threshold, replacing the data in the corresponding range on the spatial model with data representing the spatial model created from the first-person video;
Matching the object model with the data of the replaced portion;
2. The gaze position analysis system according to claim 1, wherein the object model that was matched to the data before the replacement is replaced with the newly matched object model. further comprising an object model storage unit configured to store, for the object whose shape or structure changes, the object model corresponding to each process of the change and an order relationship of the change,
The object position and orientation estimation unit
The gaze position analysis system according to claim 1, characterized in that, for a location where the object model corresponding to the changing object is matched, each time a change in the spatial model is detected, the object model corresponding to the process of the change of the object is selected from the object model storage unit, and the selected object model is matched with the spatial model. The gaze position calculation unit
When any of the object models is newly matched to any location on the spatial model,
The gaze position analysis system according to claim 1 , further comprising: determining whether or not the object model gaze position is on the object model newly matched on the spatial model with respect to the previous object model gaze position. The object position and orientation estimation unit
The user manually matches the object model to a portion of the spatial model to which the object model has not been matched;
The gaze position calculation unit
The gaze position analysis system according to claim 1, characterized in that for all of the object model gaze positions already acquired, it is determined whether the object model gaze position is on the object model that the user manually matched on the spatial model. The gaze position calculation unit
The gaze position analysis system according to claim 1, characterized in that at least one of the name of the object model, the shooting position and orientation of the first-person image in the spatial model, the first-person image gaze position in the spatial model, and the position and orientation of the object model in the spatial model is stored in correspondence with the object model gaze position on the object model. a gaze position display unit that displays a gaze position of the object model on the object model,
The gaze position display unit is
The gaze position analysis system according to claim 1, characterized in that, when displaying the gaze position of the object model on the object model, the position and orientation of the object model are adjusted so that the gaze position of the object model at each time is always displayed directly in front. a gaze measurement device that captures the first-person image of the user and measures a gaze position of the first-person image while the user is able to move freely within the space;
The spatial model creation unit includes:
The gaze position analysis system according to claim 1 , wherein the first-person video and the first-person video gaze position are acquired from the gaze measurement device. The gaze measurement device includes:
A photographing device is provided.
The spatial model creation unit includes:
The gaze position analysis system according to claim 11, wherein the surrounding spatial model is created from a plurality of images captured by the imaging device. A gaze position analysis system that automatically associates a gaze position of a user on an object model, which is a three-dimensional model of an object existing in a space,
a spatial model creation unit that creates a spatial model, which is a three-dimensional model of the surrounding space, from a plurality of captured images;
an object position and orientation estimation unit that matches the spatial model with the object model and places the object model on the spatial model based on a position and orientation obtained by the matching;
a gaze position calculation unit that calculates the gaze position on the object model based on the arranged object model and a gaze direction in the space model;
A gaze position analysis system comprising: The gaze position calculation unit
The gaze position analysis system according to claim 13, wherein the gaze position on the object model is calculated by determining an intersection between the gaze direction in the spatial model and the object model. A gaze position analysis method for automatically associating an object model gaze position, which is a gaze position of a user on an object model, with an object model that is a three-dimensional model of an object existing in a space, comprising:
acquiring a first-person image similar to the user's field of view;
acquiring a first-person image gaze position, which is a gaze position of the user on the first-person image;
creating a spatial model, which is a three-dimensional model of a space in a range in which the user is looking, from the first-person image;
calculating a shooting position and orientation of the first-person video in the spatial model;
estimating a position and orientation of the object in the spatial model by matching the object model with the spatial model;
placing the object model in the spatial model using a position and orientation of the object in the spatial model;
calculating a gaze direction in the spatial model using the shooting position and orientation of the first-person image and the first-person image gaze position in the spatial model;
calculating an intersection of the gaze direction and the object model in the spatial model to calculate the object model gaze position;
A gaze position analysis method comprising the steps of:

Images