JPH07101461B2 - Eye-tracking high-speed image generation and display method - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a high-speed line-of-sight image generation and display method, and for example, a line-of-sight for displaying a high-speed and realistic image on a screen such as computer graphics. The present invention relates to a follow-up high-speed image generation / display method.

[Prior Art] A computer generates and displays a three-dimensional image,
The technology of so-called computer graphics is being used in various fields including simulation of scientific and technological calculations, but high precision, that is, high quality and high speed of images are in conflicting conditions, and at the same time There is currently no technology available. In other words, when trying to generate a high-quality image, a huge amount of calculation time is required for image generation, and conversely, for the purpose of high-speed, real-time display, for example, displaying several tens of images per second, rough calculations must be performed. And the quality of the generated image deteriorates.

On the other hand, the application field of computer graphics is expanding to the field of image communication called intelligent coded communication. In this method, the transmitting side extracts the three-dimensional structural information of the transmission target by image recognition, parameterizes the characteristics, and transmits the parameterized information. On the receiving side, three-dimensional structure information (database) is prepared in advance, and the three-dimensional database is converted at high speed and displayed based on the transmitted characteristic parameters. In these communication methods, the receiving side is
Since an image from an arbitrary viewpoint can be freely generated and displayed based on the dimensional structure information, it has many advantages as follows.

Motion can be realized by detecting the movement of the recipient's viewpoint and displaying an image corresponding to this movement. Here, kinetic vision is a person's potential ability to perceive space from the change in the image displayed on the retina of the eyeball when the head is moved. Trying to move and raise awareness is a common experience. That is, it is expected that the stereoscopic effect of the image on the screen will be improved by artificially realizing the motion vision by computer graphics, and the display with a high sense of presence can be realized.

Binocular stereoscopic vision can be realized by generating and displaying an image from each of the two eyes of the recipient as two viewpoints. Since both eyes of a person are separated by a distance of about 6 cm, separate images are projected on the retinas of each eye without moving the head. The correspondence between these two different images is called parallax information, and a person uses this to obtain a stereoscopic sense of space. By realizing the image having the parallax information by computer graphics, the stereoscopic effect of the image on the screen is further improved.

In addition, it is possible to easily generate an image in which the person on the other side (sending side) has a line of sight and has a conversation.

However, at present, there are some difficulties in realizing such a communication method. The first is image recognition and feature parameter extraction on the transmission side, and the second is an image generation and display method for displaying a wide field of view at high speed and high resolution on the reception side. In communication, real-time processing is an essential proposition, and there is a high demand for higher speed than conventional computer graphics.

The reason why high-speed and high-precision display is difficult will be described in detail below.

FIG. 21 is a diagram for explaining a conventional representative three-dimensional image generation / display method. In Fig. 21, the screen S1
Is represented by a display reference coordinate system X (w) -Y (w) -Z (w) whose origin is the center of the screen S1. In addition,
This coordinate system has a function of describing the positional relationship of the following coordinate systems, and may be provided in a different place from the screen S1. Viewpoint coordinate system X (e) -Y (e) -Z
(E) is a display reference coordinate system X (w) -Y (w) -Z
The position and the rotation angle of the coordinate axis are known from the viewpoint of (w). The display target object obj has a three-dimensional structure, and the target structural point Pi (obj) is the target coordinate system X (obj) −Y (obj) −.
Let it be represented by Z (obj).

In such a situation, when the structural point Pi is viewed from the viewpoint Oe, the position on the screen S1 at which the Pi point is displayed will be considered. Display reference coordinate system X (w) -Y (w)
-Z (w), target coordinate system X (obj) -Y (obj)
Since −Z (obj) is known, the target coordinate system X (ob
j) -Y (obj) -Z (obj)
j) is the display reference coordinate system X (w) -Y (w) -Z (w)
Then, it is shown as Pi (w) = M1 · P1 (obj).

Here, M1 is a transformation matrix for movement and rotation. Further, since it is assumed that the viewpoint coordinate system X (e) -Y (e) -Z (e) can also be represented by the display reference coordinate system X (w) -Y (w) -Z (w), If this conversion matrix is M2, the display reference coordinate system X (w) -Y (w)-
The structural point Pi (w) viewed from Z (w) is the viewpoint coordinate system X.
From the viewpoint of (e) -Y (e) -Z (e), it can be expressed as Pi (e) = M2 ⁻¹ · Pi (w) = M1 · M2 ⁻¹ · Pi (obj). In this way, the display reference coordinate system X
(W) -Y (w) -Z (w) and target coordinate system X (obj)-
Y (obj) -Z (obj) and display reference coordinate system X (w)-
Y (w) -Z (w) and viewpoint coordinate system X (e) -Y (e)-
If the relationship with Z (e) is known, the structure point Pi (obj) shown in the target coordinate system can be freely expressed in the viewpoint coordinate system.

Here, X (np) -Y (np) -Z (np) called a normal perspective coordinate system is considered in order to find the point on the screen of the structural point Pi (e) shown in the viewpoint coordinate system. This coordinate system has an origin O (np) on the Z (e) axis of the viewpoint coordinate system, and the regions A, B, called field pyramids hatched in the viewpoint coordinate system.
C, D, A (∞), B (∞), C (∞), D (∞) are rectangular parallelepiped A, B, C, D, A ′, (∞), B ′ (∞ ), C ′
There is a correspondence that matches (∞), D ′ (∞). It is assumed that the direction of the Z (np) axis of the rectangular parallelepiped is normalized to 1. That is, Z (np) = 0 point corresponds to h on the Z (e) axis,
The point of (np) = 1 corresponds to Z (e) = ∞. Under this condition, the rod-shaped object obj shown in FIG. 21 becomes the normal perspective coordinate system X (np) −
In Y (np) -Z (np), Z (n
The larger the p) axis, the smaller the cross section. The conversion between this viewpoint coordinate system and the normal perspective coordinate system is a 4 × 4 matrix M3.
Can be shown as Therefore, Pi (np) can be represented by Pi (np) = M3 · Pi (e). Here, when Pi (np) is parallel-projected on the A, B, C, and D planes, ξi (s) shown by a dotted line is obtained.

In FIG. 21, the normal perspective coordinate system X (n
p) -Y (np) -Z (np) is the display reference coordinate system X (w)-
It is assumed that the origin and the axial direction of Y (w) -Z (w) overlap. Therefore, the Z (e) axis of the viewpoint coordinate system is in the same direction as the Z (w) axis of the display reference coordinate system.

In this case, the rectangular parallelepipeds A, B, C and D correspond to the size of the screen. Therefore, ξi (s) is the viewpoint O (e)
It is a projected image on the screen seen from. In this way, three matrix operations and parallel projection are required from the structural point Pi (obj) to ξi (s).

Next, a case where the viewpoint deviates from the Z (e) axis of the display reference coordinate system will be described. In FIG. 21, O '(e) is the moved viewpoint position. With this moved point as the origin,
Consider a new viewpoint coordinate system X '(e) -Y' (e) -Z '(e) with the O' (e) -O (w) line as the Z '(e) axis. If the position of the new viewpoint coordinate system viewed from the display reference coordinate system X (w) -Y (w) -Z (w) is known, X (e)-
From Y (e) -Z (e) to X '(e) -Y' (e) -Z '
Conversion to (e) is possible via matrix M4.
Therefore, the structural point Pi seen from the new viewpoint coordinate system is Pi
(E) ′ = M4 · Pi (e) = M4 · M1 · M2 ⁻¹ · Pi (obj)

FIG. 22 is a view as seen from the Y (w) axis of FIG. Pi
Multiplying (e) ′ by the above-mentioned normal perspective projection transformation matrix M3, Pi ′ (np) is obtained.
i ^* (s) is obtained. Here, since the position of the new viewpoint coordinate system X ′ (e) −Y ′ (e) −Z ′ (e) viewed from the display reference coordinate system is known, ξi ^* (s) and O ′ (e) It is easy to find the point where the line connecting the and intersects the screen.
This point ξi ′ (s) is the projected image on the screen viewed from the viewpoint O ′. Thus, the structure point Pi (ob
From j) to ξi '(s), four sequence operations and parallel projection are required.

As described above, the display reference coordinate system X (w) -Y (w) -Z
In (w), the target coordinate system X (obj) -Y (obj) -Z
The display target in which the position of (obj) is described and the display target in which the coordinates of each structural point Pi (obj) in the target coordinate system are described can be perspectively projected on the screen from an arbitrary viewpoint. In addition, as described above, many matrix operations need to be performed for all the constituent points to be displayed, resulting in a huge calculation time. The operation of the matrix will be considerably faster by using hardware, but the number of constituent points will be several thousand.
Beyond the point, real-time processing becomes difficult. In addition, a lot of useless processing is performed depending on the position of the display target, limiting the processing speed. This is shown below. 23 is a view of the coordinate system shown in FIG. 21 on the Y (e) axis and in the Z (e) axis direction, and FIG. 24 is a diagram showing the display object shown in FIG. is there.

In FIG. 23, the screen S1 is composed of a large number of pixels gi. The display object has a three-dimensional structure as shown in FIG. 24, and is composed of a structural point Pi and a structural surface Li formed by the point. The case where this display object is near in the display reference coordinate system is shown in FIG. 23 (A), and the case where it is far is shown in FIG. 23 (B), which are designated as obj1 and ObJ2, respectively. No. 2
According to the method described in FIGS. 1 and 22, the screen S1
The display objects h-obj1 and h-obj2 are projected on. A signal can be sent to the corresponding pixel on the screen coordinate of each constituent point for display. Since obj2 is located far away, the distance between the constituent points is smaller than that of h-obj1. If this distance is sufficiently smaller than the pixel distance, it is not necessary to calculate all the constituent points included in the pixel. In addition, human eyes have visual acuity characteristics, and it is meaningless to calculate each constituent point on the screen S1 surface at intervals equal to or lower than the resolution of the eyes.

The existing algorithm calculates all the constituent points of the target regardless of the position of the display target and without considering the visual acuity characteristics of the user, so the display speed becomes slow and a complicated target is displayed in real time. Becomes difficult.

[Problems to be Solved by the Invention] The above-mentioned problems are particularly problematic when realizing stereoscopic computer graphics. Stereoscopic computer graphics is a method in which a perspective image on a screen when a three-dimensional display target is viewed is calculated separately from the user's right eye and left eye as viewpoints, and this is calculated by time division or the like for each of the left and right users. This is a method of displaying a stereoscopic image by projecting it on a screen corresponding to the eyes.

In the stereoscopic display, since the depth interval is clearly recognized by the user, it is necessary to model a large object (such as a building) that is originally large and a small object (such as a bug) to be small. In computer graphics that is not stereoscopic, it is possible to make the object look larger by arranging it near the viewpoint without modeling it greatly. However, in stereoscopic view, since the depth interval is given by parallax information, even if a small object is in the foreground, it is not recognized as a large object, and the small object appears to be in the foreground.

If you model an object in its original size, a large object
Assuming that the object comes near the viewpoint and spreads over the entire screen, it is necessary to model the details in detail. On the other hand, when the object is distant from the viewpoint, as shown in FIG. 23 (B), even though the entire object is displayed on a part of the screen, the same amount of calculation as when the object is near is achieved. There is an irrational need.

The need for high-speed computer graphics has been described above, but even if the computer graphics are real-time, there are some inconveniences depending on the application. In the realistic communication, a virtual space is artificially created by computer graphics, and various interactions with the virtual space occur at intervals actually present in the virtual space. In this case, what is required is not only high speed, but also the property of changing the image on the screen in synchronization with the movement of the observer. However, in computer graphics, the time to generate images always remains, so even if it is possible to generate and display several tens of images per second, if there is a delay time to generate each image with respect to the observer's motion. , I feel uncomfortable and cannot get a sense of unity with the virtual space.

Further, human visual acuity characteristics are high in the vicinity of the gazing point, as will be described later, but greatly deteriorate in the vicinity thereof. One of the reasons for this is the original nature of the eye that the retina portion that is off the optical axis of the eyeball is out of focus. Therefore, in the case of artificially generating and displaying an image, it is necessary to generate a screen image so that the image reflected on the retina becomes closer to the real space. That is, it is not always appropriate to display the entire screen in high definition with a uniform algorithm from the viewpoint of realistic display.

Therefore, a main object of the present invention is to provide a highly accurate real-time computer graphics display necessary for intelligent coded communication and stereoscopic computer graphics.
It is an object of the present invention to provide a gaze-following high-speed image generation and display method that can be realized by significantly reducing the amount of calculation.

Another object of the present invention is to generate a line-of-sight-tracking high-speed image capable of reducing the time delay between the generation of an image and the display of the image in computer graphics or the like that realizes motion vision, and realizing a natural display without discomfort. It is to provide a display method.

Still another object of the present invention is to generate a screen image so that the image captured on the retina becomes close to that when the real space is viewed, so that an image having the same feeling as when the real space is viewed is obtained. It is to provide a gaze-following high-speed image generation and display method that can be realized.

[Means for Solving the Problems] The invention according to claim 1 is a gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, the object having a three-dimensional structure. The structure surface of the is described hierarchically from the coarse to the fine by the tree structure, and when projecting the target structure surface from the arbitrary viewpoint onto the two-dimensional screen, the line of sight of the user is A configuration in which a spatial gazing point that first intersects with a target constituent surface is detected, and the distance from the spatial gazing point is used as at least one parameter, and the hierarchical degree of the target constituent surface within that distance is selected. Is.

The invention according to claim 5 is a gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the structural surface of the object having the three-dimensional structure is rough due to a tree structure. It is described hierarchically by the degree of hierarchy, and when projecting the structural surface of the target from an arbitrary viewpoint onto the two-dimensional screen, it detects the user's line of sight and the vertical distance from that line of sight. Is used as at least one parameter, and the hierarchical level of the target component surface within the distance is selected.

More preferably, in addition to the invention according to the first or fifth aspect, the invention according to the second or sixth aspect is the display reference coordinate system origin or an arbitrary point of the object represented in the object coordinate system from the viewpoint. It is configured to select the hierarchy degree with the distance to as the second parameter.

Still more preferably, in the invention according to the third or seventh aspect, the vicinity of the principal points of the optical systems of the left and right eyes is set as a viewpoint, and a projection image viewed from that point is generated.

In addition to the inventions according to claims 1 and 5, the inventions according to claims 4 and 8 have a point of interest at the point where the line of sight intersects with the two-dimensional screen, and the position of the point of interest as a starting point, It is configured to display by applying a high-frequency cutoff filter to the periphery.

Furthermore, the invention according to claim 9 is, in the invention according to claim 1 to claim 8, predicts the next gaze direction or gazing point from the trajectory of the gaze of the user, and based on the result, Next, an image to be projected on the two-dimensional projection surface is generated in advance and prepared for display.

[Operation] The invention according to the first claim uses the characteristic that the visual acuity of the human eye is high at the point of gaze and sharply decreases in the vicinity thereof, and the display target is set to the size of the area, that is, the scale. By modeling hierarchically as one parameter, the gaze portion of the display target is displayed up to the fine modeling hierarchy, and the target portion of the display unit corresponding to the peripheral vision deviated from the gaze portion is displayed in the rough modeling hierarchy. Reduce the calculation amount of the part to speed up.

According to a fifth aspect of the present invention, a stereoscopic vision is obtained by detecting the line of sight of the user and selecting the hierarchical level of the target component surface within that distance with at least one vertical distance from the line of sight as a parameter. Generates computer graphic images at high speed and high quality.

In the inventions according to the second and sixth claims, when the object is near the origin of the display reference coordinate system or the viewpoint, up to the fine modeling hierarchy is displayed, and when it is far away, the coarse modeling hierarchy is displayed. The second feature of reducing the calculation amount for a distant object and increasing the speed is combined with the inventions according to the first and fifth claims, and is far from the origin of the display reference coordinate system or the viewpoint, and the periphery. For the parts corresponding to the eyes, the amount of calculation is greatly reduced and the speed is increased.

The inventions according to the third and seventh claims generate a projection image viewed from the viewpoint of the vicinity of the principal point of the optical system of the observer's eye as the viewpoint.

The invention according to claims 4 and 8 uses the position of the gazing point as a starting point and displays a high-frequency cutoff filter around the gazing point, thereby displaying each part of the display target corresponding to peripheral vision in a rough modeling hierarchy. Compensate for unnaturalness caused by displaying. When displayed in a rough modeling hierarchy, in the screen projection image, there are many discontinuities between the constituent surfaces to be displayed, and high frequency components are generated in this area, but the high-frequency cutoff filter exists in the peripheral area. By applying, it is possible to remove the component that makes the user feel uncomfortable.

The invention according to claim 9 predicts the next gaze direction or gazing point from the trajectory of the gaze of the user, and based on the result,
Next, an image to be projected on the two-dimensional projection surface is generated in advance and prepared for display.

[Embodiment of the Invention] FIG. 1 is a diagram showing the principle of the present invention, and FIG.
It is a figure which shows the hierarchical modeling of the display object shown in the figure.

The principle of the present invention will be described with reference to FIG.
The screen S1 of the display device is represented by a display reference coordinate system X (w) -Y (w) -Z (w) with the origin O (w) as a reference. The display reference coordinate system X (w) -Y (w)-
Z (w) is a coordinate system that serves as a reference for other coordinate systems shown below, and may be provided in a different place from the screen S1 in FIG. The display objects obj1 and obj2 differ from the conventional example shown in FIG.

That is, as shown in FIG. 2, the display target is a structure surface (second structure) made up of a plurality of structure points Pi and the structure points P1.
In the figure, it is defined as a surface surrounded by four structural points). Each structural point is represented by the target coordinate system X (obj) -Y (obj) -Z (obj). In setting the hierarchy in FIG. 2, the number of structure points is reduced each time the distance between the structure points or the size of the structure surface exceeds a predetermined size, as shown in (a) to (d). To
There are four levels of hierarchy. A data structure represented by such structure points and a line segment connecting the points will be referred to as a wireframe structure.

Vectors VO1 and VO2 are displayed reference coordinate system X (w)-
Display target obj1, o from origin O (w) of Y (w) -Z (w)
It is a vector of the origin obj1, obj2 of the target coordinate system of bj2. First
In the figure, a user 2 who is an observer of the screen S1 views the display objects obj1 and obj2 with both eyes eye1 and eye2. Line of sight 10
Intersects screen S1 at point ep. This point ep is called the screen gazing point. X (e) -Y (e) -Z (e), X '
(E) -Y '(e) -Z' (e) is the viewpoint coordinate system,
eo1 is the point where the line segment connecting O '(e) and the screen gazing point ep intersects the display target. The structure points Pi (obj) are feature points in the display target model of the rough hierarchy shown in FIG.

Here, the line of sight 10 captures the feature point of the eyeball with two cameras,
By measuring the stereo image from the position information of the camera image pickup surface, the position can be obtained in the display target coordinate system. As the characteristic point, a corneal reflection image created when the light source set at a known point in the pupil center or the display target coordinate system is reflected by the cornea may be used. The details are
An “image capturing device” (Japanese Patent Application No. 1-181387 (Japanese Patent Application Laid-Open No. 02-224637) by the inventors of the present application is used for feature point extraction, and a “gaze detection method” (Japanese Patent Application No. 1-296900 (Japanese Patent Publication No. 03-051407), the characteristic point (eg, Pi (obj)) at or near eo1 point in FIG. The point can be obtained by the inverse transformation of the method of performing perspective projection transformation on the screen S1.

FIG. 3 and FIG. 4 are diagrams for explaining a method of obtaining a point on the model corresponding to the screen gazing point ep (s).

In FIG. 3 and FIG. 4, in order to simplify the invention, Z
The case where there is a viewpoint o (e) on the (w) axis is shown. The display target is an object as shown in FIG. 2, and in order to obtain the position of the target corresponding to the screen gazing point ep (s), a rough modeling hierarchy is performed for speeding up. The target shown in FIG. 2 (d) is the target obj of FIG. The normal perspective transformation point of each constituent point Pi (w) is pi (np). With a coarse model, this conversion is possible fast. The parallel projection point of the normal perspective transformation point pi (np) on the screen S1 (ABCD) is ξi (s).

Here, as shown in FIG. 4, the screen gazing point ep (s)
From this point, it is possible to search on the search line 20 parallel to the Z (np) axis and find the point where the search line 20 intersects the normally projected object. Constituent points 301 to 303 and the like of the object 30 that are orthographically projected are the normal perspective coordinate system X (np) -Y (n
If it is given by p) −Z (np), it is easy to find the point eh (np) that intersects the constituent surface 301.

Further, even if the constituent plane is not given, as shown in FIG. 4, ΔS is taken on the search line 20, this plane is scanned on the line, and the structural point pi (np) that first enters in the plane. You can also ask. Here, if the point eh (np) pi (np) is inversely transformed into the display reference coordinate system X (w) -Y (w) -Z (w), the spatial gazing point corresponding to the screen gazing point ep (s) eo (w)
Alternatively, Pi (w) in the vicinity can be obtained.

For Pi (w), the screen gazing point ep (s)
It is a corresponding point a little away from. The reason why the model of the fine hierarchy is not used is that the search of the embodiment shown in FIG. 3 is based on the assumption that there is already the normal perspective coordinate transformation data to be displayed. Therefore, the use of the fine modeling hierarchy is inconsistent in terms of speed. This is because However, in one embodiment of the present invention, the case of displaying several to several tens of sheets per second is considered, and at such a speed, it is considered that there is little change between two models that are temporally continuous.

Also, the modeling information for generating the immediately preceding image to be displayed next and this normal perspective transformation data already exist. Therefore, if a search is performed using the immediately preceding data, it is possible to accurately obtain the display target position corresponding to the screen gazing point ep (s). In this way, the spatial gazing point eo corresponding to the screen gazing point ep (s)
It is possible to obtain the position of (w) or this vicinity.

The vicinity of the spatial gazing point eo (w) in the present invention is near the line of sight, and is defined as a constituent point of a target which is determined to be gazing, a constituent surface, or an origin of the target coordinate system. The above search is a simple theoretical calculation and coordinate conversion calculation, and can be implemented in hardware.

FIG. 5 is a diagram for explaining a method of selecting a model hierarchy. Referring to FIG. 5, the display reference coordinate system origin O
(W) or the viewpoint O ′ (e) to the origin of the target coordinate system, the absolute value of the vector VO1 or VO2, that is, the distance is O
The model hierarchy is selected depending on whether or not it is within a predetermined distance (Th1 or Th2) centered on (w). The target obj1 is inside the distance Th1 and the hierarchy shown in FIG. 2 (b) is selected, and the target obj2 is inside the distance Th2 and the hierarchy shown in FIG. 2 (c) is selected.

Next, consider the spatial gazing point positions (two examples are shown in FIG. 5), assume eo1 and eo2, respectively, and assume spheres with predetermined radii STh1 and STh2 around these points. The fifth
In the example shown in the figure, consider a case where eo1 and Pi (obj) shown in FIG. 1 overlap. For the inside of the radius STh1, a model of one level finer is used. Although not shown in FIG. 5, the outside of the radius STh2 is changed to a one-step coarse hierarchical model. That is, for the target obj1, the model in the radius STh1 is changed to the model shown in FIG.
Regarding j2, the radius STh1 is changed to the model shown in FIG. 2 (b). As described above, a modeling object is prepared which is relatively fine near O (w), coarse in the distance, relatively fine in the vicinity of the spatial gazing point eo (w), and relatively coarse in the periphery. It A projected image can be obtained by projecting and converting each constituent point of the model onto the screen S1 by a conventional method.

FIG. 6 is a diagram for explaining a method of selecting a model hierarchy in another example of the present invention. Referring to FIG. 6, eo1 and eo2 are the positions of the intermediate gazing points, Thi is the radius centered on eo1, and the distance d _i (V _i of the target coordinate system origin from O (w) It is a function of absolute value. The function acts to increase the radius Th _i when d _i is small and to decrease the radius Th _i when the distance d _i is large. In the example shown in FIG. 6, Th _i-1 around the position eo1 of the spatial gazing point is larger than the radius Th _i-2 of the spatial gazing point eo2. In the radius Th1, FIG. 2 (a) is selected, in the radius Th1 to Th2, FIG. 2 (b) is selected, and in the radius Th2 to Th3, the second.
FIG. 2C is selected, and FIG. 2D is selected between the radii Th3 and Th4. As described above, the spatial gazing point eo
It is possible to obtain a model in which the definition is gradually changed centering on (w).

In the example shown in FIG. 1, FIG. 5 and FIG. 6 described above, the origin O (w) of the display reference coordinate system X (w) -Y (w) -Z (w) is used as the starting point. Although the distance to an arbitrary point in the coordinate system is used as a parameter, the viewpoint origin (O (e) or O ′ (e), etc.) may be used as the starting point. In this case, if the viewpoint moves without moving the display object, there is a lot of trouble to change the hierarchy level each time, but considering that the visual acuity characteristics of the person who recognizes the details of the object greatly depend on the distance from the viewpoint. It can be said that a more reasonable hierarchy level can be selected.

Further, as will be described later, in binocular stereoscopic computer graphics and the like, the display target may be displayed in front of the screen S1, but in such a case, as shown in FIG. w) as a parameter,
Even if the object is near the viewpoint O (e), an irrational situation will occur in which a coarser hierarchy is selected than an object immediately behind (inside) the screen S1. Therefore,
Although the calculation is complicated, it is also effective to use the distance from the viewpoint to the origin of the target coordinate system as a parameter.

FIG. 7 is a diagram showing quadratic and cubic Bezier curved surfaces.
The example shown in FIG. 7 (a) is the case of the second order and the third order (n = 2,3) and is expressed by the following equation.

R (t: 3) = ( 1-t + tE) n P 0 Bezier curve when t = 0, starting from the control points P _O, t
When = 1, the end control point (P ₂ when n = ₂ , P ₃ when n = ₃ ) is reached. The curves connect smoothly between them near the control points. Increasing the number of control points corresponds to increasing the degree, and the expressiveness of the curve increases.

The Bizier surface is expressed by the following equation in the form of the product of Bezier curves.

S (u: m, v: n) = (1-u + uE) ^m (1-V + vF) ⁿ P _O0 (0 ≦ u ≦ 1) (0 ≦ v ≦ 1) E and F are shift operators, and EP _ij = P _{i + 1, j} , FP _ij = P
_It acts as _{i, j + 1} .

FIGS. 7C and 7D show quadratic and cubic Bezier curved surfaces. In the case of the third order, by preparing 16 control points from P _OO to P ₃₃ , a free-form surface smoothly connecting these is generated.

In this way, if a partial area is adaptively determined and control points sufficient to describe this area are selected, a surface having a fine smooth surface is obtained. On the other hand, if the partial area is made large and the number of control points is reduced, the surface becomes rough. So far, an example has been shown in which the number of control points in the partial area is hierarchized as a parameter.

Furthermore, in the above Bezier surface equation, the structural points are control points and u,
It depends on how to take the value of v. Therefore, a predetermined control point may be determined, and the interval taken by u and v may be used as the hierarchical parameter. That is, in FIG. 7 (d), even if the number of control points is constant at 16 points, the configuration surface becomes coarse or dense depending on how to determine the interval Δ between u and v. If the interval Δ is reduced, the constituent surface is further divided from that shown in FIG. 7D, and many small constituent surfaces are produced. In this way, the size of the area can be hierarchized using Δ as a parameter.

Next, as the method of displaying on the screen, as shown in FIG. 1, in addition to the simple method of displaying the points and lines obtained by perspective projection of the wire frame, color data is prepared on the structural surface, Colored when displayed. A pattern called a texture map is prepared as data on each structure surface, and the pattern is rotated, moved, reduced, enlarged, or the like according to the direction and position of the structure surface and displayed on the screen. In the display reference coordinate system, it is possible to assume a light source at a predetermined place, calculate the shadow of each component surface from the viewpoint by the light, and display it.

As an example, a case of modeling a life-sized person plaster image with Bezier curved surfaces will be described. In order to model this plaster image in detail, the number of control points was about 5000. That is, the 5000 control points are divided into a group of 4 × 4 control points as shown in FIG. 7 (d), and the partial area determined by the 4 × 4 control points is further divided.
By selecting the values of u and v, it is divided into 10 parts. In this way, a structural surface is formed by 500,000 structural points. When the display surface is created by dividing the image into such parts, an image in which details can be recognized in detail can be obtained.

On the other hand, if it is possible to determine whether or not a person is present, the control points may be about several tens to 100, and u and v may be divided into several. In other words, it can be expressed with about 100 structural points. In this way, 100 to several hundreds depending on the environment and need to display objects such as people.
There is a 0-fold difference in the amount of data. That is, hierarchization can be variously selected with respect to the size of the area.

FIG. 8 is a diagram showing still another example of the present invention, in which the viewpoint is shown at the eyeball optical principal point of the observer. The optical principal point can be approximated by the center position of the pupil or iris, and its position and the line-of-sight direction are known in the display reference coordinate system as described above. In FIG. 8, display objects obj11, o
bj12, obj13, and obj14 are modeling objects of the same size having the hierarchy shown in FIG. 2, O (obj12) is the origin of obj12, and el connects the screen gazing point ep (s) and the viewpoint O (e). Line segment (that is, line of sight), and d3, d4, d5, d6 are line segments el
To the target coordinate system origin of each target object obj11 to obj14 (for example, O (obj12)), and V12 is the viewpoint O (e).
To O (obj12), 221,222 is the line segment el
These are the points closest to O (obj11) and O (obj13), respectively.

Since both the target coordinate system and the line of sight of each display target are represented by the display reference coordinate system, it is possible to obtain each of the above distances. Thus, in FIG. 8, the display target obj11, ob
j12 is relatively close to the viewpoint O (e), and the display target obj
13, obj14 is far away. The display objects obj11 and obj13 are near the line segment el, and the display objects obj12 and obj14 are located away from the line segment el. An example is given for the model hierarchy selection method.

The absolute value of the vector V12 is set as the distance d12 and is set as one parameter. Also, the origin O of the target obj12 from the line segment el (obj12)
F (d12, d5) with the distance d5 to the second parameter
Select the model hierarchy according to the value of the function. f becomes smaller as the value of the distance d12 or d5 becomes smaller. Therefore, the hierarchy shown in FIG. 2B is selected for the object obj11 whose target coordinate system is close to the viewpoint and close to the line segment el, and the hierarchy shown in FIG. 2B is selected for the object obj14 far from the viewpoint and far from the line segment el. The hierarchy shown in d) is selected. Also, either the distance from the viewpoint or the distance from the line segment el is short and the other is far.
For obj12 and obj13, the hierarchy shown in FIG. 2 (c) is selected.

Further, in FIG. 8, 221 or 222 from the viewpoint O (e).
A function f may be created with the distance to as one parameter and the distance d5 as the second parameter.

In Fig. 8, the distance d from the line segment el to the origin of the target coordinate system
Although 3, d4, d5, d6 are used as parameters, the parameters do not have to be the origin, and for example, as shown in FIG. 9, the position of each component surface is given to the center position of the surface or the like. If so, this distance can also be used as a parameter. In FIG. 9, since the line segment 251 is near the line segment el, the corresponding surface 300 is modeled in the hierarchy shown in FIG. 2B, and the line segments 252 and 253 are slightly separated from the line segment el. , The model of the hierarchy shown in FIG. 2 (c) is selected.

Further, the display target may be composed of a plurality of structures and may have a main target coordinate system corresponding to the main body and a sub target coordinate system corresponding to a part of the main body or an accessory. In this case, it is natural that the distance between the origin of the sub target coordinate system and the line segment el can be used as a parameter.

FIG. 10 is a diagram showing the relationship between the declination from the gazing point and the relative visual acuity by measuring the visual acuity around a point when the human eye is gazing when the eye is flat. In FIG. 10, the horizontal axis represents the deviation angle from the gazing point, and the vertical axis represents the relative visual acuity normalized by the visual acuity of the gazing point. The two curves shown in FIG. 10 show that the visual acuity characteristics change depending on the index presentation time. In the video displayed at a speed of about 30 sheets per second, this 2
It is considered to fall within the curve of the book. It can be seen that the longer the presentation, the better the visual acuity characteristics, but both the visual acuity is greatly reduced in the periphery. The reason for this is that the peripheral visual object deviating from the direction of the line of sight does not optically form an image on the retina,
This is because blurring is occurring and the resolving power is low in the part of the retina deviated from the fovea of the retina in the gaze direction.

Furthermore, when capturing an object with a change in depth, the convergence is changed and the crystalline lens is adjusted to focus. In other words, it is a daily experience that even if the line of sight is directed toward the target, blurring still occurs unless it is in focus. From the above, it can be said that the part that is clearly visible in the real space is a very small part where the line of sight and accommodation are controlled. Other parts have a large blur. Based on such a situation, when generating the space viewed by a person by computer graphics, the space gazing point with respect to the screen gazing point is detected, and the distance from that point is used as one parameter to determine the target from the line of sight. It can be said that it is rational from the viewpoint of reducing the amount of calculation to weight the precision of the model constituting surface using the distance to the point (for example, the origin of the target coordinate system) as one parameter. In addition, since it is different from the actual situation to display finely the portion where the focus is originally out of focus and the blur is generated, a feeling of strangeness may occur.
In some cases, it may be more realistic to present a blurred image.
As described above, the method of the present invention is effective not only in increasing the processing speed but also in realizing a sense of reality.

FIG. 11 is a diagram showing still another example of the present invention. 11th
Referring to the drawing, if the relationship between the target coordinate system of the display object obj and the display reference coordinate system is known, the display reference coordinate system X (w) -Y is set for each constituent point and constituent surface data.
Two viewpoints O that can be represented by (w) -Z (w)
(E) l and O (e) r respectively correspond to the viewpoints of the left and right eyeball optical systems of the observer. The position of the viewpoint and the direction of the line of sight can be obtained in the display reference coordinate system by the above-mentioned stereo image measuring method or the like. When the sight lines of the left and right eyes are known, the point where the two sight lines intersect is the target position that the observer is paying attention to. Although this point may include an error depending on the measurement accuracy of the line of sight, the method of finding the object near the line segment (equal to the line of sight in FIG. 11) connecting the viewpoint and the gazing point shown in FIG. 8 is easier than the case of FIG. 8.

This space gazing point (or its vicinity) is eo. A sphere of radius Th is assumed around the space gazing point eo, and the inside of the sphere is the second
It is modeled in the hierarchy shown in FIG. 2B, and the outside of the sphere is modeled in the hierarchy shown in FIG. 2D. Left and right images h-obj
(L) and h-obj (R) are projected on the screen. In this example, two viewpoints O (e) L, O (e)
Since R and the line of sight constantly change according to the movement of the observer, this change is detected in real time and the left and right images h-ob
Similarly, j (L) and h-obj (R) are reproduced and displayed in real time. This allows the observer to move their head and display
You can see obj from various directions such as left and right, up and down. At this time, the two images are h-obj (L), h-obj
Since (R) has disparity information, the observer is the target obj
Can be compared three-dimensionally.

Furthermore, it is possible to create an image that assumes that the target obj has come in front of the screen S1. This method does not allow display delays. In other words, since the observer's eyes are the viewpoint, a sense of discomfort occurs unless an image is generated and displayed in synchronization with the movement. The permissible delay is considered to be about 1-2 / 30 seconds. Therefore, the method of the present invention, which is advantageous for high speed, works effectively.

In this embodiment, a binocular stereoscopic parallax device can be used to display the generated image. As a specific example of this device, a “stereoscopic display device” (Japanese Patent Application No. 2-18051) by the inventors of the present application is possible. The structure of this device is that a lens-shaped lens sheet called a lenticular lens is provided on the display surface, and a pixel for the right eye and a pixel for the left eye are arranged on both sides with the focus behind the lens. is there. That is, two pixels are arranged as a set so as to face one Kamaboko lens. The light emitted from the two pixels is spatially separated by the lens and separately enters the left and right eyes of the observer. In this way, stereoscopic viewing is possible without mounting a special device.

FIG. 12 is a diagram showing still another example of the present invention, in which the delay with respect to the movement of the observer is further reduced.
With reference to FIG. 12, the display objects obj1 and obj2 are the model objects shown in FIG. 2, and the display reference coordinate system X (w) −Y
It is represented by (w) -Z (w). Viewpoint and eyepoint O
(E) is approximated by the center position of the pupil or iris. Line of sight
e1 (1) and e1 (0) change with the rotation of the eyeball. The pupil position and the line of sight are displayed in the reference coordinate system X (w) -Y (w).
It is represented by -Z (w). The space gazing point eo is on the line of sight extension. The distance from the line of sight to the display target obj1 is d (-1), d
It is represented by (0).

FIG. 12 shows a case where about 10 to 30 images are generated and displayed per second, and therefore (0), (-1),
Each point of (+1) has an interval of about 100 to 3 msec. The hierarchy in the current state (0) and the immediately preceding state (-1) is selected as follows. The distance from the line of sight to the object described in FIG. 8 is used as one of the parameters. In (-1),
Although the display object obj2 is far from the viewpoint, but is on the line of sight, the hierarchy shown in FIG. 2 (c) is selected. Display object ob
Although j1 is near the viewpoint, but is slightly away from the viewpoint, the hierarchy shown in FIG. 2C is selected in the same manner. In this way, h-obj1 (-1), h-obj2 (-
1) is generated and displayed. In addition, in (0), the display target obj2 does not change, but the display target obj1 is further away from the line of sight, and thus the hierarchy shown in FIG. 2D is selected.
In this way, h-obj1 (0) and h-obj2 (0) are generated and displayed.

Next, the display in which the movement of the line of sight is predicted will be described. Eye movements are roughly classified into smooth following movements when tracking moving indices and jumping movements that are seen when searching for an object of interest. In the follow-up exercise, the visual acuity decreases less than when the stopped index is seen. On the other hand, in the jumping motion, the visual acuity deteriorates temporarily immediately after the gazing point moves. Figure 12 shows the state of moving the head while gazing at the spatial gazing point, but the movement of the gaze at such a time is smooth, and the gaze display reference coordinate system at the time when the next image is displayed from the locus. It is easy to obtain X (w) -Y (w) -Z (w). That is, O (e) (-i)
~ O (e) (O), el (-i) ~ el (O) to O (e)
It is possible to obtain (+1), el (+1) by extrapolation. Therefore, the perspective projection images h-obj1 (+1), h-obj2 (+1) of the object viewed from O (e) (+ 1) are generated in advance and displayed at the time (+1).

Further, in the case of this embodiment, the spatial gazing point at the times of (-1) and (0) is eo, which does not change.
It is also possible to generate h-obj1 (+1) and h-obj2 (+1) on the assumption that the same point is focused on even at (+1). At this time, the observer can also presume that he is interested in obj2. Therefore, the hierarchy of the modem can be increased by one from the case of (0) to the hierarchy shown in FIG. 2 (b). In this way, identify the gaze target,
It is also possible to adaptively display the target in a fine hierarchy. Further, for the identification of the gaze target, various knowledge such as the positional relationship of the display target, the meaning of the target, and the transition of the gaze target of the observer can be used together.

The method according to the present invention is less problematic because it is difficult to predict when the line of sight is a smooth motion, but when jumping motion is mixed, the prediction is disappointing, which seems irrational at first glance. However, there is no problem for the following reasons. That is, as described above, the visual acuity is closely related to the movement of the line of sight, the visual acuity decreases after the jumping motion, and it takes more than 200 msec to recover. Therefore, even if the predicted viewpoint is wrong, the observer's eyesight in the displayed image is greatly reduced, so that there is little discomfort. It is only necessary to detect the correct pupil position and line of sight and generate and display the perspective projection image from that point before the visual acuity is restored.

By the way, in the above embodiment, the screen gazing point ep
A model of a fine hierarchy is selected in the vicinity of (s) (central view part), and a model of a coarse hierarchy is selected and displayed in the peripheral view part. At this time, since the structure surface becomes large in the portion corresponding to the peripheral vision, it is conceivable that a boundary is formed between the structure surfaces or the continuity with the central vision portion is broken. Since such a discontinuous surface has a high spatial frequency, it may be easily perceived by peripheral vision, which may cause discomfort. Therefore, in order to eliminate such discomfort, the screen gazing point ep
A method of blurring the image by covering the area around (s) with a high threshold cutoff filter is considered.

FIG. 13 is a diagram showing a high-threshold cutoff filter which is put around the gazing point of the screen. Referring to FIG. 13, no filter is set (the original image of 6.6 cpd is displayed as it is) within 14 ° around the screen gazing point ep (s).
It is set to 4.5 cpd in the 20 ° region and 1.0 cpd in the 20 ° or more region (note that cpd is the number of spatial frequencies that can be displayed within a 1 ° viewing angle). This filter can be configured by hardware and can follow the movement of the screen gazing point ep (s) and act on the display image in real time. In designing this filter, how to set the difference in resolution between the central vision portion and the peripheral vision portion and the viewing angle of the central vision portion is an issue.

FIG. 14 is a diagram showing a result of applying the filter shown in FIG. 13 to an original image captured by a camera. As can be seen from FIG. 14, when the camera has the viewing angle division shown in FIG. 13 and the filter with the resolution of the peripheral vision part as a parameter, the maximum spatial frequency cpd of the peripheral image is 3.0 or more. It can be seen that it cannot be distinguished from the original image. Therefore, the central part
The maximum spatial frequency of the surroundings can be lowered to about 3cpd in the filter set to -20 °. By using such a filter, the discomfort is reduced due to the discontinuity of the structural surface.

FIG. 15 is a schematic block diagram of an embodiment of the present invention.
In the embodiment shown in FIG. 15, it is assumed that the viewpoint is the eyeball and the eyeball is moving, and the display target is stationary in the display reference coordinate system for simplification of description. The 15th
The thick solid line portion shown in the figure is a novel means that has not been found in the past.

Referring to FIG. 15, the viewpoint and line-of-sight detection means 10 detects the viewpoint and line-of-sight of the eyeball in the display reference coordinate system. The target coordinate system setting means 20 specifies the origin position and rotation angle (angle) of the display target in the display reference coordinate system. The distance calculation means 30 calculates the distance from the viewpoint to the origin of the target coordinate system according to the detection output of the viewpoint and line-of-sight detection means 10, the origin position and the rotation angle designated by the target coordinate system setting means 20. The distance calculating means 40 also uses line-of-sight data of another eye not shown in FIG. 15, that is, obtains a point where both lines of sight intersect as a space gazing point, or even the line of sight of one eye is near the line of sight. Find the origin of the target coordinate system in, and use that origin as the space gazing point, or calculate the distance of the perpendicular drawn from the origin of the target coordinate system to the line of sight.

In the three-dimensional hierarchical database 80, the size of the area of the constituent surface of the display target object is hierarchically described as one parameter. The component point / component surface generating means 50 generates a target three-dimensional model at a position designated by the display reference coordinate system. That is, the generation unit 50 calculates the distance from the viewpoint obtained by the distance measurement unit 30 to the target origin, the distance from the spatial gazing point obtained by the distance calculation unit 40 to the target constituent surface, the distance from the target origin to the line of sight, and the like. As a parameter, the value of this parameter is compared with a reference value to determine the hierarchy of the target component surface, receive data from the three-dimensional hierarchical database 80, and generate a three-dimensional model in the display reference coordinate system. The portion surrounded by the dotted line in FIG. 15 constitutes processing means 101 for determining the hierarchy level using the above-mentioned parameters.

The perspective projection conversion means 60 receives the viewpoint position and the above-described three-dimensional model, and perspective-projects the display target on a two-dimensional screen screen. The display image generation means 70 refers to the component plane data obtained by designating the object, the component plane, and the layer in the three-dimensional hierarchical database 80, and the pattern such as coloring on the target component plane. Make a mark. The high frequency cut-off filter means 90 cuts off the high spatial frequency overlaid on the peripheral visual field of the gazing point of the screen. A lenticular screen or the like is used as the display unit 100 to display the target object.

16 and 17 are flow charts for explaining the operation of the embodiment of the present invention.

Next, the specific operation of the embodiment of the present invention will be described with reference to FIGS. Now, referring to FIG. 8 described above as an example, the display reference coordinate system X (w) -Y (w) -Z
In (w), consider generating an image on the screen S1 in which the objects obj11 to obj14 appear to be at predetermined positions. The position of the viewpoint is the pupil position, and the viewpoint is moving. Further, the object is stationary in the display reference coordinate system for simplification of description, the three-dimensional structure data of the object is described in the object coordinate system as shown in FIG. The size of the area is hierarchized as a parameter.

First, in step (abbreviated as SP in the figure) SP1, the target coordinate system setting means 20 sets the position vector V _{obj-i of the} target coordinate system origin = (X _obj-i , Y _obj-i , Z _obj-i ). The rotation angle R _obj-i = (α _obj-i , β _obj-i , γ _obj-i ) of the target coordinate system is set and given to the processing means 101. In step SP2, the viewpoint and the viewpoint position Voe = (Xoe, Yoe, Xoe) detected by the viewpoint and line-of-sight detecting means are given to the processing means 101. In step SP3, the distance calculating means 30 reads the viewpoint position vector Voe given from the viewpoint and line-of-sight detecting means 10 at that time, and the position vector V of the origin of the target coordinate system.
distance between two points using _obj-i and viewpoint position vector Voe
dis1 _obj-i = | V _obj-i −Voe | is calculated. In step SP4, the processing means 40 obtains the distance dis2 _obj-i from the line of sight indicated by d3 to d6 in FIG. 8 to the target coordinate origin. Next, the generation means 50 uses the parameters calculated by the distance calculation means 30.
parameter d obtained by dis1 _obj-i and processing means 40
The following operation is performed using is2 _obj-i .

dis3 _obj-i = f (dis1 _obj-i , dis2 _obj-i ) = A.dis1 _obj-i + B.dis2 _{obj-i The} above formula is the simplest example of creating the parameter dis3 _obj-i . A and B are constants here. It should be noted that a more complicated formula may be created in consideration of human visual acuity characteristics.

Then, the processing means 101 in step SP6 shown in FIG. 17 compares the parameters dis3 _obj-i and the reference value Th3, greater than the reference value Th3 parameter dis3 _obj-i, i.e. the subject of Figure 8 obj14 Is far and away from the line of sight, the model with a low hierarchical level as shown in FIG. 2 (d) is selected from the three-dimensional hierarchical database 80 in step SP7, and the target coordinates are selected in step SP8. At the position of the position vector V _obj-i of the system origin, the rotation angle R of the target coordinate system
Generate a 3D model at the angle of _obj-i .

The processing means 101 uses the parameter dis3 in step SP6.
_When it is determined that _obj-i is smaller than the reference value Th3, and further it is determined that it is larger than the reference value Th2 in step SP9, that is, in the case of the objects obj12 and obj13 shown in FIG. 8, step SP10 At (3), the hierarchical model shown in FIG. 2C is selected from the three-dimensional hierarchical database 80. However, when the processing means 101 determines in step SP9 that the parameter dis3 _obj-i is smaller than the reference value Th2, in step SP12 the hierarchical model shown in FIG. 2 (b), that is, the target as shown in FIG. Select the model of obj14.

As described above, the processing means 101 reads model data for all the objects (1 to i) from the three-dimensional hierarchical database 80. The perspective projection conversion means 60 is step SP14.
, The perspective projection transformed image h _obj-11 , h
_obj-12 , h _obj-13 , h _obj-14 is generated. In step SP15, the display image generation means 70 is a three-dimensional hierarchical database.
Each constituent surface pattern data accumulated in 80 is read, and the target constituent surface of the two-dimensional screen is patterned. Further, the high frequency cutoff filter means 90 cuts off the high frequency spatial frequency overlaid on the peripheral visual area of the screen gazing point, and the display means 100 displays the target image generated in step SP16.

FIG. 18 is a flow chart for explaining the operation of another embodiment of the present invention, in which the distance from the line of sight to the origin of each target coordinate system is used as the first parameter with the distance from the viewpoint to the origin of each target coordinate system. Is compared, and the closest target origin is estimated as the spatial gazing point, and the distance from this point is used as the second parameter. Steps SP1 to SP4 are the same as those in FIG. It differs from FIG. 16 in that the processing of SP23 is performed. At step SP4, d3 in FIG.
~ After finding the distance dis _obj-i from the line of sight to the target coordinate origin shown in d6, in step SP21 find the smallest obj-min in the distance dis2 _obj-i , and estimate the target origin as the spatial gazing point . Explaining in FIG. 8, in step SP22, the origin of obj11 is temporarily set as the space gazing point, and the distance dis4 from this point to the origin of other objects obj12, obj13, and obj14.
Calculate _obj-i . The distance calculation means 40 has its parameter d
Using is1 _obj-i and dis4 _obj-i , the following formula is calculated.

dis3 _obj-i = f (dis1 _obj-i ) .g (dis4 _obj-i ) = C.dis1 _obj- i.dis4 _obj-i ) The above expression is the simplest example of creating the parameter dis3 _obj-i . C is a constant here. Hereinafter, the processing of steps SP6 to SP16 shown in FIG. 17 is performed. It should be noted that in this embodiment as well, a more complicated formula may be created in consideration of human visual acuity characteristics.

FIG. 19 is a schematic block diagram showing another embodiment of the present invention. This example is an extension to use not only the origin of the target coordinate system but also the target configuration surface in the creation of the parameter for selecting the hierarchical level of the configuration surface, except for the following points. This is the same as FIG. 15 described above. That is, the processing means 102 includes the upper layer generation means 21 and the distance calculation means 31 and 41. The upper layer generating means 21 generates a schematic configuration of the target, that is, a configuration layer having a rough hierarchical level (upper layer) at a position where the target coordinate system is set, as viewed from the display reference coordinate system. This process can be performed at high speed because the hierarchical level is high. The distance calculation means 31 calculates the distance from the viewpoint to the base point of the roughly configured surface, for example, the center of the surface. The distance calculation means 41 uses a point at which the line of sight intersects the rough constituent plane as a space gazing point, or calculates the base point of the neighboring constituent surface, or calculates the distance of a perpendicular line dropped from the base point of the rough constituent surface to the sight line. calculate. The component point / component surface generating means 50 is
This is for generating a three-dimensional model of the target. In the case of, data of an appropriate hierarchical level is fetched from the three-dimensional hierarchical database 80 while comparing the distance from the spatial gazing point with the reference value Th and Generate the constituent points / surfaces of. In the case of, the data of an appropriate hierarchy degree is fetched from the three-dimensional hierarchal database 80 while comparing with the reference value Th using the distance of the perpendicular drawn to the line of sight as a parameter,
Generate the constituent points / component planes of interest. In this embodiment, the high frequency cutoff filter means 90 shown in FIG. 15 is omitted.

FIG. 20 is a diagram showing an application example of the present invention. The example shown in FIG. 20 is one that enables computer graphics images to be observed with a high degree of realism. The viewpoint corresponds to the center position of the pupil or iris. With respect to this viewpoint or line of sight, viewpoint detection 1 is performed by a sight line detection device (not shown), and position 2 and shape 2 of the hand is detected by a detection device (not shown). These data are given to the intention comprehension processing unit 3 and analyzed to estimate the object the observer is gazing at and the work to be performed next. Based on this information, the 3D model world 6 is updated 5 with reference to the 3D shape database 4, the 3D model world 6 is generated at high speed, and image generation 7 is performed by real-time stereoscopic computer graphics. Is performed, the image is projected and converted on the screen 8, and is displayed on the screen in real time.

[Effects of the Invention] As described above, according to the present invention, since the modeling hierarchy is made different between the vicinity of the gazing target and its periphery, the amount of calculation can be significantly reduced. On the contrary, in this extra time, the vicinity of the gazing point can be precisely calculated. As a result, high-speed, high-quality image generation and display can be performed. When simulating a case where a display object having a small depth, that is, a component having a small component in the Z-axis direction is projected on the screen by the method according to the present invention, the processing time becomes 1/3 or less of the conventional case when the screen has a viewing angle of 30 °. In the case of °, it was less than 1/4. For an object having a depth direction, the processing time can be further reduced by this square.

Further, the amount of calculation can be further reduced significantly for an object that is far from the origin of the display reference coordinate system or the viewpoint. On the contrary, in this extra time, it is possible to finely calculate the details of the nearby objects. As a result, it is possible to perform image generation and display with higher speed and higher quality. If the image quality is comparable to the conventional one, the processing speed will be greatly reduced to 1 / several hundred or less.

By detecting the line of sight, it is possible to adaptively control the definition of each part of the display image, so that it is possible to reproduce what is close to the image on the retina when looking at the real space when looking at the screen, You can generate and display natural and powerful images. Furthermore, high-speed, high-quality display of binocular stereoscopic computer graphics is also possible.

Further, since the image in which the movement of the line of sight is predicted is generated in synchronization with the movement of the eyeball or the like, the sense of discomfort due to the delay in the image display is reduced and the sense of unity with the image is improved.

Further, the present invention is particularly effective for use in the computer graphics processing part of intelligent coded communication and the computer graphics for stereoscopic viewing which are rich in realism. In addition, it can be used in many computer graphics fields in which high speed processing and high image quality are required.

[Brief description of drawings]

FIG. 1 is a diagram showing the principle of the present invention. Figure 2 shows the first
It is a figure which shows the hierarchical modeling of the display object shown in the figure. FIG. 3 and FIG. 4 are diagrams for explaining a method of obtaining a point on the model corresponding to the screen gazing point ep (s). FIG. 5 is a diagram for explaining a method of selecting a model hierarchy. FIG. 6 is a diagram for explaining a method of selecting a model hierarchy in another example of the present invention. FIG. 7 is a diagram showing quadratic and cubic Bezier curves. FIG. 8 is a diagram showing another example of the present invention in which the viewpoint is the eyeball optical viewpoint of the observer. FIG. 9 is a diagram showing an embodiment in which the position of the constituent surface to be displayed is given at the center position of the surface. FIG. 10 is a diagram showing the relationship between the declination from the gazing point and the relative visual acuity by measuring the visual acuity around the point when the human eye gazes at a point on the plane. 11th
The figure shows another example of the present invention. FIG. 12 is a diagram showing the principle of still another example of the present invention and is a diagram showing an example of further reducing the delay due to the movement of the observer. FIG. 13 is a diagram showing a high-threshold cutoff filter which is put around the gazing point of the screen. FIG. 14 is a diagram showing a result of applying the filter shown in FIG. 13 to an original image captured by a camera. FIG. 15 is a schematic block diagram of an embodiment of the present invention. 16 and 17 are flow charts for explaining the operation of the embodiment of the present invention. FIG. 18 is a flow chart for explaining the operation of another embodiment of the present invention. FIG. 19 is a schematic block diagram of another embodiment of the present invention. FIG. 20 is a diagram showing an application example of the present invention. FIG. 21 is a diagram showing a conventional typical three-dimensional image generation method. FIG. 22 is a view of the coordinate system shown in FIG. 21 viewed from the Y (w) axis. FIG. 23 is a view of the coordinate system shown in FIG. 21 on the Y (e) axis and in the Z (e) axis direction. First
FIG. 24 is a diagram of the display object shown in FIG. In the figure, obj is the display target known in the display reference coordinate system, P
i (obj) is the structural point to be displayed in the target coordinate system, S1
Is a screen, h-obj is a perspective projection image, O (e) is a viewpoint, ep (s) is a screen gazing point, eo (w) is a spatial gazing point, and Th is a distance.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Haruo Takemura, Haruo Takemura, Kyoto, Japan Seika-cho, Osamu, Osamu, Osamu, 5 Mihiradani, ATR Communication Systems Laboratories, Inc. (72) Satoshi Ishibashi, Soraku-gun, Kyoto Prefecture Seika-cho Osamu Osamu Osamu 5 Hiratani No. 5 AT R Communication System Research Institute, Inc. (72) Inventor Kenji Akiyama 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corporation (72) Invention Shinji Tetsuya 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corp. (72) Inventor Hiroyuki Yamaguchi 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corp.

Claims (9)

[Claims] 1. A gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the structural surface of the object having a three-dimensional structure is hierarchically structured by a tree structure. It is described hierarchically according to the degree, and when projecting the constituent surface of the target onto a two-dimensional screen when viewed from an arbitrary viewpoint, the space gazing point at which the user's line of sight first intersects the constituent surface of the target. Is detected and the distance from the spatial gazing point is used as at least one parameter to select the hierarchy level of the target component surface within the distance, and the gaze-following high-speed image generation and display method is provided. 2. The hierarchical level is selected by using a distance from an origin of the display reference coordinate system or the space gazing point to an arbitrary point of the target represented in the target coordinate system as a second parameter. A method for generating and displaying a line-of-sight tracking high-speed image according to claim 1. 3. The line-of-sight tracking according to claim 1 or 2, wherein the projection images viewed from the points of the optical system main points of the left and right eyes are generated as viewpoints. High-speed image generation and display method. 4. A point of interest at which the line of sight intersects with the two-dimensional screen is set as a point of interest, and a high-frequency cutoff filter is displayed around the point of reference to display the point of interest.
The gaze-following high-speed image generation and display method according to any one of claims 1 to 3. 5. A gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the structural surface of the object having the three-dimensional structure changes from rough to fine by a tree structure. It is described hierarchically by the degree of hierarchy, and detects the user's line of sight when projecting the constituent surface of the target onto the two-dimensional screen when viewed from an arbitrary viewpoint, and determines the distance in the vertical direction from the line of sight. A gaze-following high-speed image generation and display method, characterized in that, as at least one parameter, the hierarchical level of a target component surface within the distance is selected. 6. The hierarchical level is selected by using a distance from a display reference coordinate system origin of the two-dimensional screen or a viewpoint to an arbitrary point of the target represented in the target coordinate system as a second parameter. The gaze-following high-speed image generation and display method according to claim 5. 7. The line-of-sight tracking according to claim 5 or 6, wherein the vicinity of the principal points of the optical systems of the left and right eyes is set as a viewpoint, and a projection image viewed from the points is generated. High-speed image generation and display method. 8. The line-of-sight tracking type according to any one of claims 5 to 7, wherein the position of the point of gaze is used as a starting point and a high-frequency cutoff filter is displayed around the point of view. High-speed image generation and display method. 9. Further, for the purpose of display, an image to be projected next on a two-dimensional projection plane is generated in advance based on the result of predicting the next line-of-sight direction or gazing point from the trajectory of the line of sight of the user. The gaze-following high-speed image generation and display method according to any one of claims 1 to 8, characterized by being prepared.