JP6509101B2 - Image display apparatus, program and method for displaying an object on a spectacle-like optical see-through type binocular display - Google Patents

Description

  The present invention uses a glasses-like optical see-through type binocular display to display a character or image object (CG (Computer Graphics)) in a real space viewed from the user as augmented reality (AR (augmented reality)) It is related with the technology which displays as superimposed. In particular, the present invention relates to a technique for improving the visibility of an object viewed from the user.

  As an application of AR, for example, when the user holds the camera of the terminal possessed by the user over the product, the product is identified, and information such as the price linked to the product is displayed at an appropriate position on the display. be able to. In recent years, with the improvement of processing performance of terminals such as mobile phones and smart phones, an environment capable of executing AR has been provided.

  For example, there is an HMD (head mounted display) as a spectacles-like device provided with a small display attached to the head of the user. The camera mounted on the HMD does not have to consciously hold the camera on its own in order to continue capturing images of the real space that appears in front of the user's gaze. Therefore, the AR image can be superimposed and displayed on the real space ahead of the user's line of sight. There is a work support system as an application in which the HMD and the AR are linked. The worker can check the object of the work instruction displayed on the display in front of the eyes while working the hand, for example.

HMD can be mainly classified into the following two types.
Video see-through type: superimposing an object on the camera preview image Optical see-through type: superimposing an object on the real space itself

  The video see-through HMD can use the same alignment method as an existing smartphone application because the preview image of the camera is the target of alignment. However, the viewing angle of the wearer and the visibility of the real environment are limited to the angle of view and resolution of the camera. For this reason, in the case of a work support system, for example, the limitation of the visibility to the worker may reduce the work efficiency.

  On the other hand, in the optical see-through HMD, the real space itself that the wearer visually recognizes through the virtual screen is an object of alignment. The optical see-through HMD has the disadvantage that the display viewing angle is narrow compared to the video see-through HMD. On the other hand, it is suitable for the work support system in that only the necessary work instruction information can be AR-displayed in the real space because the view of the wearer is open.

  According to the optical see-through HMD, it is necessary to correct the display position of the object so as to overlap the physical space in the field of view of the wearer. Such a correction is an operation unique to the optical see-through HMD, and is referred to as "HMD calibration" (see, for example, Patent Document 1 and Non-Patent Document 1). According to the technique described in Patent Document 1, calibration parameters are estimated with high accuracy for a monocular HMD.

  Among the optical see-through HMDs, there is also a technology for compensating for the narrowness of the display viewing angle of a single eye by displaying on a virtual screen seen with both eyes (see, for example, Non-Patent Document 2).

  FIG. 1 is an external view showing the field of view of the user as viewed from an optical see-through HMD.

  Specifically, according to MOVERIO BT-200 manufactured by EPSON (see, for example, Non-Patent Document 3), when the user focuses on both eyes at a predetermined depth position, the viewing angle of a single eye, for example, the left eye The viewing angle of the left eye with respect to the screen is 23 °. Here, the viewing angle of a single eye is constant regardless of the in-focus position. On the other hand, as a result of visually confirming the viewing angle of the both eyes corresponding to the binocular screen as the depth distance l = 50 cm, it was about 1.5 times the viewing angle of the monocular. That is, it seems that the viewing angle is enlarged by 1.5 times when the camera moves from a state in which the right eye is closed and only the left eye is open to a state in which both eyes are open. It is visually recognized. As the user moves away from the fixation point (focus position), the magnification of the viewing angle of both eyes decreases. Then, at a constant depth distance l, the viewing angle becomes 1.0 times, and the left eye screen and the right eye screen completely overlap. The depth distance l is referred to as the HMD focal length.

The degree to which the viewing angle is expanded using the screen of both eyes differs depending on the depth position at which the wearer focuses.
As the user looks closer, the magnification of the viewing angle when using a binocular screen increases.
The farther the user looks, the lower the magnification of the viewing angle when using a binocular screen.

  FIG. 2 is an external view showing the relationship between two screens and the field of view of the user for the optical see-through HMD.

  According to FIG. 2, in the case of the MOVERIO BT-200, half mirrors are embedded in the left and right of the glasses in the HMD. Light emitted laterally from the inside of the HMD is input to the eyes of the user by the reflecting surface of the half mirror. On the other hand, light of the real environment is directly input to the user's eyes through the non-reflecting surface of the half mirror. As a result, the user sees a screen as a virtual image (Virtual Screen) in the real environment. Unlike a physical display, the virtual screen is characterized in that the size of the screen looks different depending on the depth position viewed by the user. For example, as viewed from the user, a display screen equivalent to the 320 type can be visually recognized at a position at a depth distance of 20 m. Optical see-through HMDs often take advantage of this property to make it possible to enjoy watching images on a super large screen even in a narrow environment as a feature on product sales. Hereinafter, "screen" will be referred to as "virtual screen".

According to FIGS. 1 and 2, the binocular screen is displayed at the depth position of the user's gaze point. The fixation point is a position at which the user's eyes are in focus. This binocular screen is divided into the following three areas.
Left eye area (area with left eye screen only)
Overlap area (the area where the right and left eye screens overlap)
Right eye area (area with right eye screen only)

  For example, according to the application of the work support system, there are many cases where the depth position is a short distance (about 50 cm) as the work range of the hand. This is different from the above-mentioned usage method assumed by the HMD maker, and the left eye region and the overlap are left as shown in FIG. 1 and FIG. 2 because the screen positions of the left eye and the right eye shift to the left and right. The area and the right eye area will be viewed simultaneously. Here, the binocular screen is a combination of these three areas and is a screen that is viewed by the user's eyes.

  For each of the left-eye and right-eye screens that make up the binocular screen, the HMD calibration parameters are calculated in advance to calculate the drawing position of the appropriate object that matches the user's eye position. can do. The calibration parameter takes a constant value regardless of the focus position of the user if the position of the user's eye is fixed with respect to the HMD to be attached. Therefore, even when the screen position of both eyes is shifted to the left and right at the time of short distance visual recognition, if the calibration parameter can be accurately calculated in advance, the object can be superimposed and displayed at an appropriate position.

  FIG. 3 is an explanatory view showing a screen shown in the user's view in the case where the calibration parameter does not include an estimation error. The balloon "Don't touch this" and "Press this" in the figure is an example of a work instruction object in the case of assuming a work support system. The wearer of the HMD sees this CG overlapping the actual work target equipment.

  According to FIG. 3, the calibration parameters of the HMD are calculated completely without error, and in the overlapping area of the binocular screens, the objects drawn on the respective screens are completely overlapped. From both eyes of the user, it can be viewed without any discomfort at all.

JP, 2014-170374, A

Kato Hirokazu, M. Billinghurst, Asano Koichi, Tachibana Keihano, "Augmented Reality System Based on Marker Tracking and Its Calibration", Journal of Virtual Reality Society of Japan, Vol 4, No 4, pp. 607-616, (1999) , [Online], [October 9, 2015 search], Internet <http://intron.kz.tsukuba.ac.jp/tvrsj/4.4/kato/p-99_VRSJ4_4.pdf> Yusuke Makihara, Tatsuya Kobayashi, Haruhisa Kato, Hiromasa Yanagihara, "Hands-Free Teleoperation Support AR System with HMD Calibration and On-Site Learning," Research Report Audio-Visual Compound Information Processing (AVM), 2015-AVM-88 (2), 1-6 (2015-02-20), [online], [October 9, 2015 search], Internet <URL: http://ci.nii.ac.jp/naid/110009877748> EPSON, "MOVERIOBT-200AV / BT-200", [online], [October 9, 2015 search], Internet <http://www.epson.jp/products/moverio/bt200/>

  In reality, the mounting position of the HMD may be shifted halfway. In addition, the calibration parameter differs depending on the user depending on the shape of the user's head and the presence or absence of eyesight correction glasses. However, since the calibration operation is time-consuming, it is conceivable to use a calibration parameter calculated by a third party. As a result, calibration parameters often include errors.

  FIG. 4 is an explanatory view showing a screen shown in the user's field of view in the case where the calibration parameter includes an error.

If the left and right calibration parameters (P _l, P _r ) include an error, the drawing position of the object (CG) for each screen is relative to the real environment as seen from the user as shown in FIG. It is misaligned. Here, in the HMD of both eyes, in addition to the deviation of the drawing position, a problem related to the visibility of the object peculiar to both eyes arises. Specifically, as shown in FIG. 4, the objects do not overlap completely in the overlapping area of the left-eye screen and the right-eye screen, and the objects are double-viewed.

  In addition, even in the case where the left and right calibration parameters do not include an estimation error, the objects drawn on the left and right screens overlap in the overlapping area, thereby increasing the luminance of only the overlapping area. As a result, the existence of boundaries at both ends of the overlapping area is noticeable, and the visibility of the entire object is reduced.

  Therefore, the present invention is an image display capable of improving the visibility of an object in an overlapping area of each screen, which is visually recognized by the user through the display when the object is displayed on a display of an optical see-through type of glasses. It aims at providing an apparatus, a program, and a method.

According to the present invention, it has an optical see-through type display arranged for each of the right eye and the left eye of the user and visible through the real space, in accordance with the gaze point of the real object the user gazes An image display device for controlling a virtual object to be displayed on a display,
Attention point position acquisition means for acquiring the depth position from the camera to the user's attention point;
An overlapping area calculation unit that calculates an overlapping area where the right eye screen and the left eye screen that are viewed at the depth position through the display intersect with each other from the right eye and the left eye of the user;
Drawing area selecting means for selecting either the right eye screen or the left eye screen for the virtual object to be displayed in the overlapping area;
It is characterized by including object drawing means for drawing the virtual object on the selected right eye screen or left eye screen.

According to another embodiment of the image display device of the present invention,
The drawing area selection means is
Divide a binocular screen consisting of a right eye screen and a left eye screen into a right eye area, an overlapping area and a left eye area;
Using the end coordinates of the virtual object with respect to the binocular screen and the end coordinates of the overlapping area,
The virtual object to be displayed across the right eye area and the overlapping area selects the right eye screen as a screen to be drawn in the overlapping area,
It is also preferable that the virtual object to be displayed across the left eye area and the overlap area select the left eye screen as a screen to be drawn in the overlap area.

According to another embodiment of the image display device of the present invention,
The drawing area selection means is
Regarding the virtual object to be displayed in the overlapping area,
If the accuracy of the calibration parameter P _r of the right eye is higher than the accuracy of the calibration parameter P _l of the left eye, select the right eye screen as the screen to be drawn in the overlapping area,
Conversely, if the accuracy of the calibration parameter P _r of the right eye is lower than that of the calibration parameter P _l of the left eye, it is also preferable to select the left eye screen as the screen to be drawn in the overlapping area .

ｓ：スケール係数 According to another embodiment of the image display device of the present invention,
The back projection calculation of the overlapping area calculation means is performed by the back projection on the depth position of the three-dimensional space through an arbitrary point of the right eye to the right eye screen (or the left eye to the left eye screen). The position relationship between the coordinate system (u _r , v _r ) of the right eye screen and the coordinate system (u _l , v _l ) of the left eye screen by back projection-projection calculation which projects on the screen (or right eye screen) It is also preferable to calculate the following transformation matrix H _{l, r} as shown.

s: scale factor

According to another embodiment of the image display device of the present invention,
It is also preferable that the overlapping area calculation means calculates a displacement amount d defined by the following transformation matrix H _{l, r} by back projection-projection calculation when the screens of both eyes are shifted only in the left and right direction.

ｓ：スケール係数
（ｕ_ｒ，ｖ_ｒ）＝（０，０）及びＺ_C＝Ｚ_α（奥行き距離のスカラー値）を代入することで、ｕ_ｌを変位量ｄとして算出するか、
又は、
（ｕ_ｌ，ｖ_ｌ）＝（０，０）及びＺ_C＝Ｚ_α（奥行き距離のスカラー値）を代入することで、ｕ_ｒを変位量−ｄとして算出することも好ましい。 According to another embodiment of the image display device of the present invention,
Backprojection overlapping area calculating means - projection calculations, the right eye screen coordinate system (u _{r, v r)} and left-eye screen coordinate system (u _{l, v l)} and, in the right eye calibration parameters P _{Using r} , the calibration parameter P _l for the left eye, and the camera coordinate system (Xc, Yc, Zc) used by the gaze point position acquisition means, the following formula is defined:

s: Calculate u _l as a displacement amount d by substituting a scale factor (u _r , v _r ) = (0, 0) and Z _c = Z _α (scalar value of depth distance),
Or
It is also preferable to calculate u _r as the displacement amount −d by substituting (u _l , v _l ) = (0, 0) and Z _c = Z _α (scalar value of depth distance).

ｓ：スケール係数
注視点位置取得手段は、世界座標系における注視点の座標を、変換行列Ｗc,wを用いてカメラ座標系へ変換し、そのＺcを奥行き距離のスカラー値Ｚ_αとして算出することも好ましい。 According to another embodiment of the image display device of the present invention,
When using a visible light camera, the gaze point position acquiring means stores in advance a marker image for specifying the gaze point, and detects the marker image from the photographed image, thereby setting the world coordinates in which the markers are arranged as follows: Calculate the transformation matrix (position and orientation parameters) Wc, w from the system (Xw, Yw, Zw) to the camera coordinate system (Xc, Yc, Zc),

s: scale factor The gaze point position acquiring means converts coordinates of the gaze point in the world coordinate system into the camera coordinate system using the transformation matrix Wc, w, and calculates the Zc as the scalar value Z _α of the depth distance Is also preferred.

According to another embodiment of the image display device of the present invention,
It is also preferable that the point-of-gaze position acquisition means sets the barycentric coordinates in the area of the marker image as the point of gaze.

According to another embodiment of the image display device of the present invention,
Object drawing means
For each virtual object, draw only on one screen not selected as the drawing target of the overlapping area,
Fill each overlapping area of the left eye screen and right eye screen with the transparent color of the display,
It is also preferable to draw the virtual object on the other screen not drawn.

According to the present invention, in a real object to be gazed by the user, the display has an optical see-through type binocular display which is disposed in front of each of the user's right and left eyes and can be seen through the real space. A program that causes a computer mounted on a device to control a virtual object to be displayed on a display to match a fixation point,
Attention point position acquisition means for acquiring the depth position from the camera to the user's attention point;
An overlapping area calculation unit that calculates an overlapping area where the right eye screen and the left eye screen that are viewed at the depth position through the display intersect with each other from the right eye and the left eye of the user;
Drawing area selecting means for selecting either the right eye screen or the left eye screen for the virtual object to be displayed in the overlapping area;
The computer is caused to function as an object drawing unit that draws the virtual object on the selected right-eye screen or left-eye screen.

According to the present invention, in a real object to be gazed by the user, the display has an optical see-through type binocular display which is disposed in front of each of the user's right and left eyes and can be seen through the real space. An image display method of an apparatus for controlling a virtual object to be displayed on a display in accordance with a fixation point,
The device is
A first step of acquiring a depth position from a camera to a user's gaze point;
Calculating, from each of the right eye and the left eye of the user, an overlapping area where the right eye screen and the left eye screen viewed at the depth position through the display intersect;
A third step of selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
And a fourth step of drawing the virtual object on the selected right-eye screen or left-eye screen.

  According to the image display apparatus, program, and method of the present invention, when displaying an object on a spectacles-shaped optical see-through type binocular display, the visibility of the object in the overlapping area of each screen which the user visually recognizes through the display Can be improved. That is, the present invention focuses on the fact that the binocular viewing angle of the optical see-through HMD can be expanded by utilizing the property at the time of short distance visual recognition which is not assumed by the HMD.

It is an external view showing a user's visual field seen from optical see-through type HMD. It is an external view showing the relationship between two screens and a user's visual field about optical see-through type HMD. It is explanatory drawing showing the screen reflected to a user's visual field in case a calibration parameter does not contain an estimation error. It is explanatory drawing showing the screen reflected to a user's view in case a calibration parameter contains an error. It is a functional block diagram of the image display apparatus in this invention. It is explanatory drawing showing the relationship between a world coordinate system and a camera coordinate system. It is explanatory drawing showing the displacement amount between both screens. It is an explanatory view showing position relation and each field of a left eye screen, a right eye screen, and a binocular screen. It is a first explanatory view showing a drawing area for a virtual object. It is a second explanatory view showing a drawing area for a virtual object. It is a third explanatory view showing a drawing area for a virtual object. It is a 4th explanatory view showing the drawing area to a virtual object. It is a 5th explanatory view showing the drawing field to a virtual object. It is explanatory drawing showing the drawing procedure of an object drawing part. It is explanatory drawing showing the screen reflected to the user's visual field in this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 5 is a functional block diagram of the image display apparatus in the present invention.

  According to FIG. 5, the function of the image display device 1 may be built in the optical see-through HMD, or may be an external terminal connected to the optical see-through HMD. The terminal is, for example, a smartphone, a tablet, or a personal computer. In this case, the function of the image display device is provided as an application installable on a terminal.

  According to FIG. 5, the display of the HMD main body is an optical see-through type that can be seen through the real space, and is disposed in front of each of the left and right eyes of the user. The image display device 1 includes a gaze point position acquisition unit 11, an overlapping area calculation unit 12, a drawing area selection unit 13, and an object drawing unit 14. These functional components are realized by executing a program that causes a computer installed in the device to function. The flow of processing of these functional components can also be understood as an image display method of the apparatus.

[Point of gaze position acquisition unit 11]
The gaze point position acquisition unit 11 acquires a depth position up to the gaze point as viewed from the user wearing the HMD. By acquiring the depth position to the point of gaze, the virtual object can be displayed with its position and size controlled at the focus position of the line of sight of the user. Specifically, the depth position is the distance from the camera mounted on the HMD to the gaze point of the user with respect to the real space. Then, the acquired gaze point distance is output to the overlapping area calculation unit 12.

The gaze point position acquisition unit 11 can acquire the gaze point distance up to the depth position using various devices. Examples of devices include the following.
Visible light camera depth sensor (depth camera)
Ultrasonic sensor or magnetic sensor (consisting of a dedicated transmitter and a sensor attached to the HMD body and the object to be watched)

<When using a visible light camera>
The fixation point position acquisition unit 11 stores in advance a marker image that specifies the fixation point. The marker image may be an image of the target object itself to be watched by the user, or a dedicated marker image may be attached to the target object. The marker image is detected by image recognition from the image captured by the visible light camera. The marker image may be, for example, a QR code (registered trademark). At this time, it is also preferable to set the barycentric coordinates in the area of the marker image as the gaze point.

  FIG. 6 is an explanatory view showing the relationship between the world coordinate system and the camera coordinate system.

ｓ：スケール係数
そして、注視点位置取得部１１は、世界座標系における注視点の座標を、変換行列Ｗc,wを用いてカメラ座標系へ変換し、そのＺcを奥行き距離のスカラー値Ｚ_αとして算出する。 The point of gaze position acquisition unit 11 acquires the position and orientation of the target object (see, for example, Non-Patent Document 1). This calculates the transformation matrix (position and orientation parameter) Wc, w from the world coordinate system (Xw, Yw, Zw) where the marker image is arranged to the camera coordinate system (Xc, Yc, Zc) according to the following equation See, for example, Non-Patent Document 1).

s: scale factor Then, the gaze point position acquisition unit 11 converts coordinates of the gaze point in the world coordinate system into the camera coordinate system using the transformation matrix Wc, w, and sets Zc as the scalar value Z _α of the depth distance calculate.

  The transformation matrix Wc, w may be calculated from a point correspondence set of the world coordinate system and the pixel coordinate system of the image plane using a robust estimation algorithm such as RANSAC (RANdom SAmple Consensus), for example. False point correspondences can also be removed from point correspondence sets.

  As shown in FIG. 6, when the optical axis direction of the camera coordinate system of the visible light camera can be approximated to the positive or negative direction of the normal vector of the virtual screen, between the origin of the coordinate system of the visible light camera and the gaze point The distance of can be taken as the gaze point distance.

As the three-dimensional position X ¹ w of the gaze point in the world coordinate system, an average value or a median value of vertex groups constituting the virtual object may be used. Also, for example, in the case of the work support system, when the position of the work instruction target can be acquired by another embodiment, this may be X ¹ w.

<When using a depth sensor>
When a depth sensor (depth camera) is used, not only the specific target object to which the marker image is attached but also the depth value of the entire visual recognition area can be acquired. The depth sensor can acquire the position of a real object as a gaze point position by capturing the reflection of infrared radiation emitted from the infrared radiation sensor.

[Overlap area calculation unit 12]
The overlap area calculation unit 12 calculates an overlap area where the right eye screen and the left eye screen which are viewed as “depth distance” through the display intersect each other from the right eye and the left eye of the user. The calculated coordinate information of the overlapping area is output to the drawing area selection unit 13.

ｓ：スケール係数 The overlapping area calculation unit 12 performs back projection on the depth position of the three-dimensional space via an arbitrary point of the right eye screen to the right eye screen (or the left eye screen to the left eye screen), and then the other left eye screen (or right eye screen). The following transformation, which shows the positional relationship between the coordinate system (u _r , v _r ) of the right-eye screen and the coordinate system (u _l , v _l ) of the left-eye screen, by back projection-projection calculation onto the screen Calculate the matrix H _{l, r} .

s: scale factor

The overlap area calculation unit 12 can calculate the displacement amount d defined by the following transformation matrix H _{l, r} by back projection-projection calculation when the screens of both eyes are shifted only in the left and right direction.

  FIG. 7 is an explanatory view showing the amount of displacement between both screens.

  According to FIG. 7, after the back projection to the depth position of the three-dimensional space through the upper left vertex of the right eye screen from the right eye, the displacement between the screens by the back projection-projection calculation projected to the other left eye screen The quantity d is represented.

ｓ：スケール係数 Further, the back projection-projection calculation of the overlapping area calculation unit 13 may be defined by the following equation.
Coordinate system of the right eye screen (u _r , v _r )
Coordinate system of left eye screen (u _l , v _l )
Camera coordinate system (Xc, Yc, Zc) used by gaze point position acquisition means
Right eye calibration parameter P _r
Left eye calibration parameter P _l

s: scale factor

The displacement amount d is calculated by any of the following.
(1) By substituting (u _r , v _r ) = (0, 0) and Z _c = Z _α (scalar value of depth distance), u ₁ is calculated as the displacement amount d.
(2) By substituting (u _l , v _l ) = (0, 0) and Z _c = Z _α (scalar value of depth distance), u _r is calculated as a displacement amount −d.
Thus, back projection position of the binocular screen plane defined by the Z _C = Z _alpha is obtained. By substituting the three-dimensional coordinates of this back projection position into the other equation, the corresponding point on the other screen is calculated.

[Drawing Area Selection Unit 13]
The drawing area selecting unit 13 selects only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area.

  FIG. 8 is an explanatory view showing the positional relationship between the left eye screen, the right eye screen, and the binocular screen and each region.

According to FIG. 8, the binocular screen is divided into four regions A to D by combining the left eye screen and the right eye screen according to the displacement d between the screens and the width w of the screens w It is done.
A area: an area displayed only with the left eye screen B area: an overlap area with the right eye screen in the left eye screen C area: an overlap area with the left eye screen in the right eye screen D area: an area displayed with the right eye screen only

  The range of the overlapping area changes in real time according to the focus position of the user's eye. Specifically, the smaller the wearer sees in the distance, the smaller the viewer's eye looks in the vicinity. For this purpose, the gaze point position acquisition unit 11 specifies the depth distance at which the user gazes, and determines in real time the range of the overlapping area according to the distance.

For each virtual object to be drawn (one object may be divided into two), if the drawing position of the object spans the overlapping area between the left and right screens, only the area on the left screen (area B) Decide whether to display in or the area (C area) of the right screen. Here, in the binocular screen, the position of the boundary in the range of the overlapping area is important. The drawing area selection unit 13 selects a screen on which a virtual object is to be drawn by one of the following two processes.
<Screen Selection Method Based on Visibility of Virtual Object>
<Screen selection method based on calibration parameter accuracy>

<Screen Selection Method Based on Visibility of Virtual Object>
A binocular screen composed of a right eye screen and a left eye screen is divided into a right eye area, an overlapping area and a left eye area.
Then, for each object, the drawing area is limited to one of the following.
Left eye screen (area A, B), right eye screen (area D), or
Left eye screen (region A) · Right eye screen (regions C, D)
The virtual object to be displayed across the right eye area and the overlapping area selects the right eye screen as a screen to be drawn in the overlapping area using both end coordinates of the virtual object with respect to the binocular screen and both end coordinates of the overlapping area Do.
In addition, a virtual object to be displayed across the left eye area and the overlapping area using the both end coordinates of the virtual object with respect to the binocular screen and the both end coordinates of the overlapping area is a left eye screen as a screen to be drawn in the overlapping area Choose

<Screen selection method based on calibration parameter accuracy>
If the accuracy of the calibration parameter P _r of the right eye is higher than the accuracy of the calibration parameter P _l of the left eye, the right eye screen is selected as the screen to be drawn in the overlapping area.
Left eye screen (region A) · Right eye screen (regions C, D)
Conversely, if the accuracy of the calibration parameter P _r of the right eye is lower than that of the calibration parameter P _l of the left eye, the left eye screen is selected as the screen to be drawn in the overlapping area.
Left eye screen (area A, B), right eye screen (area D)
Alternatively, if a specific condition regarding the display position of the virtual object is met, the screen selection method based on the visibility of the virtual object is used, otherwise, the screen selection method based on the accuracy of the calibration parameter is used May be

  FIG. 9 is a first explanatory diagram showing a drawing area for a virtual object.

  According to FIG. 9, for example, with regard to the vertex coordinates of each polygon constituting the virtual object, the projection position is calculated using the equation (3). This calculation is for grasping the approximate drawing position of the virtual object before actually rendering the virtual object. The calculation cost may be reduced by downsampling the vertex group.

<< Selection criteria when virtual objects exist only in two areas (left eye area and overlapping area, or overlapping area and right eye area) in the binocular screen (FIGS. 9 to 11) >>
According to FIG. 9, the minimum value and the maximum value of the u coordinate are defined as follows for the point group on which the point group constituting the virtual object is projected onto the screen based on the equation (3).
Drawing range of u coordinate of virtual object in left eye screen: (u _l ^min , u _l ^max )
Drawing range of u coordinate of virtual object on right eye screen: (u _r ^min , u _r ^max )
Here, when the virtual object is drawn by the left eye screen of the area A and the right eye screen of the area C, the continuity of the virtual object is divided at the overlapping area boundary.

  FIG. 10 is a second explanatory view showing a drawing area for a virtual object.

According to FIG. 10, with respect to FIG. 9, the virtual object is drawn only on the left eye screen of the regions A and B. According to FIG. 9 and FIG. 10, the following conditions are satisfied.
u _l ^min <d, d <u _l ^max , u _r ^max <w-d
In this case, the virtual object is preferably displayed only with the left eye screen (areas A and B). By this, the continuity of the virtual object only in the left eye screen can be maintained, and the visibility can be enhanced.

  FIG. 11 is a third explanatory view showing a drawing area for a virtual object.

According to FIG. 11, for the projected point group of the virtual object, the minimum value and the maximum value of the u coordinate satisfy the following conditions.
^{_{d <u l min, u r}} min <w-d, w-d <u r max
In this case, in order to maintain the continuity of the virtual object in the right eye screen, it is preferable to display the virtual object with only the right eye screen (areas C and D).

<< Selection criteria when virtual objects exist only in three areas (left eye area, overlapping area and right eye area) in a binocular screen (FIGS. 12 to 13) >>
FIG. 12 is a fourth explanatory view showing a drawing area for a virtual object.

Than the length range of the d-u _l ^min can not be displayed only in the left eye screen, the right eye can not be displayed only on the screen u _r ^max - the longer of (w-d) of the length of the range.
d-u _l ^min <u _r ^max- (w-d)
u _l ^min <d, w <u _l ^max , u _r ^min <0, w-d <u _r ^max
In this case, in order to give priority to the area that can be displayed only on the right eye screen, the virtual object preferably displays areas C and D on the right eye screen and area A on the left eye screen.

  FIG. 13 is a fifth explanatory view showing a drawing area for a virtual object.

According to FIG. 13, among the virtual objects, displaying the text portions to be asserted to the user as display in the areas A and B of the left eye screen can maintain the continuity of the text portions. In this case, u _l ^min , u _l ^max , u _r ^min and u _r ^max are extracted for text objects only, and the following conditions are satisfied.
d-u _l ^min > u _r ^max- (w-d)
In this case, it is preferable that the virtual object display areas A and B with the left eye screen and display area D with the right eye screen in order to prioritize the area that can be displayed only with the left eye screen. That is, among the virtual objects, the screen to be drawn is selected according to the size of the text object.

[Object drawing unit 14]
The object drawing unit 14 draws the virtual object on the selected right eye screen or left eye screen.

  FIG. 14 is an explanatory view showing a drawing procedure of the object drawing unit.

The object drawing unit 14 executes the following drawing process on the binocular screen. FIG. 14 shows the case where the right-eye screen is selected as a drawing target of the overlapping area.
(S11) For each virtual object, drawing is performed on only one screen which is not selected as the drawing target of the overlapping area. For example, the objects in FIGS. 12 and 14 and CG 2 in FIG. 15 draw only on the left eye screen. Also, the object in FIG. 13 and CG1 in FIG. 15 are drawn only on the right eye screen.
(S12) Fill the overlapping area of each of the left eye screen and the right eye screen with the transparent color of the display.
(S13) The virtual object is drawn on the other screen not drawn in (S11). For example, the objects in FIGS. 12 and 14 and CG 2 in FIG. 15 draw only on the right eye screen. Also, the object in FIG. 13 and CG1 in FIG. 15 are drawn only on the left eye screen.

  As described above in detail, according to the image display device, program, and method of the present invention, when displaying an object on the display of the optical see-through type binocular of eyeglass-like binocular, viewed from the user, The visibility of the object in the overlapping area of the screen can be improved.

  FIG. 15 is an explanatory view showing a screen appearing in the field of view of the user in the present invention.

  According to FIG. 15, for each virtual object, either the left eye screen or the right eye screen is selected to maintain its visibility. Therefore, with the optical see-through HMD, even if the viewing angle is enlarged (even when the operation is at the hand), the visibility of the virtual object in the overlapping area of the binocular screen is not impaired. That is, in the overlapping area, the virtual object is not displayed twice, and the luminance is also unified. According to the present invention, it is possible to prevent the deterioration of the visibility caused by the estimation error of the calibration parameter and the luminance difference.

  For the various embodiments of the present invention described above, various modifications, corrections and omissions of the scope of the technical idea and aspect of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The present invention is limited only as defined in the following claims and the equivalents thereto.

1 Image Display Device, Head Mount Display, Terminal 11 Attention Point Position Acquisition Unit
12 overlapping area calculation unit 13 drawing area selection unit 14 object drawing unit

Claims (12)

An optical see-through type binocular display disposed in front of each of the user's right and left eyes and visible through the real space, in accordance with the gaze point of the real object that the user gazes, An image display device for controlling a virtual object to be displayed on the display, wherein
Attention point position acquisition means for acquiring the depth position from the camera to the user's attention point;
Overlap area calculation means for calculating an overlap area where the right eye screen and the left eye screen which are viewed at the depth position through the display intersect from each of the right eye and the left eye of the user;
Drawing area selecting means for selecting either the right eye screen or the left eye screen for the virtual object to be displayed in the overlapping area;
An image display apparatus comprising: object drawing means for drawing the virtual object on the selected right eye screen or left eye screen. The drawing area selecting unit
The binocular screen composed of the right eye screen and the left eye screen is divided into a right eye area, an overlapping area and a left eye area,
Using the end coordinates of the virtual object with respect to the binocular screen and the end coordinates of the overlapping area,
The virtual object to be displayed across the right eye area and the overlapping area selects the right eye screen as a screen to be drawn in the overlapping area,
2. The image display apparatus according to claim 1, wherein the virtual object to be displayed across the left eye area and the overlapping area selects the left eye screen as a screen to be drawn in the overlapping area. The drawing area selecting unit
Regarding the virtual object to be displayed in the overlapping area,
If the accuracy of the calibration parameter P _r of the right eye is higher than the accuracy of the calibration parameter P _l of the left eye, select the right eye screen as the screen to be drawn in the overlapping area,
Conversely, if the accuracy of the calibration parameter P _r of the right eye is lower than that of the calibration parameter P _l of the left eye, the left eye screen is selected as the screen to be drawn in the overlapping area.
The image display apparatus according to claim 1, The back projection-projection calculation of the overlapping area calculation means is performed after the back projection to the depth position of the three-dimensional space through an arbitrary point of the right eye to the right eye screen (or the left eye to the left eye screen). back projection projecting into the eye screen (or the right-eye screen) - by projection calculation, the right eye screen coordinate system (u _{r, v r),} the left eye screen coordinate system (u _{l, v l)} a positional relationship of To calculate the following transformation matrix H _{l, r}

The image display apparatus according to any one of claims 1 to 3, wherein s: scale factor. The overlapping area calculation means calculates a displacement amount d defined by the following transformation matrix H _{l, r} by back projection-projection calculation when the screens of both eyes are shifted only in the left and right direction.

The image display apparatus according to claim 4, characterized in that: The overlapping area calculation means performs back projection on the depth position in three-dimensional space through an arbitrary point of the right eye screen to the right eye screen (or left eye to the left eye screen), and then the other left eye screen (or right In the case where the screens are shifted only in the left-right direction, a displacement amount d defined by the transformation matrix H _{l, r} is calculated by back projection-projection calculation that projects onto the eye screen). The image display apparatus as described in 4 or 5. The back projection-projection calculation of the overlapping area calculation means is performed using the coordinate system (u _r , v _r ) of the right eye screen, the coordinate system (u _l , v _l ) of the left eye screen, and the calibration parameters of the right eye It is defined by the following equation using P _r , the calibration parameter P _{l of} the left eye, and the camera coordinate system (Xc, Yc, Zc) used in the gaze point position acquisition means,

s: Calculate u _l as a displacement amount d by substituting a scale factor (u _r , v _r ) = (0, 0) and Z _c = Z _α (scalar value of depth distance),
Or
By substituting (u _l , v _l ) = (0, 0) and Z _c = Z _α (scalar value of depth distance), u _r can be calculated as a displacement amount −d. The image display device according to When using the visible light camera, the gaze point position acquiring unit stores in advance a marker image for specifying the gaze point, and detects the marker image from the photographed image, thereby arranging the marker as follows: Calculate transformation matrix (position and orientation parameters) Wc, w from the coordinate system (Xw, Yw, Zw) to the camera coordinate system (Xc, Yc, Zc),

s: scale factor The gaze point position acquiring unit converts coordinates of the gaze point in the world coordinate system into the camera coordinate system using the transformation matrix Wc, w, and calculates Zc as the scalar value Z _α of the depth distance The image display apparatus according to any one of claims 1 to 7, characterized in that: The image display apparatus according to claim 8, wherein the point-of-gaze position acquisition means sets, as the point of gaze, barycentric coordinates in a region of the marker image. The object drawing means
For each virtual object, draw only on one screen not selected as the drawing target of the overlapping area,
Fill each overlapping area of the left eye screen and right eye screen with the transparent color of the display,
The image display apparatus according to any one of claims 1 to 9, wherein the virtual object is drawn on the other screen not drawn. An optical see-through type binocular display disposed in front of each of the user's right and left eyes and visible through the real space, in accordance with the gaze point of the real object that the user gazes, A program that causes a computer mounted on a device to control a virtual object to be displayed on the display to function.
Attention point position acquisition means for acquiring the depth position from the camera to the user's attention point;
Overlap area calculation means for calculating an overlap area where the right eye screen and the left eye screen which are viewed at the depth position through the display intersect from each of the right eye and the left eye of the user;
Drawing area selecting means for selecting either the right eye screen or the left eye screen for the virtual object to be displayed in the overlapping area;
A program causing a computer to function as object drawing means for drawing the virtual object on the selected right-eye screen or left-eye screen. An optical see-through type binocular display disposed in front of each of the user's right and left eyes and visible through the real space, in accordance with the gaze point of the real object that the user gazes, An image display method of an apparatus for controlling a virtual object to be displayed on the display, wherein
The device
A first step of acquiring a depth position from a camera to a user's gaze point;
Calculating, from each of the right eye and the left eye of the user, an overlapping area where the right eye screen and the left eye screen viewed at the depth position through the display intersect;
A third step of selecting either the right eye screen or the left eye screen for the virtual object to be displayed in the overlapping area;
And a fourth step of drawing the virtual object on the selected right-eye screen or left-eye screen.

Images