NZ621149B2

NZ621149B2 - System and method for image registration of multiple video streams

Info

Publication number: NZ621149B2
Application number: NZ621149A
Authority: NZ
Inventors: Drew Steven Deaton; Marcus W Dillavou; Barton L Guthrie; Matthew Benton May; Mahesh B Shenai; Phillip Corey Shum
Original assignee: Help Lightning Inc; Uab Research Foundation
Priority date: 2011-08-12
Filing date: 2012-08-10
Publication date: 2016-03-30

Abstract

Disclosed is a method for image registration. The method comprises receiving a first image representing a first element from a local device; and receiving a second image representing a second element physically disposed remotely from the first element from a remote device. The second image is received in a transformed state resulting, at least in part, from a skew, shift, crop, translation, or combination thereof of the second image during transmission. A common field of interest is generated based on the first image and the second image such that the presence of at least one of the first element and the second element is spatially registered relative to the other of the first element and the second element. The common field of interest is generated by de-transformation of at least the second image and wherein the de-transformation comprises applying one or more image transforms to one or more of the first image and the second image. The one or more image transforms are based on a first transformation characteristic of one or more of the local device and the remote device and a second transformation characteristic resulting from transmission of one or more of the first image and the second image. The common field of interest is rendered and the first element and the second element in the common field of interest are rendered to a remote viewer and a local viewer with a substantially similar alignment and orientation. ed in a transformed state resulting, at least in part, from a skew, shift, crop, translation, or combination thereof of the second image during transmission. A common field of interest is generated based on the first image and the second image such that the presence of at least one of the first element and the second element is spatially registered relative to the other of the first element and the second element. The common field of interest is generated by de-transformation of at least the second image and wherein the de-transformation comprises applying one or more image transforms to one or more of the first image and the second image. The one or more image transforms are based on a first transformation characteristic of one or more of the local device and the remote device and a second transformation characteristic resulting from transmission of one or more of the first image and the second image. The common field of interest is rendered and the first element and the second element in the common field of interest are rendered to a remote viewer and a local viewer with a substantially similar alignment and orientation.

Description

SYSTEM AND METHOD FOR IMAGE REGISTRATION OF MULTIPLE VIDEO STREAMS FEDERAL GOVERNMENT SUPPORT CLAUSE Portion of the t disclosure were developed with Government funds provided by the Department of Energy under Grant/Contract numbers H30912, H34612, and H35662. The Government has certain rights in this invention.

CROSS REFERENCE TO RELATED PATENT APPLICATION This application claims priority to US. Patent Application No. 13/208,926 ﬁled August 12, 2011, herein incorporated by reference in its BACKGROUND Video combiners in extensive use today combine video from two separate sources using various multiplexing and media combining technology.

When two video sources are combined or multiplexed the two video s are not necessarily in a d alignment. Misalignments are often due to lack of synchronization between the video sources.

As an e, conventional video streaming rely on algorithms to atically remove data during compression and decompression processes that allow more efﬁcient transmission of video/image feeds. Loss of data and changes of resolution, aspect ration, skew, and other factors impact (e.g. transform) the raw viewing ofvideo. Accordingly, when two video s are combined. from two distinct sources, the resultant composite video or series of images can be misaligned.

SUMMARY Disclosed are systems and methods for establishing image registration of multiple video/image streams to align multiple video/image streams for a desired rendering of . A Spatial Image Registration System (SIRS) uses detection s, systems, symbols and/or detection points introduced within the images of separate video feeds to verify and if ary, apply multiple image transformations to perform an alignment between the original and transmitted feeds.

In the case of virtual interactive ce (VIP), SIRS aides in creating a common field of interest with users interactions spatially ered or matched regardless of transmission or display differences. [0005a] ing to a first , the present ion provides a method for image registration comprising: receiving a first image from a local device, the first image representing a first element; receiving a second image from a remote device, the second image representing a second element physically disposed remotely from the first element, wherein the second image is received in a transformed state resulting, at least in part, from a skew, shift, crop, translation, or combination thereof of the second image during transmission; generating a common field of interest based on the first image and the second image such that the ce of at least one of the first element and the second element is spatially registered relative to the other of the first element and the second element, wherein generating the common field of interest ses a de-transformation of at least the second image and wherein the de-transformation comprises applying one or more image transforms to one or more of the first image and the second image, and wherein the one or more image transforms are based on a first transformation characteristic of one or more of the local device and the remote device and a second transformation characteristic resulting from transmission of one or more of the first image and the second image; and rendering the common field of interest, wherein the first element and the second element in the ite image are rendered to a remote viewer and a local viewer with a substantially similar alignment and orientation. [0005b] According to a second aspect, the present invention provides a method for image registration comprising: generating, using a local device, a first image representing a first element; generating, using a remote device, a second image enting a second element, wherein the second element is physically disposed remotely from the first element, and wherein at least one of the first image and the second image is part of a live video ; determining an image orm based on a transformed state of one or more of the first image and the second image, wherein the ransformed state of the one or more of the first image and the second image is based, at least in part, on a skew, shift, crop, translation, or combination thereof of the one or more of the first image and the second image caused by transmission of the one or more of the first image and the second image due at least in part to compression or decompression, or both of the one or more of the first image and the second image, and wherein the transformed state is based on a transformation characteristic of each of the local device and the remote ; and rendering a composite image including the first image and the second image, n at least one of the first image and the second image is spatially registered relative to the other of the first image and the second image based upon a registration feature inserted onto at least one of the first image and the second image, wherein the first image and the second image in the composite image are rendered to a remote viewer and a local viewer with a substantially similar alignment and ation and wherein rendering the common field of interest comprises a de-transformation of one or more of the first image and the second image, and wherein the de-transformation comprises applying the image transform to one or more of the first image and the second image. [0005c] According to a third aspect, the present invention provides a system for image registration comprising: a y configured for displaying a common field of st; a sensor ured for obtaining image data; a processor in signal communication with the display and the sensor, wherein the processor is configured to perform steps comprising, rendering a common field of interest that reflects a ce of a plurality of elements based upon the image data, wherein at least one of the elements is a remote element located remotely from another of the elements; determining an image transform based on a transformed state of the image data, wherein the transformed state of the image data is based, at least in part, on a skew, shift, crop, translation, or combination thereof of one or more images represented by the image data and a transformation characteristic of one or more of the y, the sensor, and the processor; updating the common field of interest such that the presence of the at least one of the elements is registered relative to another of the elements, wherein updating the common field of interest comprises a nsformation of the image data and wherein the de-transformation comprises applying the image transform to the image data; and outputting the common field of interest to the display.

Methods are described for image registration or co-localization. One method comprises: rendering a common field of interest that reflects a presence of a plurality of elements, wherein at least one of the elements is a remote element located remotely from r of the elements; and updating the common field of interest such that the presence of the at least one of the elements is ered relative to r of the elements.

Another method comprises: generating a first image enting a first t; generating a second image representing a second element ed remotely from the first element; and rendering a composite image including the first image and the second image, wherein at least one of the first image and the second image is registered relative to the other of the first image and the second image based upon a registration feature.

A system for image registration is described. The system comprises: a display ured for displaying a common field of interest; a sensor configured for obtaining image data; a processor in signal communication with the display and the sensor, wherein the processor is configured to m steps sing, rendering a common field of interest that reflects a presence of a plurality of elements based upon the image data, wherein at least one of the elements is a remote element located remotely from another of the ts; updating the common field of st such that the presence of the at least one of the elements is registered relative to another of the elements; and outputting the common field of interest to the display.

Additional advantages will be set forth in part in the description which follows or may be learned by practice. The advantages will be realized and attained by means of the elements and combinations particularly d out in the appended inventive concepts. It is to be understood that both the foregoing general description and the following detailed description are exemplary and atory only and are not to be considered restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this speciﬁcation, illustrate embodiments and together with the description, serve to n the ples of the s and systems provided: Figure 1 illustrates virtual interactive presence; Figure 2A illustrates virtual interactive presence; Figure ZB illustrates a local expert ing a remote user; Figure 3 illustrates an exemplary registration system; Figure 4A illustrates an exemplary process med by the registration system of Figure 3; Figure 48 illustrates a local ﬁeld and a remote ﬁeld of an exemplary process performed by the registration system of Figure 3; Figure 4C illustrates a local ﬁeld and a remote ﬁled of an exemplary process performed by the registration system of Figure 3; Figure 4D rates an offset of a registration mechanism; Figure E illustrates the transformation of the offset of the registration mechanism of Figure 4D; Figure 4F illustrates an exemplary process performed by the registration system of Figure 3; Figure 5 illustrates an exemplary virtual presence system; Figure 6 illustrates exemplary processes performed within a graphics server; Figure 7 illustrates exemplary ses performed within a network server; Figure 8 illustrates a side View of an exemplary VIP display; Figure 9 illustrates a user’s View of an exemplary VIP display; Figure 10 illustrates a user’s View of an exemplary VIP display; Figure l 1 illustrates an ary method; Figure 12 illustrates another exemplary method; Figure 13 illustrates virtual presence in a remote surgical nment; Figure 14 illustrates merging of medical imaging with an operative ﬁeld; and Figure 15 illustrates an exemplary operational environment.

DETAILED DESCRIPTION [001 1] Before the present methods and systems are disclosed and described, it is to be understood that the s and systems are not limited to speciﬁc synthetic methods, speciﬁc components, or to particular compositions, as such may, of course, vary. [t is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

As used in the cation and the appended inventive ts, the singular forms “a, ’5 66an” and “the” include plural nts unless the context y dictates otherwise.

Ranges may be expressed herein as from “about” one particular value, and/or to ” another particular value. When such a range is expressed, r embodiment includes from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent “about,” it will be understood that the particular value forms another embodiment. It will be further understood that the endpoints of each of the ranges are signiﬁcant both in relation to the other endpoint, and independently of the other endpoint.

“Optional” or “optionally” means that the uently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not. hout the description and claims of this cation, the word “comprise” and variations of the word, such as “comprising” and “comprises,” means “including but not limited to,” and is not intended to exclude, for example, other additives, components, integers or steps. “Exemplary” means “an example of” and is not intended to convey an indication of a preferred or ideal embodiment.

Disclosed are components that can be used to perform the disclosed methods and systems. These and other components are disclosed herein, and it is understood that when combinations, subsets, interactions, groups, etc. of these ents are disclosed that while speciﬁc reference of each various individual and tive combinations and permutation of these may not be explicitly disclosed, each is speciﬁcally plated and described herein, for all methods and. systems. This applies to all aspects of this application including, but not limited to, steps in disclosed methods. Thus, if there are a variety of additional steps that can be performed it is understood that each of these additional steps can be performed with any speciﬁc embodiment or combination of embodiments of the disclosed methods.

The present s and systems may be understood more readily by reference to the following detailed description of preferred embodiments and the Examples included therein and to the Figures and their us and. following description.

Disclosed are methods and systems for image registration of multiple video streams. The disclosed methods and systems can utilize virtual reality.

Virtual reality (VR) refers to a computer-based application which provides a human-computer interface such that the computer and its devices create a sensory nment which is dynamically controlled by the actions of the individual, so that the environment appears “real” to the user. With VR, there is communication n a computer system and a user. The computer creates a sensory nment for the user to experience which may be, in one aspect, multisensory ugh this is not essential) and the computer creates a sense of reality in se to user inputs.

In one ary aspect, the system disclosed can e at least two types of VR, lmmersivc and Non-immersive. lmmersivc VR creates the on that the user is actually in a different environment. In one aspect, the system accomplishes this through the use of such s as Head Mounted Displays (HMD’s), earphones, and input devices such as gloves or wands. In another aspect, in order to enhance to realism of the experience, a plurality of Degrees of Freedom (DOF’s) are utilized, which the software can simulate. lly, the more the DOF’s, the better the realism of the experience. Exemplary DOF's include, without limitation: X,Y,Z, roll, pitch, and yaw. mersive VR creates an environment that is differentiable from the user's surrounding environment. It does not give the illusion that the user is transported to another world. Non-immersive VR works by creating a 3- dimensional image and surround sound through the use of stereo projection systems, computer monitors, and/or stereo speakers. Non-immersive VR can be run from a personal computer without added hardware.

In one aspect, movement in I'm-mersive VR can be realized by a system h the use of optical, acoustical, magnetic, or mechanical hardware called trackers. Preferably, the input devices have as many of these trackers as le, so that movement can be more accurately represented. For instance, virtual gloves can have up to 3 trackers for each index, and more for the palm and wrist, so that the user can grab and press objects. In one , the trackers can be equipped with positioning sensors, that tell a computer which direction the input is facing and how the input device is tilted in all directions. This gives a sensor with six s of freedom.

Output devices bring the user to the virtual world. An e of an output device that can be used in the present system include, without limitation, head mounted displays (HMD) in the form of s or goggles, which allow a user to wear a display system on their head. One approach to the HMD is to use a single Liquid l Display (LCD), wide enough to cover both eyes.

Another approach is to have two separated displays — one for each eye. This takes somewhat more computer power, since the images displayed are different.

Each display has a separate image rendered from the correct angle in the environment. Eye—tracking can be combined with HMDs. This can allow, for example, surgeons to move their eyes to the part of an image they want to enhance.

Another e of an output device that can be used in an embodiment of the present system is shuttered glasses. This device updates an image to each eye every other frame, with the shutter closed on the other eye. Shuttered glasses require a very high frame rate in order to keep the images from ﬂickering. This device is used for stereo monitors, and gives an te 3-d representation of a 2-d object, but does not immerse the user in the Virtual world.

Another output device that can be used in an embodiment of the present system is a screen with le projectors. The screen can be either a plane or bent. A challenge when using multiple projectors on the same screen is that there can be visible edges between the projections. This can be remedied be using a soft-edge system wherein the projection goes more and more transparent at the edges and the projections overlap. This produces an almost perfect tion n the images. In order to e a desired 3D effect, shuttered glasses can be used. Special glasses can be used, that alternate between making the glass either completely opaque or completely arent. When the left eye is , the right one is transparent. This is synchronized to the projectors that are projecting corresponding images on the screen.

In another aspect, a Cave Automatic Virtual Environment (CAVE) can also be used in the t system. A CAVE can use mirrors in a cube-shaped room to project stereo images onto the walls, giving the illusion that you are standing in a virtual world. The world is ntly updated using trackers, and the user is d to move around almost completely uninhibited.

Disclosed are methods and systems for image registration. Such methods and systems can render a number of elements/participants virtually present into a ﬁeld of interest in a manner such that the users can interact for any given purpose, such as the delivery of remote expertise. A ﬁeld of interest can comprise varying amounts of “real” and “virtual” ts, depending on a point of View. Elements can include any “real” or “virtual” object, subject, participant, or image representation. Various components of the disclosed methods and systems are illustrated in A common ﬁeld of st 101 can be a ﬁeld within which ts are physically and/or virtually present. Point of Reality (or Point of View) can refer to the vantage of the element/participant that is experiencing the common ﬁeld of interest. In exemplary points of reality, or points of View, are shown at 102» and 103, representing displays. The common ﬁeld of interest 101 can appear similar from both vantages, or points of View, but each comprises differing combinations of local (physical) and remote (virtual) elements/participants.

Local elements can be elements and/or participants which are physically present in the common ﬁeld of interest. In t A 105 is a local element for ﬁeld A 104 and is physically present in ﬁeld A 104. Element B 107 is a local element for ﬁeld B 106 and is physically present in ﬁeld B 106. It is understood that virtual elements (not shown) can be inserted or overlaid in ﬁeld A 104 and/or ﬁeld B 106, as desired.

Remote elements can be elements and/or participants that are not physically present in the common ﬁeld of interest. They are experienced as “virtually present” from any other local vantage point. As shown in element B 107 is a remote element to ﬁeld A 104 and is lly present in ﬁeld A 104. t A 105 is a remote element in ﬁeld B 106 and is Virtually present in ﬁeld B 106.

Methods for rendering a virtual interactive presence by combining local and remote elements and/or participants can comprise one or more of the following steps. A common local ﬁeld can be rendered in a manner that s the presence of the ﬁeld, elements and/0r participants. As shown in , Participant A can experience real elements in ﬁeld A through a viewer. The common local ﬁeld can be rendered such that it is experienced remotely in a manner that s remote participants to experience it similarly to the local persons. As shown in , this is rated by Participant A experiencing element B as virtually present in ﬁeld A.

Remote persons can insert themselves and/or interact with the virtual ﬁeld as rendered to them. For example, Participant A can insert hands, instruments, etc. into ﬁeld A and interact with the Virtual clement(s) B. Viewer B can view a al compliment’ to this, with Viewer B’s real elements interacting with ipant A’s virtual elements.

The common local ﬁeld can be continuously updated such that the presence of the remote participants can be rendered in real time. For e, the remote scene can be the most up-to-date available with the time lag between the remote capture and the local render kept as low as possible. Conversely, if there is a need to introduce a timing difference, this can be accomplished as well.

The common local ﬁeld can be scaled to a size and depth to meaningfully match the local scene. And the common local ﬁeld can be conﬁgurable, such that remote elements can be made more or less transparent, removed entirely, or otherwise altered to suit the needs of the local user.

Each ﬁeld is captured by a digital . The resulting image is ally distorted from its reality, based upon the physical characteristics of the camera. A processor, therefore, receives and displays a “physically” ted version of the local reality. Likewise, a digital camera also captures the remote ﬁeld(s), but the incoming stream is d through a transmission device and across a k. The processor, therefore, receives the remote stream that contains both physical and transmission-based tion. The processor must then apply a series of transformations that removes the al and transmission-based distortion from the common local ﬁeld.

The local participants can experience the virtually present participants in a manner that enables continuous interaction in the common local ﬁeld. FIG. ZB illustrates a local expert assisting a remote user. The hands of the local expert 201 are slightly transparent and superimposed. into the ﬁeld that is viewed by the remote user. The remote user can View the local expert’s hands, the remote user‘s hands and a puzzle located at the remote user’s location. The local expert is assisting the remote user in assembling a . illustrates an exemplary registration system 300. As shown, the registration system 300 can comprise a ﬁrst display 302 and a second display 304 conﬁgured for displaying one or more of an image, a video, a composite video/image, and a common ﬁeld of interest, for example. However, it is tood that any number of displays can be included in the system 300. In certain aspects, the second display 304 can be disposed ly from the ﬁrst display 302. As an example, each of the ﬁrst display 302 and the second display 304 can be conﬁgured to render the common ﬁeld of interest thereon. As a further example, each of the ﬁrst display 302 and the second display 304 can be conﬁgured to render at least one of the local ﬁeld and the remote ﬁeld thereon.

In certain aspects, at least one of the ﬁrst display 302 and the second display 304 can be a VIP display, as described in further detail herein. r, it is understood that each of the ﬁrst display 302 and the second y 304 can be any type of display including a monoscopic display and a stereoscopic display, for example. It is understood that any number of any type of display can be used.

A ﬁrst sensor 306 can be in signal communication with at least the ﬁrst display 302 and can be conﬁgured for obtaining image data such as a Virtual presence data, for example. In n aspects, the ﬁrst sensor 306 can be one or more of a camera, an infrared sensor, a light sensor, a RADAR device, a SONAR device, a depth scan sensor, and the like. It is understood that the ﬁrst sensor 306 can be any device or system capable of capturing/obtaining an image data representative of at least one of a “real” element” and a “virtual” element.

A second sensor 308 can be in signal communication with at least the second display 304 and can be conﬁgured for obtaining image data such as virtual presence data, for example. In n aspects, the second sensor 308 can be one or more of a , an infrared sensor, a light sensor, a RADAR device, a SONAR device, a depth scan sensor, and the like. It is understood that the second sensor 308 can be any device or system capable of capturing/obtaining an image data representative of at least one of a “real” element” and a “virtual” element. It is further understood that any number of sensors can be used.

A plurality of sors 310, 312 can be in direct or indirect signal ication with at least one of the ﬁrst display 302, the second display 304, the ﬁrst sensor 306, and the second sensor 308. Each of the processors 310, 312 can be conﬁgured to render the image data collected by the sensors 306, 308 onto at least one of the displays 302, 304. It is understood that the sors 310, 312 can be conﬁgured to modify the image data and the resultant image for transmission and display. It is further understood that any number of processors can be used, including one. In certain aspects, the system 300 ses only the processor 310, 312 in data communication with each other.

In certain aspects, each of the displays 302, 304 can comprise an ated one of the processors 310, 312 for rendering images onto the displays 302, 304. Each of the processors 310, 312, or another system comprising a processor, can icate with each other through a network connection. For example, remote sites can t Via the Internet. Tasks can be divided amongst each of the processors 310, 312. For example, one of the processors 310, 312 can be conﬁgured as a graphics processor or graphics server and can gather images from one of the sensors 306, 308 and/or a network server, perform an image composition tasks, and drive one or more of the displays 302, 304. illustrates ary processes 400 that can be performed with at least one the processors 310, 312. As shown, the processors 310, 312 can be conﬁgured to render a common ﬁeld of interest that reﬂects a presence of a plurality of elements based upon the image data obtained by at least one of the sensors 306, 308. As an example, at least one of the elements rendered in the common ﬁeld of interest can be a remote t ally located remotely from another of the elements. In certain aspects the processors 310, 312 can be conﬁgured to update the common ﬁeld of interest such that the presence of at least one of the elements is registered relative to another of the elements. The processors 310, 312 can also be conﬁgured to render/output the common ﬁeld of interest to at least one of the displays 302, 304. As an example, the processors 310, 312 can render interaction between a remote user and a local user in the common ﬁeld of interest. As a further example the presence of the remote t can be rendered in real time to the local user and. the ce of a local element can be rendered in real time to the remote user.

In step 402, a ration ism can be generated. As an example, the registration mechanism can be an image including a registration feature having a pre-determined location and arrangement. In an aspect, at least one of the ﬁrst sensor 306 and the second sensor 308 can capture image data to generate a registration image representing the registration mechanism. As an example, a pro-deﬁned pattern of dots or s having any shape and size can be digitally inserted into the registration image to deﬁne the registration feature. As a further example, the registration feature can be a virtual image overlaid or combined with the registration image.

An exemplary case is shown in , wherein the sensors 306, 308 capture image data to produce a registration mechanism. As illustrated in , each of a pair of coordinate systems 421, 431 with identical dimensions is disposed in each of a local ﬁeld 422 and a remote ﬁeld 432. As an e, each of the coordinate systems 421, 431 can be a square grid system. However, other coordinate systems can be used. The coordinate systems 421, 431 are situated identically in position and orientation relative to the respective sensors 306, 308 in each of the respective ﬁelds 422, 432. In an aspect, mounts 424, 434 d to the sensors 306, 308 are perpendicular to an edge of the respective coordinate system 421, 431. A ﬁducial 425, 435 or reference point can be deﬁned in substantially cal or identical pre-determined locations in each of the ﬁelds 422, 432 based upon the nate systems 421, 431, such as a corner of a lower—left nt, for example.

In an aspect, a pre—deﬁned ﬁrst pattern of dots 426 or s can be disposed on the coordinate system 421 as a registration e, and. the nates of each of the dots 426 with respect to the ﬁducial 425 can be stored.

A second pattern of dots 436 or markers can be disposed on the coordinate system 431 with respect to the ﬁducial 435 so that a coordinate of each of the dots 436 of the second pattern corresponds to a coordinate of one of the dots 426 of the ﬁrst pattern. Image data of the ﬁeld 422 captured by the sensor 306 can be deﬁned as the registration mechanism including the dots 426 as the registration feature. As an example, data relating to the nates of each of the dots 426, 436 can be stored on a storage device for subsequent retrieval. As a further example, the data relating to the coordinate of each of the dots 426, 436 can be transmitted n the processors 310, 312 ofthe system 300 to calibrate each of the ﬁelds 422, 432.

In an exemplary aspect, illustrated in , a spatial tracking, motion ng, or similar system can be used in lieu of a grid. As an example, in , a Flock of Birds sion Technology Corp, Burlington, VT) motion tracking system is used. However, other spatial tracking and/or motion tracking systems can be used. As shown in , a ity of planes 440, 450 with cal dimensions can be positioned in each ﬁeld 422, 432. The planes 440, 450 can be arranged to deﬁne a bottom of the respective ﬁelds 422, 432. As an example, the planes 440, 450 can be ed identically in position and orientation with respect to the sensors 306, 308 in each of the ﬁelds 422, 432. In an aspect, mounts 424, 434 coupled to the sensors 306, 308 can be perpendicular to an edge of the respective plane 440, 450. A ﬁducial 442, 452 or reference point can be recognized in identical pre-determined locations of each of the ﬁelds 422, 432 such as a corner of a lower-left quadrant, for example.

As an example, shown in , the ﬁducials 442, 452 are devices that are conﬁgured to transmit a pulsed DC magnetic ﬁeld. As a further example, a pro—deﬁned ﬁrst pattern of dots 444 or markers can be disposed on the plane 440 as a registration feature. The coordinates of each of the dots 444 with respect to the ﬁducial 442 can be noted by afﬁxing magnetic receivers (not shown) to each of the dots 444 to e the magnetic ﬁeld produced by the ﬁducial 442 at the various locations of the dots 444. From the measurements, the position and. coordinates of each of the dots 444 relative to the ﬁducial 442 can be ationally derived by the Flock of Birds system and stored. As an example, data ng to the nates of each of the dots 444 can be stored on a storage device for subsequent retrieval. As a further example, the data relating to the coordinate of each of the dots 444 can be transmitted n the processors 310, 312 of the system 300 to calibrate each of the ﬁelds 422, 432.

A second pattern of dots 454 or markers can be disposed on the plane 450 with t to the ﬁducial 452 and based upon the position and coordinates of each of the dots 444 relative to the ﬁducial 442. Accordingly, a coordinate of each of the dots 454 of the second pattern can pond to a coordinate of one of the dots 444 of the ﬁrst pattern, relative to the tive l 442, 452. It is understood that the on and coordinate ation relating to the dots 444, 454 can be transmitted between the ﬁelds 422, 432 or sor 310, 312 by a system external to the system 300. It is further understood that the position and coordinate information relating to the dots 444, 454 can be transmitted along with image data. Other methods for generating a registration mechanism can be used Returning to , the registration mechanism having the registration feature is transmitted to one of the processors 310, 312 for transformation and rendering, as shown in step 404. During the transmission of the registration mechanism, the underlying data can be compressed and decompressed and may be modiﬁed or lost. Accordingly, when the registration mechanism is received and rendered by the one of the processors 310, 312, the elements represented in the registration mechanism may be skewed, shifted, cropped, or transformed.

Likewise, the registration feature would also be transformed in a similar manner, whereby the resultant pattern and location of the registration feature would not match the pro-defined location and n prior to transmission.

In step 406, the one of the processors 310, 312» receives the registration mechanism to de-transform or register the elements represented thereby. As an example, the one of the processor 310, 312 analyzes the registration mechanism to er a location and arrangement of the registration feature. Since the registration feature has pre—determined characteristics such as location and arrangement, one or more ofthe processors 310, 312 can compare the current transformed on and arrangement to the known, pro-determined location and arrangement. As an illustrative example, shows the registration feature in an offset position 456 due to transformations caused form transmitting the registration mechanism. Accordingly, one or more of the sors 310, 312 can compare the offset position 456 of the registration feature to an known, original position 458.

From the comparison, the one of the processors 310, 312 can determine a set of transforms that should be applied to the registration mechanism in order to return the underlying data or image to the desired state (eg the al position 458), as shown in . One d in the art of image processing would understand that various orm methods and algorithms can be applied to an image to “dc-transform” the image to an original state. One or more of the processors 310, 312 can continue to apply transforms to the registration ism until the original arrangement and location of the registration features are achieved, as shown in . ' illustrates an exemplary process 460 that can be performed with the system 300. In step 462, the ﬁrst sensor 306 can e image data to generate a ﬁrst image representing a ﬁrst element. Alternatively, the ﬁrst image can be generated from stored image data or the like. As an example, the ﬁrst image can comprise a registration feature having a pre-determined location and arrangement relative to the ﬁrst image. In certain s, the registration feature can be inserted into the ﬁrst image as a detectable marker. For example, a pre- deﬁned pattern of dots can be digitally inserted into the image. As a r example, the registration feature can be a virtual image overlaid or combined with the ﬁrst image. However, any registration e or mechanism can be used.

The ﬁrst image comprising the registration feature can be transmitted to the remote processor 312 for ormation and rendering, at step 463. During the transmission of the ﬁrst image, the underlying data can be compressed and decompressed and may be modiﬁed or lost. Accordingly, when the ﬁrst image is received and ed by the processor 312, the elements represented in the ﬁrst image may be skewed, shifted, cropped, or transformed. Likewise, the registration feature would also be transformed in a manner similar to the ﬁrst image, whereby the resultant pattern and location of the registration feature would not match the pre—deﬁned location and pattern prior to transmission.

In step 464, the second sensor 308 can capture image data to generate a second image enting a second element disposed remotely from the ﬁrst element. It is understood that the second image can be generated from stored image data or the like. As an e, the second image comprises a registration feature having a pre-determined location and arrangement relative to the second image. In certain aspects, the registration feature can be ed into the second image as a detectable marker. For example, a pre—deﬁned pattern of dots can be digitally inserted into the image. As a further example, the registration feature can be a virtual image overlaid or combined with the second image.

In step 465, the second image having the registration feature can be transmitted to the second y 304 disposed. locally relative to the second sensor 308. During the transmission of the second image, the ying data can be altered, for formatting and compatibility with the second display 304.

Accordingly, when the second image is received and rendered onto the second display 304, the elements represented in the second image may be skewed, shifted, cropped, or transformed. Likewise, the registration feature would also be transformed in a manner similar to the ﬁrst image, whereby the resultant pattern and location of the registration feature would not match the pre-deﬁned location and pattern prior to transmission. Following this transmission, the second image rendered on the second display 304 can be analyzed by processor 312 to determine the transformation of the second image.

In step 466, the processor 312 transmits the ﬁrst image onto the second display 304 for rendering thereon. Similar to the rendering of the second image, a transformation may occur when ing the ﬁrst image to the second display 304. As such, each of the ﬁrst image and the second image may be transformed or d from intended previous state prior to ission and rendering.

Following this ission, the ﬁrst image rendered on the second display 304 can be analyzed by processor 312 to determine the ormation of the second image.

In an aspect, where transform ters are known based upon the components and. equipment being used, the processors 310, 312 can retrieve the known transform parameters and tically calculate reverse transforms in order to return the images to an original state. As an example, data relating to known transforms for particular ent and components can be stored and processed using a priori algorithms and s. However, other methods can be used.

In step 468, the processor 312 can render a composite image to the second y 304 including the ﬁrst image and the second image wherein each of the ﬁrst image and the second image can be registered relative to the other of the ﬁrst image and the second image based upon the registration feature. As an example, the processor 312 analyzes each of the ﬁrst image and the second image to discover a location and arrangement of the registration feature for each of the ﬁrst image and the second image. Since the registration feature for each of the ﬁrst image and the second image have known, pre-determined characteristics such as location and arrangement, the processor 312 can compare the current transformed location and ement to the known, pre-determined location and arrangement. From the comparison, the processor 312 can determine a set of transforms that should be applied to each of the ﬁrst image and the second image in order to return the ﬁrst image and the second image to a desired state. One skilled in the art of image processing would understand that s transform methods and algorithms can be applied to an image to “de- transform” the image to an original state. The processor 312 can continue to apply orms to at least one of the ﬁrst image and the second image until the original arrangement and location of the registration features is achieved.

In an aspect, the ration feature of the second image can be used as a local reference to which the registration feature of the ﬁrst image can be registered and aligned. As an example, the processor 312 can locate a centroid of each of the elements of the registration features for the ﬁrst image and the second image. As a further example, the ﬁrst image can be linearly shifted along any number of dimensional axes to align the ﬁrst image to the second image. It is understood that non-linear ng and alignment techniques can also be applied. It is further understood that the ﬁrst image can be padded or cropped in order to match the boundaries of the second. image (or some other reference image).

It is understood that the second image can be transmitted to the ﬁrst display and combined in a similar manner to generate a composite image on the ﬁrst display. In this way, the ﬁrst display 302 and the second display 304 render composite images representing the same alignment of the ﬁrst image and the second image relative to each other. Accordingly, a viewer of the second display 304 is assured that the ﬁrst image received from a remote location is rendered in an intended orientation and ent relative to the second image. se, a viewer of the ﬁrst display 302 is assured that the second image received from a remote location is ed in an intended orientation and alignment relative to the ﬁrst image that is generated local to the ﬁrst display 302.

In step 469, the composite image rendered on each of the ﬁrst display and the second display can be updated to reflect any changes to the ﬁrst image and/or the second image. For example, Where the sensors 306, 308 are video cameras, each sequentially captured video frame can represent a modiﬁed version of the ﬁrst image and the second image, tively. Accordingly, as the ﬁrst image and the second image are modiﬁed, the composite image can be updated and each of the modiﬁed ﬁrst image and the modiﬁed second image can be registered and aligned for rendering on the displays 302, 304. It is understood that any number of images and displays can be aligned in a similar manner as described herein. illustrates an exemplary virtual presence system. One such system can be used by each remote participant that is to join the same n.

Each system can communicate with each other through a network connection.

For e, remote sites can t via the intemet. Tasks can be divided amongst a plurality of computers in each system. For example, one computer (a graphics ) can gather images from local cameras and a network server, perform the stereo image composition tasks, and drive a local stereoscopic display system. As a further e, the processor(s) 310 of system 300 can be embodied by the graphics . illustrates exemplary processes that can be med with the graphics server. Images can be ed into local data structures (frame rings).

Local images can be gathered from a plurality of cameras, for example two cameras. Remote images can be provided by the network server via a high— speed remote direct memory access (RDMA) connection, for example. These images can be combined so that the remote user and the local user can be seen in the same scene (as in . This composite result can be itted to a local scopic display system. A second computer can act as the network server, which can perform network encoding/decoding tasks as well as depth map generation, for e. illustrates exemplary processes that can be performed with the k server. Local images gathered from the graphics server via the RDMA tion can be analyzed and mapped with depth information, d for efﬁcient network transmission, and sent to an external network connection to be received by a corresponding network server at the remote site. Simultaneously, encoded images and depth maps can be received from the remote site, d, and provided to the local graphics server via the RDMA connection.

The system can be user-controlled by a control terminal connected to the network server; the user can then access and control the cs server via the dedicated network connection to the network server. ters of virtual interactive presence can be red depending on the system used. Conﬁgurable ters include, but are not limited to, size of virtual elements, presence of virtual elements (opaque, translucent, etc.), time of virtual presence (time can be conﬁgured to be delayed, slowed, increased, etc.), mposition of elements such that any combination of virtual and real can be superimposed and/or 'ﬁtted' over one another, and the like. illustrates a side View of an exemplary VIP display. illustrates a user’s view of an exemplary VIP display. illustrates a user’s View of an exemplary VIP display.

As used herein, a “local” ﬁeld of interest can refer to a local physical ﬁeld and local user, thus making every other ﬁeld remote. Each ﬁeld can be local to its local physical user, but remote to other users. The composite of the ﬁelds can be a common ﬁeld of interest. This is distinct from common “virtual worlds” in that there can be components of “real” within the local rendering of the common ﬁeld of interest and interactions can be between actual video (and other) renderings of physical objects and not just graphic avatars representing users and objects. The methods and systems provided allow for virtual interactive presence to modify/optimize a physical domain by the lay of real and Virtual.

In an aspect, illustrated in , provided are methods for virtual interactive ce comprising rendering a common ﬁeld of interest that reﬂects the physical presence of a remote user and a local user at 1101, rendering ction n the remote user and the local user in the common ﬁeld of interest at 1102, and continuously updating the common ﬁeld of interest such that the presence of the remote user is rendered in real time to the local user and the presence of the local user is rendered in real time to the remote user at 1103.

The common ﬁeld of interest can be rendered such that the remote user experiences the common ﬁeld of interest similarly to the local user. The local user can ence the remote user’s physical presence in a manner that enables continuous interaction in the common ﬁeld of interest with the remote user. The s can further comprise rendering the al presence of a local object in the common ﬁeld and rendering interaction between the local user and the local object in the common ﬁeld. The methods can r comprise rendering the physical presence of a local object in the common ﬁeld of interest and rendering interaction between the remote user and the local object in the common ﬁeld of interest.

In another aspect, illustrated in , provided are methods for virtual interactive presence comprising rendering a local ﬁeld of interest that reﬂects the physical presence of a local object, a volumetric image of the local object, and a local user at 1201, rendering interaction between the local object, the volumetric image, and the local user in the local ﬁeld of interest at 1202, and continuously updating the local ﬁeld of interest such that the presence of the local object and the volumetric image of the local object is ed in real time to the local user at 1203.

The local object can be, for e, a patient and the volumetric image of the local object can be, for example, a medical image of a part of the patient.

However, the local object can be any object of interest and the image of the local object can be any accurate rendering of that object. For example, could be an automobile engine and a 3D graphic of the engine, etc.

The medical image can be, for example, one of, an x-ray image, an MRI image, or a CT image. The methods can further comprise superimposing, by the local user, the volumetric image onto the local object. The superimposition can be performed automatically by a computer.

The methods can further comprise adjusting, by the local user, a property of the volumetric image. The property can be one or more of transparency, spatial location, and scale.

The methods can further comprise rendering a local tool in the local ﬁeld of interest. The methods can further comprise rendering the local tool in te l relation to the ing of the local object. The tool can be any type of tool, for example, a surgical tool.

In another aspect, provided are systems for virtual presence, comprising a virtual presence display, conﬁgured for displaying a common ﬁeld of interest, a local sensor, conﬁgured for obtaining local virtual ce data, a network interface, red for transmitting local virtual presence data and receiving remote virtual ce data, and a processor, coupled to the virtual presence display, the local , and the network interface, wherein the processor is conﬁgured to perform steps comprising, rendering a common ﬁeld of interest that reﬂects the physical presence of a remote user and a local user based on the local l presence data and the remote virtual presence data, rendering interaction between the remote user and the local user in the common ﬁeld of interest, continuously updating the common ﬁeld of interest such that the ce of the remote user is rendered in real time to the local user and the presence of the local user is rendered in real time to the remote user, and outputting the common ﬁeld of interest to the virtual presence display.

The virtual presence display can be one or more of a stereoscopic y, a monoscopic display (such as a CRT, LCD, etc), and the like. The sensor can be one or more of a camera, an infrared sensor, a depth scan sensor, and the like. The common ﬁeld of interest can be rendered such that the remote user ences the common ﬁeld of st similarly to the local user. The local user can experience the remote user’s physical presence in a manner that enables continuous interaction in the common ﬁeld of interest with the remote user.

The processor can be further conﬁgured to perform steps comprising rendering the physical ce of a local object in the common ﬁeld of st and rendering interaction between the local user and the local object in the common ﬁeld of interest.

The processor can be further conﬁgured to perform steps comprising ing the physical presence of a local object in the common ﬁeld of interest and rendering interaction between the remote user and the local object in the common ﬁeld of interest.

Further provided are systems for virtual presence, comprising a virtual presence display, conﬁgured for displaying a local ﬁeld of interest, a local sensor, conﬁgured for obtaining local virtual presence data, a processor, coupled to the virtual presence y and the local sensor, wherein the processor is conﬁgured to perform steps comprising, rendering a local ﬁeld of interest that reﬂects the physical presence of a local object and a local user based on the local l presence data and a volumetric image of the local object, ing interaction between the local object, the volumetric image, and the local user in the local ﬁeld of interest, continuously ng the local ﬁeld of interest such that the presence of the local object and the tric image of the local object is rendered in real time to the local user, and outputting the local ﬁeld of interest to the virtual presence display.

The Virtual presence display can be one or more of a stereoscopic y, a monoscopic display (such as a CRT, LCD, etc.), and the like. The sensor can be one or more of a camera, an ed sensor, a depth scan sensor, and the like.

The local object can be, for example, a patient and the volumetric image of the local object can be, for example, a medical image of a part of the patient.

The medical image can be, for example, one of, an x-ray image, an MRI image, or a CT image. However, the local object can be any object of interest and the image of the local object can be any accurate rendering of that object. For example, could be an automobile engine and a 3D graphic of the engine, etc.

The processor can be further conﬁgured to perform steps comprising superimposing, by the local user, the volumetric image onto the local object.

The sor can be r conﬁgured to perform steps comprising adjusting, by the local user, a property of the volumetric image. The property can be one or more of transparency, spatial location, and scale.

The sor can be further red to perform steps comprising rendering a local tool in the local ﬁeld of st. The processor can be further conﬁgured to perform steps comprising rendering the local tool in accurate spatial relation to the rendered local object.

The disclosed methods and systems can have broad applications. For example, y, gaming, mechanics, munitions, battle ﬁeld presence, instructional s (training) and/or any other situation where ction is part of the scenario.

Also disclosed are methods and systems that enable a remote expert to be virtually present within a local surgical ﬁeld. Virtual interactive presence can be used to enable two surgeons remote from each other to interactively perform a surgical procedure. The methods and system enable two or more operators to be virtually present, and interactive, within the same real ive ﬁeld, thus supporting remote assistance and exporting surgical expertise.

The methods and systems can also be used to superimpose imaging data of the operative anatomy onto the anatomy itself for guidance and orientation (augmented, reality). The methods and systems can be used for training of students. The s and systems augment and enhance the ﬁeld of robotics by virtually bringing an expert into the robotic ﬁeld to guide the robot operator.

The methods and systems are applicable to endoscopic procedures by inserting the expert's hands directly into the endoscopic ﬁeld for guidance. The s and systems expand remote y by providing the assistance of a remote expert to an actual local n, whose basic skills can handle emergencies, and who will learn from the virtual interaction. The methods and systems can be used at trauma sites and other medical environments. The methods and systems can be used to provide remote assistance in other areas such as engineering, construction, architecture, and the like. The methods and systems sed can be used to transmit expertise to a remote 'site of need', merge contemporary imaging directly into the surgical ﬁeld, and train surgical students An exemplary remote surgical assistance system for transmitting al maneuvers of a local expert to a remote surgeon for the purpose of g/assisting the remote surgeon is illustrated in . The remote surgical ﬁeld can be viewed by the remote surgeon with a binocular video system. The video system can show the ﬁeld with his hands and instruments ming the procedure. The viewing system can be referred to as a surgical videoscope.

The binocular video rendering of the remote ﬁeld can be transmitted to the local expert), who can View the (now virtual) stereoscopic rendering of the procedure through a second al videoscope . The local expert can insert his hands into the virtual ﬁeld, thus seeing his real hands within the virtual ﬁeld.

The Video image of the local expert's hands can be transmitted back to the remote surgeon's surgical videoscope system superimposed into the real ﬁeld. The remote surgeon can then see the expert‘s virtual hands within his surgical ﬁeld in a spatially/anatomically relevant context. With this , the local expert can use his hands to show the remote surgeon how to perform the case.

Exemplary elements of the system can comprise a remote station where the remote n can perform the operative procedure, a remote surgical videoscope system sed of, for example, a ﬁxed stereoscopic videoscope that may resemble a mounted microscope. This apparatus can be used by the remote surgeon to view the operative ﬁeld. Any other type of suitable VIP display can be used. The system can project the binocular video image to a similar local surgical videoscope at a local station. The local surgical videoscope can receive the binocular video image of the remote procedure and allow the local expert to view it. The local videoscope can View the local surgeons hands as they move within the virtual remote ﬁeld as viewed through the local videoscope. The local cope can then transmit the local expert's hands back to the remote videoscope so that the remote surgeon can see the 's Virtual hands within the real ﬁeld.

With this system, the local expert can show the remote surgeon the riate maneuvers that result in successful completion of the case. The remote surgeon can have a basic skill set to carry out the new procedure.

Therefore, the local expert can simply demonstrates to the remote surgeon new ways to apply the skill set. This system does not have to supplant the remote surgeon, but can be used to enhance his/her capability. The remote n can be on hand to y deal with any ncies. Time delay is minimized because the remote n can use his/her own hands to perform the task, eliminating the need for the local expert to manipulate remote robotic apparatuses.

Also disclosed are methods and systems for merging contemporary medical imaging onto an operative ﬁeld. A volume image can be obtained of the operative ﬁeld. For example, a volume MRI ofthe head, prior to the surgical procedure. The image data can be reconstructed into a three dimensional rendering of the anatomy. This rendering can be itted to the surgical videoscope that will be used to view the operative ﬁeld. Through the videoscope, the surgeon can view this 3D rendering in a translucent manner superimposed onto the al ﬁeld. In this case, the surgeon would see a rendered head superimposed on the real head. Using software tools in the surgical videoscope interface, the surgeon can rotate and scale the rendered image until it “ﬁts” the real head. The videoscope system can allow the surgeon to differentially fade the rendered head and real head so that the surgeon can “look into” the real head and plan the y.

Exemplary elements of the system can comprise a surgical videoscope viewing system through which the surgeon views the surgical ﬁeld. A computer for reconstruction of a volume-acquired MRI/CT (or other) image with sufﬁcient resolution to enable ng it to the real surgical anatomy. The volume rendered image can be displayed through the videoscope system so that the surgeon can see it stereoscopically. A re interface can enable the surgeon to vary the translucency of the rendered and. real anatomy so that the rendered anatomy can be superimposed onto the real anatomy. The surgeon can “open up” the rendered anatomy to View any/all internal details of the image as they relate to the real anatomy. Surgical tools can be spatially registered to the rendered anatomy so that behavior can be tracked and applied to the image.

As shown in , an example of such a task is placing small objects inside a jar of dark n so that they are not visible to the surgeon. The task is for the surgeon to use a long forceps to reach into the gelatin and touch or grasp the objects. The Surgical Videoscope system can obtain a volume scan of the gelatin jar and render the jar in three dimensions and display a binocular rendering through the videoscope. The n can view the rendering and the real jar through the scope system and ﬁt the ed jar onto the real jar. By differentially adjusting translucency, the surgeon can reach into the real jar with a forceps and grasp a selected object, while avoiding other designated objects.

The ng instrument can be spatially registered onto the volumetric rendering of the surgical ﬁeld, thereby allowing a c of the tool to be displayed on the rendering of the al ﬁeld in appropriate ic orientation. This can provide enhanced guidance. This can be implemented by touching designated landmarks on the real object (jar) with a digitizer that communicates with the image rendering , thus deﬁning the object/probe relationship. Because the object (jar) is registered. to the image of the jar by superimposition, a graphic of the probe can be displayed in relation to the image of the jar ng virtual surgery.

There are many situations in which the present system can be used. For example, remote surgery, medical training, and tcle-medicine, which can be used for third world countries or in a military situation. Surgeons remotely d from patients can assist other surgeons near the patient, can assist medics near the patient, and can perform surgical operations when coupled to a robotic surgery . Other examples include, ted or ed surgery - normal surgery using virtual environments, an example of which is endoscopic surgery. Surgical procedures can also be simulated. Surgeons d remote from each other may plan and practice a procedure before carrying out the operation on a real patient.

Other applications include the preparation of patient before surgery, medical therapy, preventative medicine, exposure therapy, reducing phobias, training people with disabilities and skill enhancement, and the like.

The Viewer then views the projection through passive stereoscopic polarized glasses (similar to sunglasses) that route the left—eye image to the left eye, and the right-eye image to the right eye. This provides an illusion of psis when the correctly—offset images are properly rendered by the software. The system can be replaced by other types of stereoscopic displays with no functional detriment to the system. The stereoscopic y can comprise at least two y projectors ﬁtted with polarizing lenses, a back- projeetion screen material that maintains light polarization upon diffusion, special glasses that ct each eye to see only light of a particular polarization, and the viewer. The image to be Viewed can be rendered with two ly different View transformations, reﬂecting the different locations of the ideal viewer’s two eyes. One projector displays the image ed for the left eye’s position, and the other projector displays the image ed for the right eye’s position. The glasses restrict the light so that the left eye sees only the image rendered for it, and the right eye sees only the image rendered for it. The viewer, presented with a reasonable stereoscopic image, will perceive depth. is a block diagram illustrating an exemplary operating environment for performing the disclosed methods. This exemplary operating environment is only an example of an operating environment and is not ed to suggest any limitation as to the scope of use or functionality of operating environment architecture. r should the operating environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment.

The methods can be operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known ing systems, environments, and/or conﬁgurations that may be le for use with the system and method include, but are not limited to, personal computers, server computers, laptop devices, and multiprocessor systems. Additional examples include set top boxes, programmable consumer onics, network PCs, minicomputers, mainframe computers, distributed computing environments that e any of the above systems or devices, and the like.

The s may be described in the general context of computer instructions, such as program modules, being executed by a computer.

Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The system and method may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a ications network. In a distributed ing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The methods disclosed herein can be implemented via one or more l-purpose computing s in the form of a computer 1501. The components of the computer 1501 can include, but are not limited to, one or more processors or processing units 1503, a system memory 1512, and a system bus 1513 that s various system components including the processor 1503 to the system memory 1512.

The system bus 1513 represents one or more of several possible types of bus structures, including a memory bus or memory ller, a peripheral bus, an accelerated cs port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro l Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video onics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus also known as a Mezzanine bus. This bus, and all buses speciﬁed in this description can also be implemented over a wired or ss network connection. The bus 1513, and all buses speciﬁed in this description can also be implemented over a wired or wireless network connection and each of the subsystems, including the processor 1503, a mass storage device 1504, an operating system 1505, application software 1506, data 1507, a network adapter 1508, system memory 1512, an Input/Output Interface 1510, a display adapter 1509, a display device 1511, and a human machine interface 1502, can be contained within one or more remote computing devices 1514a,b,c at physically separate locations, connected. through buses of this form, in effect implementing a fully distributed system.

] The computer 1501 typically includes a variety of er readable media. Such media can be any available media that is accessible by the computer 1501 and includes both volatile and non-volatile media, removable and non-removable media. The system memory 1512 includes computer readable media in the form of volatile memory, such as random access memory (RAM), and/or non-volatile memory, such as read only memory (ROM). The system memory 1512 typically ns data such as data 1507 and/or program s such as operating system 1505 and application software 1506 that are immediately accessible to and/or are presently operated on by the processing unit 1503.

The computer 1501 may also include other removable/non—removable, volatile/non-volatile computer storage media. By way of example, illustrates a mass storage device 1504 which can provide latile storage of computer code, computer readable instructions, data structures, program modules, and other data for the computer 1501. For e, a mass storage device 1504 can be a hard disk, a removable magnetic disk, a removable optical disk, magnetic cassettes or other magnetic e devices, ﬂash memory cards, CD-ROM, digital versatile disks (DVD) or other optical storage, random access memories (RAM), read only memories (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

Any number ram modules can be stored on the mass storage device 1504, including by way of example, an ing system 1505 and application software 1506. Each of the operating system 1505 and application software 1506 (or some combination thereof) may include elements of the programming and the application re 1506. Data 1507 can also be stored on the mass storage device 1504. Data 1507 can be stored in any of one or more databases known in the art. Examples of such databases include, DB2®, Microsoft® Access, Microsoﬁ® SQL Server, Oracle®, mySQL, PostgreSQL, and the like. The databases can be centralized or buted across multiple systems.

A user can enter commands and information into the computer 1501 via an input device (not shown). Examples of such input devices include, but are not limited to, a keyboard, pointing device , a “mouse”), a microphone, a joystick, a serial port, a scanner, tactile input devices such as gloves, and other body ngs, and the like. These and other input devices can be connected to the processing unit 1503 via a human e interface 1502 that is coupled to the system bus 1513, but may be connected by other interface and bus structures, such as a parallel port, game port, or a universal serial bus (USB).

A display device 1511 can also be connected to the system bus 1513 via an interface, such as a display adapter 1509. A computer 1501 can have more than one display adapter 1509 and a computer 1501 can have more than one display device 1511. For example, a display device can be a monitor, an LCD (Liquid Crystal Display), or a projector. In addition to the display device 1511, other output peripheral devices can e components such as speakers (not shown) and a printer (not shown) which can be connected to the computer 1501 via Input/Output Interface 1510.

The computer 1501 can operate in a networked nment using logical connections to one or more remote computing devices 1514a,b,c. By way of example, a remote ing device can be a personal computer, portable er, a server, a router, a network computer, a peer device or other common network node, and so on. Logical connections between the computer 1501 and a remote computing device 1514a,b,c can be made via a local area network (LAN) and a general wide area network (WAN). Such k connections can be h a network adapter 1508. A network adapter 1508 can be implemented in both wired and wireless nments. Such networking environments are commonplace in ofﬁces, enterprise-wide computer networks, intranets, and the Internet 1515. [001 10] One or more VIP displays 1516a,b,c,d,e can communicate with the computer 1501. In one aspect, VIP display 1516e can communicate with computer 1501 h the input/output interface 1510. This communication can be wired or ss. Remote VIP displays 1516a,b,c can communicate with computer 1501 by communicating ﬁrst with a respective remote ing device 1514a,b,c which then communicates with er 1501 through the network adapter 1508 via a network such as the Internet 1515. Remote VIP display 1516d can communicate with computer 1501 without the need for a remote computing device. Remote VIP display 1516d can communicate via a network, such as the Internet 1515. The VIP displays 1516a,b,c,d,e can communicate wireless or through a wired connection. The VIP displays 1516a,b,c,d,e can icate individual or tively as part of a VIP display network.

For purposes of illustration, ation programs and other executable program components such as the operating system 1505 are illustrated herein as discrete blocks, although it is recognized that such programs and components reside at various times in ent storage components of the computing device 1501, and are ed by the data processor(s) of the computer. An implementation of application re 1506 may be stored on or transmitted across some form of computer readable media. Computer readable media can be any available media that can be accessed by a computer. By way of e, and not limitation, computer readable media may comprise “computer storage media” and “communications media.” “Computer storage media” include volatile and non-volatile, removable and non-removable media ented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer e media includes, but is not limited to, RAM, ROM, EEPROM, ﬂash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a er. [001 12] Unless otherwise expressly , it is in no way intended that any method set forth herein be construed as requiring that its steps be performed in a speciﬁc order. Accordingly, where a method claim does not actually recite an order to be followed by its steps or it is not otherwise speciﬁcally stated in the inventive ts or descriptions that the steps are to be limited to a speciﬁc order, it is no way ed that an order be inferred, in any t. This holds for any possible non-express basis for interpretation, including: matters of logic with respect to arrangement of steps or operational flow; plain meaning derived from grammatical organization or punctuation; the number or type of embodiments described in the speciﬁcation.

It will be apparent to those d in the art that various modiﬁcations and variations can be made in the present methods and systems without departing from the scope or spirit. Other embodiments will be apparent to those skilled in the art from consideration of the speciﬁcation and practice disclosed . It is intended that the speciﬁcation and examples be considered as ary only, with a true scope and spirit being indicated by the following claims.

Claims

The claims defining the present invention:

1. A method for image registration comprising: receiving a first image from a local device, the first image representing a first t; receiving a second image from a remote device, the second image representing a second element physically disposed remotely from the first element, wherein the second image is received in a transformed state resulting, at least in part, from a skew, shift, crop, translation, or combination thereof of the second image during transmission; generating a common field of interest based on the first image and the second image such that the presence of at least one of the first element and the second element is spatially registered relative to the other of the first element and the second element, n ting the common field of interest comprises a de-transformation of at least the second image and wherein the de-transformation comprises applying one or more image transforms to one or more of the first image and the second image, and wherein the one or more image transforms are based on a first transformation characteristic of one or more of the local device and the remote device and a second transformation characteristic resulting from transmission of one or more of the first image and the second image; and rendering the common field of interest, wherein the first t and the second element in the common field of interest are rendered to a remote viewer and a local viewer with a substantially r alignment and orientation.

2. The method of claim 1, further comprising rendering the presence of each of the first element and the second element in real time, wherein each of the first element and the second element is registered relative to r of the elements.

3. The method of claim 1, further comprising the step of rendering ction between the first element and the second element in the common field of interest.

4. The method of claim 1, further comprising the step of continuously ng the common field of interest such that the presence of the first element and the second element is rendered in real time to a local viewer.

5. The method of claim 1, further sing the step of continuously updating the common field of interest such that the presence of the first element and the second t is rendered in real time to a remote viewer.

6. The method of claim 1, further sing the step of calibrating the common field of interest by aligning a calibration feature of a remote field and a calibration feature of a local field.

7. The method of claim 6, n the calibration features of the remote field and the local field are aligned relative to a pre-defined coordinate system.

8. The method of claim 6, wherein the calibration features of the remote field and the local field are aligned relative to a fined virtual fiducial.

9. A method for image registration comprising: generating, using a local device, a first image representing a first element; generating, using a remote device, a second image representing a second element, wherein the second element is physically disposed remotely from the first element, and wherein at least one of the first image and the second image is part of a live video ; determining an image transform based on a transformed state of one or more of the first image and the second image, wherein the transformed state of the one or more of the first image and the second image is based, at least in part, on a skew, shift, crop, translation, or ation thereof of the one or more of the first image and the second image caused by transmission of the one or more of the first image and the second image due at least in part to compression or decompression, or both of the one or more of the first image and the second image, and wherein the transformed state is based on a transformation characteristic of each of the local device and the remote device; and rendering a ite image including the first image and the second image, wherein at least one of the first image and the second image is spatially registered relative to the other of the first image and the second image based upon a registration feature inserted onto at least one of the first image and the second image, wherein the first image and the second image in the composite image are rendered to a remote viewer and a local viewer with a substantially similar alignment and orientation and n rendering the composite image comprises a de-transformation of one or more of the first image and the second image, and wherein the de-transformation comprises ng the image transform to one or more of the first image and the second image.

10. The method ing to claim 9, wherein the registration feature is a third image inserted onto at least one of the first image and the second image.

11. The method of claim 9, wherein the registration feature is a virtual image rendered in the composite image.

12. The method of claim 9, wherein at least one of the first image and the second image is part of a video stream.

13. The method of claim 9, r comprising the step of continuously updating the composite image to render the composite image in real time to at least one of a local viewer and a remote viewer.

14. A system for image registration comprising: a display configured for displaying a common field of interest; a sensor configured for obtaining image data; a processor in signal communication with the display and the sensor, wherein the processor is configured to perform steps comprising, rendering a common field of interest that reflects a presence of a ity of elements based upon the image data, wherein at least one of the elements is a remote element located remotely from another of the elements; determining an image transform based on a transformed state of the image data, n the transformed state of the image data is based, at least in part, on a skew, shift, crop, translation, or combination thereof of one or more images ented by the image data and a transformation characteristic of one or more of the display, the sensor, and the processor; updating the common field of interest such that the presence of the at least one of the elements is registered relative to another of the elements, wherein updating the common field of interest comprises a nsformation of the image data and n the nsformation comprises applying the image transform to the image data; and ting the common field of interest to the display.

15. The system of claim 14, wherein the sensor is at least one of a camera, an infrared sensor, a light sensor, a RADAR device, a SONAR device, and a depth scan sensor.

16. The system of claim 14, wherein the processor is configured to perform the step of continuously updating the common field of interest such that the presence of the remote t is rendered in real time to a local viewer.

17. The system of claim 14, wherein the sor is configured to perform the step of continuously updating the common field of interest such that the presence of a local one of the elements is rendered in real time to a remote viewer.

18. The method of claim 1, wherein the second image comprises a registration mechanism and wherein the ration mechanism is received in a transformed state resulting, at least in part, from a skew, shift, crop, translation, or combination thereof of the registration mechanism during transmission.