JP7560591B2 - Method, computing system, and computer-readable storage medium for controlling a user interface - Patents.com - Google Patents

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63/393,178, entitled "CONTROLLING A USER INTERFACE WITH A TRACKPAD AND A SMART WATCH," filed on July 28, 2022, and U.S. Provisional Patent Application No. 63/393,181, entitled "CONTROLLING A USER INTERFACE WITH AN XR SHORTCUT FEATURE AND A SMART WATCH," filed on July 28, 2022, both of which are incorporated by reference in their entirety.

[0002] Artificial reality, extended reality, or extra reality (collectively "XR") is a type of reality that is conditioned in some manner before presentation to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and/or derivative thereof. A variety of XR environments exist that allow users' representations to move around and talk to each other. However, the user interface (UI) and user experience (UX) have limitations. For example, user interfaces often rely on hand tracking, which does not always work effectively and may even be uncomfortable for users. That is, in the current existing UI/UX used by XR devices, controlling objects displayed in the user interface of the XR device is performed via hand gestures captured by the XR device. These hand gestures are not precise.

[0003] The present disclosure is directed to controlling a user interface using a wearable device, such as a smart watch, substantially as shown and/or described in connection with at least one of the figures, and as more fully set forth in the claims.

[0004] The present disclosure will be more fully understood from the detailed description given hereinafter and from the accompanying drawings of various embodiments, which should not be taken as limiting the embodiments described and illustrated herein, but are for illustration and understanding only.

FIG. 1 is a block diagram illustrating an overview of a device in which some embodiments of the disclosed technology can function. FIG. 1 is a wiring diagram of a head mounted display (HMD) according to some embodiments. FIG. 1 is a wiring diagram of a mixed reality HMD system including a mixed reality HMD and a core processing component. FIG. 1 illustrates using the display surface of a smartwatch as a trackpad and using the trackpad to control a cursor. FIG. 1 illustrates using the display surface of a smartwatch as a trackpad and using the trackpad to control a cursor. 1 is a flow diagram of some embodiments of a process for controlling a user interface in an extended reality (XR) device using a trackpad with a smartwatch. 1A-1C illustrate several embodiments of a virtual shortcut menu with virtual buttons around the smartwatch for controlling a user interface in a virtual environment generated by an XR device. 1A-1C illustrate several embodiments of a virtual shortcut menu with virtual buttons around the smartwatch for controlling a user interface in a virtual environment generated by an XR device. 1A-1D illustrate several embodiments of a display surface of a smartwatch. 1 is a flow diagram of some embodiments of a process for controlling a user interface on an XR device using an XR shortcut interface in conjunction with a smartwatch or other wearable device.

[0013] In the following description, numerous details are set forth in order to provide a more thorough explanation of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the present disclosure.

[0014] A method and apparatus for controlling a user interface using a wearable device, such as a smart watch, is disclosed. In some embodiments, the method includes detecting the wearable device with an extended reality (XR) device, communicatively connecting the XR device to the wearable device, and using the wearable device to manipulate a user interface in a virtual environment displayed on a display in the XR device. The manipulation can be based on a user performing a movement, such as a movement with one or more fingers, across a display touch surface of the wearable device. The manipulation can also be based on finger or hand movements made in relation to the wearable device along with one or more virtual graphical user elements (GUIs) displayed by the XR device in the virtual environment proximate to the wearable device. The XR device can be implemented as a head-mounted display (HMD) system, an example of which is described below.

[0015] The disclosed embodiments of the technology may include or be implemented in combination with artificial, extended, or extra reality (collectively "XR"), which is a type of reality that is adjusted in some manner prior to presentation to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and/or derivative thereof. Extended reality content may include fully generated content or generated content combined with captured content (e.g., real-world photos). Additionally, in some embodiments, extended reality may be associated with, for example, applications, products, accessories, services, or any combination thereof that are used to perform activities in the artificial reality. Extended reality systems that provide extended reality content can be implemented on a variety of platforms, including head mounted displays (HMDs) connected to a host computer system, standalone HMDs, mobile devices or computing systems, "cave" environments or other projection systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0016] "Virtual reality" or "VR", as used herein, refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user sees images of the real world after those images pass through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display them on the tablet's screen opposite the camera. The tablet can process and adjust or "augment" those images as they pass through the system, such as by adding virtual objects. "Mixed reality" or "MR" refers to a system in which some of the light that enters the user's eyes is generated by the computing system and some is light reflected from objects in the real world. For example, an MR headset can be shaped as glasses with a pass-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset to allow the MR headset to present virtual objects mixed with the real objects the user can see. "Artificial reality", "extra reality", or "XR", as used herein, refers to either VR, AR, MR, or any combination or hybrid thereof.

[0017] Some embodiments are discussed in more detail below with reference to the figures. FIG. 1 is a block diagram showing an overview of devices in which some embodiments of the disclosed technology may function. The devices may comprise hardware components of a computing system 100 that may create, manage, and provide interaction modes for an XR environment. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate over wired or wireless channels to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone headset that may provide a computer-generated or augmented experience to a user without the need for external processing or sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a headset and a core processing component (such as a console, mobile device, or server system), where some processing operations are performed on the headset and other processing operations are offloaded to the core processing component. Exemplary headsets are described below in connection with Figures 2A and 2B. In some implementations, location and environmental data may be gathered solely by sensors built into the headset device, while in other implementations, one or more of the non-headset computing devices may include sensor components that may track environmental or location data.

[0018] Computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a holographic processing unit (HPU), etc.). Processor 110 may be a single processing unit, multiple processing units in one device, or distributed across multiple devices (e.g., distributed across two or more of computing devices 101-103).

[0019] Computing system 100 may include one or more input devices 120 that provide input to processor 110 and inform processor 110 of actions. The actions may be mediated by a hardware controller that interprets signals received from the input devices and communicates the information to processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touch pad, a wearable input device (e.g., a haptic glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, e.g., an infrared sensor), a microphone, or other user input device.

[0020] The processor 110 can be coupled to other hardware devices using internal or external buses, such as, for example, a PCI bus, a SCSI bus, or a wireless connection. The processor 110 can communicate with hardware controllers for devices, such as for a display 130. The display 130 can be used to display text and graphics. In some implementations, the display 130 includes an input device as part of the display, such as when the input device is a touch screen or includes a system that monitors eye or head direction. In some implementations, the display is separate from the input device. Examples of display devices include LCD display screens, LED display screens, projected, holographic, or augmented reality displays (such as head-up display devices or head-mounted devices), etc. Other I/O devices 140 can also be coupled to the processor, such as network chips or cards, video chips or cards, audio chips or cards, USB, Firewire or other external devices, cameras, printers, speakers, CD-ROM drives, DVD drives, disk drives, etc.

[0021] Computing system 100 may include a communication device capable of wirelessly or wired-based communication with other local computing devices or network nodes. The communication device may communicate with another device or server over a network using, for example, TCP/IP protocols. Computing system 100 may utilize the communication device to distribute operations across multiple network devices.

[0022] The processor 110 can access memory 150, which can be included in one of the computing devices of the computing system 100 or distributed across multiple computing devices of the computing system 100 or other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage and can include both read-only and writeable memory. For example, memory can include one or more of random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writeable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and the like. Memory is not a propagating signal separate from the underlying hardware, and therefore memory is non-transitory. Memory 150 can include program memory 160, which stores programs and software, such as an operating system 162, an XR operating system 164, and other application programs 166. Memory 150 may also include data memory 170, which may contain information to be provided to program memory 160 or any element of computing system 100.

[0023] Some implementations may be operational with many other computing system environments or configurations. Examples of computing systems, environments, and/or configurations that may be suitable for use with the present technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular telephones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

2A is a wiring diagram of a head mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a band 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computational units 230. The position sensors 220, the IMU 215, and the computational units 230 may be internal to the HMD 200 and may be invisible to the user. In various implementations, the IMU 215, the position sensors 220, and the locator 225 may track the movement and location of the HMD 200 in the real world and in a virtual environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locator 225 can emit infrared light beams that generate light points on real objects around the HMD 200. As another example, the IMU 215 can include, for example, one or more accelerometers, gyroscopes, magnetometers, other non-camera-based position, force, or orientation sensors, or combinations thereof. One or more cameras (not shown) integrated with the HMD 200 can detect the light points. The computation unit 230 in the HMD 200 can use the detected light points to estimate the position and movement of the HMD 200 and to identify the shape and position of real objects surrounding the HMD 200.

[0025] Electronic display 245 may be integrated with front rigid body 205 and may provide image light to the user as directed by computation unit 230. In various embodiments, electronic display 245 may be a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of electronic display 245 include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a display including one or more quantum dot light emitting diode (QOLED) subpixels, a projector unit (e.g., microLED, laser, etc.), some other display, or some combination thereof.

[0026] In some implementations, the HMD 200 can be coupled to a core processing component, such as a personal computer (PC) (not shown) and/or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), which the PC can use in combination with output from the IMU 215 and position sensor 220 to determine the location and movement of the HMD 200.

2B is a wiring diagram of a mixed reality HMD system 250 including a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as shown by link 256. In other embodiments, the mixed reality system 250 includes only a headset without an external computing device or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a pass-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., laser, LED, etc.), a camera, an eye tracking sensor, MEMS components, networking components, etc.

[0028] The projector can be coupled, for example, via optical elements to a pass-through display 258 to display media to a user. The optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, diffraction gratings, etc., to direct light from the projector to the user's eyes. Image data can be transmitted from the core processing component 254 to the HMD 252 via link 256. A controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted via optical elements as output light to the user's eyes. The output light can be mixed with light passing through the display 258, allowing the output light to present virtual objects that appear as if they exist in the real world.

[0029] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., that enable HMD system 250 to, for example, track itself in 3DoF or 6DoF, track parts of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear stationary as HMD 252 moves, and make virtual objects responsive to gestures and other real-world objects.

[0030] In various embodiments, the HMD 200 or 250 may also include additional subsystems, such as an eye tracking unit, a head tracking unit, an audio system, various network components, etc., to monitor for indications of the user's interaction and intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in the HMD 200 or 250 or from an external camera may monitor the position and pose of the user's hands to identify gestures and other hand and body movements.

User interface control via trackpad with smartwatch
[0031] In some embodiments, a user can use a wearable device, such as a smartwatch, to control a user interface (UI) displayed in a virtual environment generated by an XR device in a similar manner to using a trackpad on a laptop computer system. In some embodiments, the XR device allows a user to use a smartwatch or other wearable device to control a cursor and/or select graphical user interface (GUI) elements in the user interface of the virtual environment generated by the XR device. The advantage of this is that the user can control the cursor on the smartwatch without seeing their hands, while still providing physical feedback to the user. Thus, the user can get a sense of touch from controlling the cursor even while interacting in the virtual environment. An added benefit is that this technique is very accurate because the trackpad is linked to the smartwatch device display touch sensor.

[0032] In some embodiments, the smartwatch is communicatively connected to the XR device. In some embodiments, the connection is wireless, such as, for example, through a Bluetooth connection as a result of pairing. In some other embodiments, other short-range wireless communications, such as, for example, ZigBee, UWB, infrared, etc., are used between the smartwatch (wearable device) and the XR device. The link between the smartwatch and the XR device can include a link between a display touch sensor on the display surface of the smartwatch.

[0033] During use, in response to a user manipulating the display surface of the smartwatch (e.g., moving one or more fingers, tapping, etc.), touch data obtained from the touch sensor of the smartwatch device is transmitted to the XR device and translated into one or more commands, which are executed by the XR device. Thus, the touch capabilities of the smartwatch device can be used to manipulate the user interface and objects displayed in the virtual environment by the XR device. In some embodiments, manipulations include moving and/or controlling a cursor or other GUI elements, scrolling a window (e.g., a browser window), selecting one or more objects in the user interface of the XR device, rotating an object, scaling an object, moving an object, etc. Thus, by detecting the user's movements on the touchpad interface of the smartwatch (or phone) device, the user can control the XR device to manipulate the user interface.

3A and 3B illustrate using the display surface of a smartwatch as a trackpad to control a cursor with the trackpad. Referring to FIG. 3A, a smartwatch 310 having a display surface 311 is shown with a user interface window 301 in a virtual environment 300. A cursor 320 is shown on the window 301. By moving one or more fingers on or across the display surface of the trackpad 311 of the smartwatch 310, a user can control the cursor 320 with the display surface 311. For example, a user can move the cursor over one or more objects displayed in the user interface window 301 and select those objects by tapping the display surface 311 of the smartwatch 310 with one or more fingers, such as finger 350. A user can slide finger 350 across the display surface 311 of the smartwatch 310 with a finger, such as finger 350, to move the cursor 320 within the window 301. An example of cursor 320 movement is represented by arrow 321. Note that cursor 320 can be moved in many, if not all, directions, not just the direction represented by arrow 321. In FIG. 3B, a user can use multiple (e.g., two or more) fingers, such as with index and middle fingers 351, to scroll a window, such as window 461, in virtual environment 460 by moving the fingers along display surface 311 of smartwatch 310, whereby display surface 311 of smartwatch 310 acts as a trackpad.

[0035] Note that when a user interacts with the display surface 311, for example with one or more fingers, the touch sensors of the display surface 311 capture touch data from the finger movements and transmit the touch to the XR device. In some embodiments, the touch data indicates one or more of a position, orientation, velocity, direction, distance, etc. In response, the XR device converts or otherwise translates the data into commands (e.g., cursor control commands, window control commands, object/GUI element selection commands) that the XR device executes with respect to the virtual environment.

[0036] Figure 4 is a flow diagram of some embodiments of a process for controlling a user interface in an extended reality (XR) device using a trackpad with a smartwatch. In some embodiments, the process is performed at least in part by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0037] Referring to FIG. 4, the process begins by processing logic detecting a wearable device (e.g., a wearable device) worn by a user (processing block 401). In some embodiments, the detection is performed by an XR device. In some embodiments, the device worn by the user includes a wristwatch. In some embodiments, a wristwatch, including a smartwatch. In some other embodiments, the device may be another wearable device.

[0038] After detecting the presence of a smartwatch, processing logic communicatively connects the XR device to the smartwatch device (processing block 402). In some embodiments, communication between the XR device and the smartwatch is performed wirelessly. In some embodiments, this connection includes Bluetooth or short-range wireless communication.

[0039] Once the XR device and the smartwatch are in communication with each other, processing logic manipulates the user interface that is generated and displayed on the display of the XR device using the smartwatch (processing block 403). In some embodiments, manipulating the user interface includes moving a cursor in the user interface based on a detected motion made by the user on the smartwatch. The detected motion may include the user moving one or more fingers across the display surface of the smartwatch. The detected motion is captured by a touch sensor on the smartwatch, and touch data is transmitted by the smartwatch to the XR device. The XR device then translates the touch data into commands or actions that the XR device executes on objects or other GUI elements displayed in the user interface of the XR device.

[0040] In some other embodiments, manipulating the user interface includes selecting an object displayed on a user interface in the virtual environment generated by the XR device based on a detected action made by the user on the smartwatch. In some embodiments, the detected action may include the user tapping the display of the smartwatch with, for example, their finger. The action is captured by the smartwatch in the form of touch data, which is transmitted by the smartwatch to the XR device, which converts the touch data into a command or action performed by the XR device on the object displayed in the user interface of the XR device.

[0041] In some other embodiments, manipulating the user interface includes scrolling a window displayed on the user interface based on a detected motion made by the user on the smartwatch. In some embodiments, the detected motion includes a movement of one or more fingers across the display of the smartwatch. Again, this motion is captured by a touch sensor on the display surface of the smartwatch, and the resulting touch data is transmitted to the XR device, which converts the touch data into a command or action performed by the XR device on an object displayed in the user interface of the XR device. In this case, the command/action is scrolling a window in the user interface in the virtual environment generated by the XR device.

User interface control using XR shortcut function and smart watch
[0042] In current XR systems, it is possible to perform a particular action by pinching for a particular number of seconds, pressing a particular button, or performing other such functions. However, current XR devices do not use virtual shortcut buttons near the smartwatch (or other wearable device) to indicate a shortcut the user wanted to create to the user interface being displayed in the virtual environment by the XR device.

[0043] In some embodiments, the XR device displays a virtual shortcut menu with a set of one or more shortcut virtual buttons in a user interface generated by the XR device that appears around the smartwatch. In some embodiments, the location of the shortcut menu is identified or calculated according to one or more image tracking markers on the smartwatch. In some embodiments, the one or more image tracking markers appear on the display surface of the smartwatch. Using this technique, a user can use the virtual shortcut button menu by simply looking at the smartwatch device and performing a gesture, such as, for example, a button press gesture with one or more fingers on any one of the virtual buttons. In some embodiments, the virtual buttons are scalable and follow the position of the watch device due to the one or more image tracking markers.

[0044] The use of virtual buttons facilitates a user to select an action and/or event to occur in a user interface in a virtual environment generated and displayed by an XR device. In some embodiments, the XR device uses hand tracking to recognize hand gestures to perform the selection. That is, a user performs a gesture with his/her hand, such as tapping one of the virtual buttons with a finger, and an image camera of the XR device captures data associated with the user's movement. The XR device identifies the gesture based on the data corresponding to the movement, which may include one or more of position, orientation, speed, direction, distance, etc. Based on the identified gesture, the XR device identifies which of the virtual buttons the user has selected and wants to invoke and perform the action and/or command associated with that virtual button. Thus, in this manner, the user can view a virtual shortcut interface and control a user interface in a virtual environment generated and displayed by an XR device through the use of his/her smartwatch.

5A and 5B show some embodiments of a virtual shortcut menu with virtual buttons around the smartwatch for controlling a user interface in a virtual environment generated by an XR device. Referring to FIG. 5A, a virtual shortcut menu 501 is displayed by the XR device near the smartwatch 502 as part of a user interface in the virtual environment. In some embodiments, the XR device uses an image tracking marker 503 on the smartwatch 502 to sense the position of the smartwatch 502. Based on the position of the smartwatch 502, the XR device can place the virtual shortcut menu 501 near the smartwatch 502. In some embodiments, the XR device places the virtual shortcut menu 501 around a portion of the smartwatch 502. Note that the virtual shortcut menu 501 and/or its virtual buttons can be placed on one, multiple, or all sides of the smartwatch 502.

[0046] In some embodiments, the virtual shortcut interface 501 includes a number of virtual buttons that can be selected by the user to allow the user to jump to a particular function, such as, for example, performing one or more functions or actions. The user can press the button that the user wants to use to invoke the function associated with each of the virtual buttons.

5A and 5B, the virtual shortcut interface 501 includes five virtual buttons, although the techniques disclosed herein can have more or less than five virtual buttons. In some embodiments, the virtual shortcut interface 501 includes a home button 510, a button 511 for opening a browser (or other window), a button 512 for checking open applications, a back button 513, and a forward button 514. The home button 510, when selected, allows the user to go to a home screen in the user interface of the virtual environment, which can be a start screen or a main menu (e.g., a redisplay of the virtual shortcut interface 501). The back button 513, when selected, allows the end user to go to a previously viewed page (e.g., a web page, etc.) in the user interface. The forward button 514, when selected, allows the end user to go to a previously viewed page (e.g., a web page, etc.) in the user interface that was originally viewed after the current page. In some embodiments, one of the virtual buttons of the virtual shortcut interface 501, when selected, causes the virtual shortcut interface 501 to display another set of virtual buttons. The new set of virtual buttons can include one or more buttons in the previously displayed set of buttons in the virtual shortcut interface 501. In some embodiments, the virtual shortcut interface 501 is customizable and can have any set of buttons that a user (or multiple users) want to include.

[0048] In some embodiments, the virtual buttons in the virtual shortcut interface 501 are configurable by the user. In this manner, the user can use the virtual shortcut interface 501 to interact with a user interface in a virtual environment generated and displayed by the XR device.

[0049][0050] More specifically, through the use of hand tracking and image recognition performed by the XR device, the XR device is able to identify that the user is looking at the smartwatch 502 and then display the virtual shortcut interface 503 in response to that identification. That is, in some embodiments, the actual smartwatch screen is changed as shown in 502 (while the smartwatch shows a normal screen when the user is not wearing the XR device). At this point, the user can make hand gestures to select one or more of the virtual buttons.

[0051] FIG. 5B illustrates some embodiments of a virtual shortcut menu interface in which one of the virtual shortcut buttons 510-514 is selected to cause the function associated with that virtual button to be invoked by the XR device. With reference to FIG. 5B, a user moves a finger 570 to perform a pressing gesture on the virtual button 511. In some embodiments, any pressing gesture made on one of the virtual buttons 510-514 is recognized by the XR device using hand tracking with one or more cameras in the XR device. These functions can include going to the home button, going forward or back through previously viewed display windows.

[0052] In some embodiments, the display surface of the smartwatch can be used to select one or more of the selectable functions through the use of a virtual shortcut interface. In such cases, the display surface of the smartwatch can be used as a trackpad as described above, and functions can be invoked by touching the display surface. In some embodiments, pre-specified sections of the display surface can be associated with one or more invokeable functions, such that when a user taps or otherwise touches each of the pre-specified sections, the associated function(s) can then be invoked. FIG. 5C illustrates some embodiments of a display surface of a smartwatch. With reference to FIG. 5C, a display surface element 590 can be associated with a home button 510, and a user can tap on the display surface element 590 to invoke the function of the home button 510. Similarly, a display surface element 591 can be associated with a forward button 514, and a user can tap on the display surface element 591 to invoke the function of the home button 514. These are merely examples, and others can be implemented.

[0053] FIG. 6 is a flow diagram of some embodiments of a process for controlling a user interface on an XR device using an XR shortcut interface in conjunction with a smartwatch or other wearable device. In some embodiments, the process is performed at least in part by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0054] Referring to FIG. 6, the process begins with processing logic in which the XR device processing logic detects a smartwatch (or other wearable device) (processing block 601). The processing logic then identifies marker(s) on the smartwatch using the camera (processing block 602), and the processing logic detects a location of the wearable device using the marker(s) (processing block 603). In some embodiments, detecting the location of the wearable device using the markers includes detecting the location of the markers on a display of the smartwatch.

[0055] In response to detecting the smartwatch, processing logic displays a virtual interface in a user interface displayed by the XR device on the display of the XR device (processing block 604). In some embodiments, the virtual interface is displayed near the wearable device and includes graphical user interface (GUI) elements selectable by the user. In some embodiments, the virtual interface is displayed near the smartwatch by displaying the virtual interface around the smartwatch (e.g., around the display of the wearable device). In some embodiments, the GUI elements include buttons, although other types of GUI elements may be used in the virtual interface.

[0056] Thereafter, with the virtual interface displayed, processing logic records a user's hand gesture made over one of the GUI elements (processing block 605) and recognizes the hand gesture as a selection of the one of the GUI elements (processing block 606). In some embodiments, image data associated with the user's hand movement (e.g., finger movement) is captured by a camera of the XR device and analyzed by the XR device to determine that a gesture was performed to indicate selection of one of the GUI elements. In response to this recognition, processing logic invokes a function associated with the one of the GUI elements (processing block 607). That is, the XR device executes a command to perform an action and/or function associated with the selected virtual button.

[0057] It should be noted that the virtual trackpad described above in Figures 3A-4 can be used in conjunction with the virtual shortcut menu interface described above in Figures 5A-6. For example, the virtual trackpad of the smartwatch can be used to cause a virtual shortcut menu to appear. From that point, the user can invoke functions using virtual buttons around the smartwatch and/or using the display surface of the smartwatch. Other uses of combining these two technologies will be recognized by those skilled in the art.

[0058] There are several exemplary embodiments described herein.

[0059] Example 1 is a method that includes detecting a wearable device by an extended reality (XR) device, communicatively connecting the XR device to the wearable device, and using the wearable device to manipulate a user interface in a virtual environment displayed on a display of the XR device.

[0060] Example 2 is the method of Example 1, where the step of manipulating the user interface includes moving a cursor relative to the user interface based on a detected motion made by the user on the wearable device that is transmitted to the XR device, and can optionally include the detected motion being moving one or more fingers of the user across a display surface of the wearable device.

[0061] Example 3 is the method of Example 1, in which the step of operating the user interface includes selecting an object displayed on the user interface based on a detected motion made by the user on the wearable device that is transmitted to the XR device, and can optionally include the detected motion being a tap by the user on a display surface of the wearable device.

[0062] Example 4 is the method of Example 1, in which the step of operating the user interface includes scrolling a window displayed on the user interface based on a detected motion made by the user on the wearable device that is transmitted to the XR device, and can optionally include the detected motion being a movement of one or more fingers by the user across a portion of the wearable device.

[0063] Example 5 is the method of example 4, which can optionally include that the at least one finger includes two fingers and the portion of the wearable device includes a display of the wearable device.

[0064] Example 6 is the method of Example 1 that can optionally include displaying a virtual interface in the user interface near the wearable device to include user-selectable graphical user interface (GUI) elements.

[0065] Example 7 is the method of example 6 that can optionally include that the user-selectable GUI element is a virtual button.

[0066] Example 8 is the method of Example 6 that may optionally include capturing one or more finger movements made by a user in relation to one of the GUI elements, recognizing the movements as a selection of the one GUI element, and in response to recognizing the movements, invoking a function associated with the one GUI element.

[0067] Example 9 is the method of example 1 that can optionally include customizing the virtual GUI elements to associate separate functions with each of the virtual GUI elements.

[0068] Example 10 is the method of Example 1, which may optionally include using a camera to identify a marker on the wearable device and using the marker to sense a position of the wearable device, and wherein displaying a virtual interface in the user interface near the wearable device includes displaying the virtual interface based on the position of the wearable device.

[0069] Example 11 is the method of Example 10, which can optionally include the step of detecting a position of the wearable device using the markers including the step of detecting a position of a display of the wearable device using the markers.

[0070] Example 12 is the method of example 1, which can optionally include that the virtual interface is displayed around the wearable device, and the wearable device includes a smart watch device.

[0071] Example 13 is a computing system for controlling a user interface, the computing system including one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process including detecting a wearable device with an extended reality (XR) device, communicatively connecting the XR device to the wearable device, and using the wearable device to manipulate a user interface in a virtual environment displayed on a display of the XR device.

[0072] Example 14 is the computing system of Example 13, in which operating the user interface includes moving a cursor relative to the user interface based on a detected motion made by the user on the wearable device that is transmitted to the XR device, and can optionally include the detected motion being moving one or more fingers of the user across a display surface of the wearable device.

[0073] Example 15 is the computing system of Example 13, in which operating the user interface includes selecting an object displayed on the user interface based on a detected action made by the user on the wearable device that is transmitted to the XR device, and the detected action can optionally include a tap by the user on a display surface of the wearable device.

[0074] Example 16 is the computing system of Example 13, in which operating the user interface includes scrolling a window displayed on the user interface based on a detected motion made by the user on the wearable device that is transmitted to the XR device, and can optionally include the detected motion being a movement of one or more fingers by the user across a portion of the wearable device.

[0075] Example 17 is the computing system of Example 13, which may optionally include that the process further includes displaying a virtual interface in the user interface near the wearable device to include graphical user interface (GUI) elements selectable by the user.

[0076] Example 18 is the computing system of Example 17 that can optionally include that the user-selectable GUI elements are virtual buttons.

[0077] Example 19 is the computing system of Example 17, which may optionally include further including capturing one or more finger movements made by a user in relation to one of the GUI elements, recognizing the movements as a selection of the one GUI element, and in response to recognizing the movements, invoking a function associated with the one GUI element.

[0078] Example 20 is one or more non-transitory computer-readable storage media having instructions stored thereon that, when executed by a system having at least a processor and memory therein, cause the system to perform operations including detecting a wearable device with an extended reality (XR) device, communicatively connecting the XR device to the wearable device, and using the wearable device to manipulate a user interface in a virtual environment displayed on a display of the XR device.

[0079] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, words, numbers, or the like.

[0080] However, please keep in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, as will be apparent from the following discussion, throughout this description, discussions utilizing terms such as "processing," "calculating," "computing," "determining," "displaying," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data that are also represented as physical quantities in the memory or registers of the computer system, or other such information storage, transmission, or display device.

[0081] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes or may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0082] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description that follows. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present disclosure as described herein.

[0083] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media includes read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared, digital, etc.), and the like.

[0084] Although many variations and modifications of the present disclosure will no doubt become apparent to those of ordinary skill in the art after reading the foregoing description, it should be understood that any particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. Thus, references to details of various embodiments are not intended to limit the scope of the claims, which inherently recite only those features deemed essential to the present disclosure.

Claims (12)

detecting the wearable device by an extended reality (XR) device;
communicatively connecting the XR device to the wearable device;
displaying a virtual environment with a user interface on a display of the XR device;
Displaying a virtual shortcut interface within the user interface to appear next to and near the wearable device but not on top of the wearable device, comprising positioning the virtual shortcut interface based on a position of the wearable device determined using a tracking marker on a display of the wearable device;
the virtual shortcut interface includes a user-selectable virtual button;
The user selectable virtual button may include:
a home button that allows the user to navigate to a home screen or main menu;
A button to open a browser
a back button that allows the user to navigate to a previously displayed page within the user interface;
a forward button that allows the user to navigate to the first page displayed after the currently displayed page that was previously displayed within the user interface; and
one or more buttons from a group including a button for checking open applications, a button for displaying another set of virtual buttons in the virtual shortcut interface that includes one or more buttons in a previously displayed set of buttons, and the user customizable buttons;
displaying the virtual shortcut interface,
capturing one or more finger movements made by the user relative to one of the virtual buttons;
recognizing said finger movement as a selection of said one virtual button;
activating a function associated with the one virtual button in response to recognizing the finger movement, the function including opening an application window with the browser in the user interface;
Manipulating the user interface in the virtual environment using the wearable device, the manipulating the user interface comprising:
moving a cursor displayed in the application window or scrolling the application window displayed on the user interface based on detected movements made by a user on the wearable device and transmitted to the XR device;
the detected motion comprises a motion of one or more fingers of the user across a display surface of the wearable device; and manipulating the user interface to provide physical feedback to the user without seeing the user's one or more fingers;
A method comprising: The step of operating the user interface includes:
selecting an object displayed on the user interface based on a detected action made by a user on the wearable device, the detected action being transmitted to the XR device;
Including,
The method of claim 1 , wherein the detected motion is a tap by the user on a display surface of the wearable device. The method of claim 1 , wherein the portion of the wearable device includes a display of the wearable device. The method of claim 1 , wherein the user-selectable GUI elements are virtual buttons. The method of claim 1, wherein the virtual GUI elements are customizable to associate separate functions with each of the virtual GUI elements. The method comprises:
using a camera to identify a marker on the wearable device;
detecting a position of the wearable device using the marker;
Further comprising:
The method of claim 1 , wherein displaying a virtual interface near a display position of the wearable device in the virtual environment of the user interface includes displaying the virtual interface based on the display position of the wearable device. Detecting a position of the wearable device using the marker,
using the marker to sense the position of a display of the wearable device;
The method of claim 6 , comprising: The method of claim 1 , wherein a virtual interface is displayed around a display position of the wearable device in the virtual environment of the user interface , the wearable device comprising a smart watch device. 1. A computing system for controlling a user interface, comprising:
one or more processors;
one or more memories storing instructions;
Equipped with
The instructions, when executed by the one or more processors,
detecting the wearable device by an extended reality (XR) device;
communicatively connecting the XR device to the wearable device;
displaying a virtual environment with a user interface on a display of the XR device;
Displaying a virtual shortcut interface within the user interface to appear next to and near the wearable device but not on top of the wearable device, comprising positioning the virtual shortcut interface based on a position of the wearable device determined using a tracking marker on a display of the wearable device;
the virtual shortcut interface includes a user-selectable virtual button;
The user selectable virtual button may include:
a home button that allows the user to navigate to a home screen or main menu;
A button to open a browser
a back button that allows the user to navigate to a previously displayed page within the user interface;
a forward button that allows the user to navigate to the first page displayed after the currently displayed page that was previously displayed within the user interface; and
one or more buttons from a group including a button for checking open applications, a button for displaying another set of virtual buttons in the virtual shortcut interface that includes one or more buttons in a previously displayed set of buttons, and the user customizable buttons;
displaying the virtual shortcut interface,
capturing one or more finger movements made by the user relative to one of the virtual buttons;
recognizing said finger movement as a selection of said one virtual button;
activating a function associated with the one virtual button in response to recognizing the finger movement, the function including opening an application window with the browser in the user interface;
Manipulating the user interface in the virtual environment using the wearable device, the manipulating the user interface comprising:
moving a cursor displayed in the application window or scrolling the application window displayed on the user interface based on a detected movement made by a user on the wearable device and transmitted to the XR device;
the detected motion comprises a motion of one or more fingers of the user across a display surface of the wearable device; and manipulating the user interface to provide physical feedback to the user without seeing the user's one or more fingers;
A computing system that causes the computing system to execute a process including the steps of: The step of operating the user interface includes:
selecting an object displayed on the user interface based on a detected action made by a user on the wearable device, the detected action being transmitted to the XR device;
Including,
The computing system of claim 9 , wherein the detected action is a tap by the user on a display surface of the wearable device. The computing system of claim 9 , wherein the user-selectable GUI element is a virtual button. One or more computer-readable storage media having instructions stored thereon,
The instructions, when executed by a system having at least a processor and a memory therein,
detecting the wearable device by an extended reality (XR) device;
communicatively connecting the XR device to the wearable device;
displaying a virtual environment with a user interface on a display of the XR device;
Displaying a virtual shortcut interface within the user interface to appear next to and near the wearable device but not on top of the wearable device, comprising positioning the virtual shortcut interface based on a position of the wearable device determined using a tracking marker on a display of the wearable device;
the virtual shortcut interface includes a user-selectable virtual button;
The user selectable virtual button may include:
a home button that allows the user to navigate to a home screen or main menu;
A button to open a browser
a back button that allows the user to navigate to a previously displayed page within the user interface;
a forward button that allows the user to navigate to the first page displayed after the currently displayed page that was previously displayed within the user interface; and
one or more buttons from a group including a button for checking open applications, a button for displaying another set of virtual buttons in the virtual shortcut interface that includes one or more buttons in a previously displayed set of buttons, and the user customizable buttons;
displaying the virtual shortcut interface,
capturing one or more finger movements made by the user relative to one of the virtual buttons;
recognizing said finger movement as a selection of said one virtual button;
activating a function associated with the one virtual button in response to recognizing the finger movement, the function including opening an application window with the browser in the user interface;
Manipulating the user interface in the virtual environment using the wearable device, the manipulating the user interface comprising:
moving a cursor displayed in the application window or scrolling the application window displayed on the user interface based on detected movements made by a user on the wearable device and transmitted to the XR device;
the detected motion comprises a motion of one or more fingers of the user across a display surface of the wearable device; and manipulating the user interface to provide physical feedback to the user without seeing the user's one or more fingers;
causing the system to perform operations including
One or more computer-readable storage media.

Images