JP7741026B2 - Digital supplemental association and retrieval for visual search - Google Patents

Description

REFERENCE TO RELATED APPLICATIONS This application is a division of DIGITAL SUPPLEMENT ASSOCIATION AND
This application is a continuation of and claims priority to U.S. Non-provisional Patent Application No. 16/014,520, entitled "Visual Search and Retrieval," the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND Mobile computing devices, such as smartphones, often include cameras. These cameras can be used to capture images of entities in the environment surrounding the computing device. Various types of content or experiences related to these entities can be made available to users via the mobile computing device.

This disclosure describes systems and methods for digital supplement association and retrieval for visual search. For example, the systems and techniques described herein can be used to provide digital supplements, such as augmented reality (AR) content or experiences, that are responsive to visual searches. The visual search can be based, for example, on images or entities identified within the images. The digital supplement can include, for example, providing information or functionality associated with the images.

One aspect is a computer-implemented method that includes receiving data specifying a digital supplement, the data identifying the digital supplement and a supplemental anchor for associating the digital supplement with image content. The method also includes generating a data structure instance that specifies the digital supplement and the supplemental anchor. After generating the data structure instance, the method further includes enabling triggering of the digital supplement by the image based on storing at least the data structure instance in a database that includes multiple other data structure instances. Each of the other data structure instances specifies the digital supplement and one or more supplemental anchors.

Another aspect is a computing device including at least one processor and a memory storing instructions. When executed by the at least one processor, the instructions cause the computing device to receive data specifying a digital supplement, the data receiving the digital supplement, a supplemental anchor for associating the digital supplement with image content, and contextual information. The instructions also cause the computing device to generate a data structure instance specifying the digital supplement, the supplemental anchor, and the contextual information. The instructions further cause the computing device, after generating the data structure instance, to enable triggering of the digital supplement by an image based at least on storing the data structure instance in a database including multiple other data structure instances, each of which specifies the digital supplement and one or more supplemental anchors.

Yet another aspect is a computer-implemented method that includes receiving an image content query from a computing device and identifying supplemental anchors based on the image content query, the method also including generating an ordered list of digital supplements based on the identified supplemental anchors and transmitting the ordered list to a client computing device.

The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

FIG. 1 is a block diagram illustrating a system according to an exemplary embodiment. 2 is a third-person perspective of an exemplary physical space in which an embodiment of the client computing device of FIG. 1 is accessing a digital capture. 10A-10C are diagrams of exemplary methods for enabling triggers for digital capture according to embodiments described herein. 10A-10C are diagrams of exemplary methods for enabling triggers for digital capture according to embodiments described herein. 1 is a diagram of an exemplary method for retrieving and presenting a digital supplement according to embodiments described herein. 1 is a diagram of an exemplary method for identifying and presenting a digital supplement based on an image according to embodiments described herein. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. 3A-3C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 for conducting various image content searches within a store. 3A-3C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 for conducting various image content searches within a store. 3A-3C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 for conducting various image content searches within a store. 2A-2C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 during various image content searches. 2A-2C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 during various image content searches. 2A-2C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 during various image content searches. 1 is a schematic diagram of an example of a computing device and a mobile computing device that can be used to implement the techniques described herein.

DETAILED DESCRIPTION This disclosure describes technical improvements that simplify the identification and presentation of digital supplements based on image content. Certain embodiments of the techniques described herein generate an index of digital supplements relevant to a particular type of image content and provide those digital supplements in response to image content queries received from client computing devices. This index allows users to access relevant digital supplements provided by network-accessible resources (e.g., web pages) located around the world. This can provide a functional data structure that enables more efficient retrieval of information.

For example, a client computing device such as a smartphone may obtain an image of a supplemental anchor, such as an entity. The client computing device may then send an image content query based on the image to a server computing device to search for a digital supplement associated with the identified supplemental anchor. In some embodiments, the supplemental anchor is based on the physical environment surrounding the client computing device, and the digital supplement is virtual content that may supplement a user's experience in the physical environment.

Image content queries may include images or data determined from an image (e.g., indicators of identified supplemental anchors). An example of data determined from an image is, for example, text extracted from an image using optical character recognition. Other examples of data extracted from an image include barcodes, QR codes, etc., in the image, identifiers or descriptions of entities, products, or types of entities identified in the image.

Entities, products, or entity types may be identified in an image using, for example, a neural network system such as a convolutional neural network system. Identifiers or descriptions of entities, products, or entity types may include metadata or references to records in a database associated with the entities, products, or entity types. Non-limiting examples of entities include buildings, pieces of technology, products, books, posters, photographs, catalogs, signs, documents (e.g., business cards, receipts, coupons, catalogs), people, and body parts.

Various types of digital supplements may be available that can be associated with supplemental anchors. Digital supplements may be provided by network-accessible resources, such as web pages available on the Internet. A method is needed to locate and provide these digital supplements in response to image content queries. One embodiment generates and maintains an index of digital supplements associated with entities for use in responding to image content queries. The index may be populated, for example, by crawling network-accessible resources to determine whether they contain or provide digital supplements and determining the supplemental anchors associated with those digital supplements.

For example, a network-accessible resource may include metadata that identifies a supplemental anchor (e.g., text, code, entity, or type of entity) with which the digital supplement is associated. The metadata may be included by the network-accessible resource in response to a Hypertext Transfer Protocol (HTTP) request. The metadata may be provided in a variety of formats, such as extensible markup language (XML), JavaScript Object Notation (JSON), or other formats.

Metadata for a digital supplement may include one or more of the following: a type indicator, an anchor indicator, a name, a description, a content snippet (i.e., an excerpt or preview of a portion of the content), an associated image, a link, such as a URL, to the digital supplement, and an identifier for an application associated with the digital supplement. The metadata may also include information about the publisher of the digital supplement. For example, the metadata may include one or more of a publisher name, a publisher description, and an image or icon associated with the publisher. In some embodiments, the metadata includes contextual information associated with providing the digital supplement. For example, the metadata may also include conditions associated with providing or accessing the digital supplement (e.g., geographic conditions, required applications).

The identified digital supplement may be added to an index stored in memory. In at least some embodiments, the associated supplement anchor for the digital supplement is used as a key to the index. The digital supplement may also be associated with various scores. For example, a digital supplement may be associated with a prestige score based on how many other links referencing the digital supplement or a network-accessible resource associated with the digital supplement are found (e.g., while crawling the network-accessible resource) and the prestige of the network-accessible resource providing those links. As another example, a digital supplement may be associated with one or more relevance scores corresponding to the relevance of the digital supplement (or associated network-accessible resource) to a particular anchor. Relevance scores may also be associated with keywords or subjects. Relevance scores may be determined based on one or more of the content of the digital supplement, the content of the network-accessible resource, the content of sites linking to the network-accessible resource, and the content (e.g., text) of the links to the network-accessible resource.

FIG. 1 is a block diagram illustrating a system 100 according to an exemplary embodiment. The system 100 can associate digital supplements with entities or entity types and can retrieve digital supplements in response to a visual search. A visual search is a search based on image content. For example, a visual search can be performed based on an image content query. An image content query is a query based on images or other image content. For example, an image content query can include an image. In some embodiments, an image content query can include text or data based on an image. For example, the text or data can be generated by recognizing one or more entities in an image. Some image content queries do not include images (e.g., an image content query can include only data or text generated from an image). In some embodiments, the system 100 includes a client computing device 102, a search server 152, and a digital supplement server 172. Also shown is a network 190 over which the client computing device 102, the search server 152, and the digital supplement server 172 can communicate.

The client computing device 102 may include a processor assembly 104, a communications module 106, a sensor system 110, and a memory 120. The sensor system 110 may include various sensors, such as a camera assembly 112, an inertial motion unit (IMU) 114, and a global positioning system (GPS) receiver 116. Embodiments of the sensor system 110 may also include other sensors, including, for example, light sensors, audio sensors, image sensors, distance and/or proximity sensors, contact sensors such as capacitive sensors, timers, and/or other sensors, and/or different combinations of sensors. In some embodiments, the client computing device 102 is a mobile device (e.g., a smartphone).

Camera assembly 112 captures images or footage of the physical space around client computing device 102. Camera assembly 112 may include one or more cameras. Camera assembly 112 may also include an infrared camera. Images captured by camera assembly 112 may be used to capture anchors and to identify them for forming image content queries.

In some embodiments, images acquired by camera assembly 112 may also be used to determine the position and orientation of client computing device 102 within a physical space, such as an interior space, based on a representation of the physical space received from memory 120 or an external computing device. In some embodiments, the representation of the physical space may include visual features of the physical space (e.g., features extracted from an image of the physical space). The representation may also include positioning data associated with those features that may be used by a visual positioning system to determine a position and/or location within the physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structures within the physical space. In some embodiments, the representation does not include a three-dimensional model of the physical space.

The IMU 114 may detect the motion, movement, and/or acceleration of the client computing device. The IMU 114 may include a variety of different types of sensors, such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors. The orientation of the client computing device 102 may be detected and tracked based on data provided by the IMU 114 or the GPS receiver 116.

The GPS receiver 116 may receive signals transmitted by GPS satellites. The signals include the time and location of the satellites. Based on receiving signals from several (e.g., at least four) satellites, the GPS receiver 116 may determine the global location of the client computing device 102.

Memory 120 may include application 122, other applications 140, and device positioning system 142. Other applications 140 include any other applications installed or otherwise available to run on client computing device 102. In some embodiments, application 122 may cause one of other applications 140 to provide a digital supplement. In some embodiments, some digital supplements may be available only if other applications 140 include a particular application associated with or required to provide the digital supplement.

The device positioning system 142 determines the location of the client computing device 102. The device positioning system 142 may use the sensor system 110 to determine the location and orientation of the client computing device 102 globally or within physical space. In one embodiment, the device positioning system 142 determines the location of the client computing device 102 based on, for example, cellular triangulation.

In some embodiments, client computing device 102 may include a visual positioning system that compares images acquired by camera assembly 112 (or features extracted from those images) with known arrangements of features within a representation of physical space to determine a six-degree-of-freedom pose (e.g., position and orientation) of client computing device 102 within physical space.

The application 122 may include a supplemental anchor identification engine 124, a digital supplemental search engine 126, a digital supplemental presentation engine 128, and a user interface engine 130. Some embodiments of the application 122 may include fewer, additional, or other components.

The supplemental anchor identification engine 124 identifies supplemental anchors based on, for example, images captured by the camera assembly 112. In one embodiment, the supplemental anchor identification engine 124 analyzes the images to identify text. The text can then be used to identify anchors. For example, the text can be mapped to nodes in a knowledge graph. For example, the text can be recognized as the name of an entity, such as a person, place, product, building, artwork, movie, or other type of entity. In one embodiment, the text can be recognized as a phrase commonly associated with a particular entity or as a phrase that describes a particular entity. For example, the text can then be recognized as an anchor associated with the particular entity.

In one embodiment, the supplemental anchor identification engine 124 identifies one or more codes within the image, such as a barcode, QR code, or another type of code. The codes can then be mapped to supplemental anchors.

The supplemental anchor identification engine 124 may include a machine learning module capable of recognizing at least some types of entities in an image. For example, the machine learning module may include a neural network system. A neural network is a computational model used for machine learning and is composed of nodes organized into layers with weighted connections. Training a neural network uses multiple training examples, each of which is an input and a desired output, and over a series of iterative rounds determines weight values for the connections between layers that increase the likelihood that the neural network will provide the desired output for a given input. During each training round, the weights are adjusted to address erroneous output values. Once trained, the neural network can be used to predict outputs based on provided inputs.

In one embodiment, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolutional layer is a layer in which the layer's values are calculated based on applying a kernel function to a subset of the values in the previous layer. Training a neural network may include adjusting the weights of the kernel function based on training examples. Typically, the same kernel function is used to calculate each value in a convolutional layer. Therefore, the number of weights that must be learned while training a convolutional layer is far fewer than in a fully connected layer in a neural network (e.g., a layer in which each value in a layer is calculated as an independently adjusted and weighted combination of each value in the previous layer). Because convolutional layers typically have fewer weights, training and using a convolutional layer may require less memory, processor cycles, and time than an equivalent fully connected layer.

After the supplemental anchor identification engine 124 recognizes an entity or entity type in an image, a text description of the entity or entity type may be generated. Additionally, the entity or entity type may be mapped to a supplemental anchor. In some embodiments, the supplemental anchor is associated with one or more digital anchors.

In one embodiment, the supplemental anchor identification engine 124 determines a confidence score for the recognized anchor. A higher confidence score may indicate that content from the image (e.g., image, extracted text, barcode, QR code) is more likely to be associated with the determined anchor than if a lower confidence score were determined.

Although the example of FIG. 1 shows the supplemental anchor identification engine 124 as a component of the application 122 on the client computing device 102, some embodiments include the supplemental anchor identification engine on the search server 152. For example, the client computing device 102 can transmit an image captured by the camera assembly 112 to the search server 152, and the search server 152 can identify supplemental anchors in the image.

In one embodiment, the supplemental anchor identification engine 124 identifies potential supplemental anchors. For example, the supplemental anchor identification engine 124 may identify (recognize) various entities within the image. Identifiers of the recognized entities may then be transmitted to the search server 152, which may determine whether any of the entities are associated with any supplemental anchors. In one embodiment, the search server 152 may use the identified entities as contextual information, even if the identified entities are not supplemental anchors.

The digital supplement search engine 126 searches for digital supplements. For example, the digital supplement search engine 126 may search for digital supplements associated with supplement anchors identified by the supplement anchor identification engine 124. In some embodiments, the digital supplement search engine 126 searches for digital supplements from the search server 152 or the digital supplement server 172.

For example, after a supplemental anchor is identified, the digital supplemental search engine 126 may search for one or more digital supplements associated with the identified supplemental anchor. The digital supplemental search engine 126 may generate an image content query that includes the image (or an identifier of the supplemental anchor or entity within the image) and send the image content query to the search server 152. The image content query may also include contextual information, such as the location of the client computing device 102. In an embodiment, data associated with the digital supplement, such as a name, image, or description, is retrieved and presented to the user (e.g., by the user interface engine 130). If multiple digital supplements are presented, the user may select one of the digital supplements via a user interface generated by the user interface engine 130.

The digital supplement presentation engine 128 presents or causes the digital supplement to be presented on the client computing device 102. In some embodiments, the digital supplement presentation engine 128 causes the client computing device to launch one of the other applications 140. In some embodiments, the digital supplement presentation engine 128 causes information or content to be displayed. For example, the digital supplement presentation engine 128 may cause the user interface engine 130 to generate a user interface including information or content from the digital supplement that is displayed by the client computing device 102. In some embodiments, the digital supplement presentation engine 128 is triggered by the digital supplement search engine 126 searching for the digital supplement. The digital supplement presentation engine 128 may then trigger the display device 108 to display content associated with the digital supplement. In some embodiments, the digital supplement presentation engine 128 causes the digital supplement to be displayed at a time different from when the digital supplement search engine 126 searches for the digital supplement. For example, the digital supplement may be searched for in response to an image content query at a first time, and the digital supplement may be presented at a second time. For example, a digital complement may be retrieved in response to an image content query based on an image of a household item or furniture from a catalog or store at a first time (e.g., while the user is viewing the catalog or while in the store). A digital complement including AR content of the household item or furniture may be presented at a second time (e.g., while the user is in a room where the household item or furniture may be installed).

The user interface engine 130 generates a user interface. The user interface engine 130 can also cause the client computing device 102 to display the generated user interface. The generated user interface may, for example, display information or content from the digital supplement. In some embodiments, the user interface engine 130 generates a user interface that includes multiple user-actuable controls, each associated with a digital supplement. For example, a user may actuate one of the user-actuable controls (e.g., by touching the control on a touchscreen, clicking the control using a mouse or another input device, or otherwise actuating the control).

Search server 152 is a computing device. Search server 152 may respond to search requests, such as image content queries. The response may include one or more digital supplements potentially related to the image content query. In one embodiment, search server 152 includes memory 160, processor assembly 154, and communications module 156. Memory 160 may include a content crawler 162, a digital supplement search engine 164, and a digital supplement data store 166.

The content crawler 162 may crawl network-accessible resources to identify digital supplements. For example, the content crawler 162 may access web pages accessible via the Internet, such as web pages provided by a digital supplement server 172. Crawling a network-accessible resource may include requesting the resource from a web server and analyzing at least a portion of the resource. Digital supplements may be identified based on metadata provided by the network-accessible resource, such as XML or JSON data that provides information about the digital supplement. In some embodiments, the crawler identifies network-accessible resources based on extracting links from previously crawled network-accessible resources. The content crawler 162 may also identify network-accessible resources to crawl based on receiving input submitted by a user. For example, a user may submit URLs (or other information) to network-accessible resources, including digital supplements, via a web form or an application programming interface (API). In some embodiments, the content crawler 162 generates an index of the identified digital supplements. The content crawler 162 may also generate a score associated with the digital supplement, such as a relevance score or a popularity (fame) score.

The digital supplement search engine 164 receives a search query and generates a response that may include one or more potentially related digital supplements. For example, the digital supplement search engine 164 may receive an image content query from the client computing device 102. The image content query may include an image. The digital supplement search engine 164 may identify supplement anchors within the image and, based on the identified supplement anchors, may identify associated or potentially related digital supplements. The digital supplement search engine 164 may send a response to the client computing device 102 that includes the digital supplement or information that can be used to access the digital supplement. In some embodiments, the digital supplement search engine 164 may return information associated with multiple digital supplements. For example, a list of digital supplements may be included in the response to the query. The list may be ordered based on relevance to the supplement anchors, popularity, or other characteristics of the digital supplements.

The image content query may include, for example, an image captured by camera assembly 112, or text or other data associated with an image captured by camera assembly 112. The image content query may also include other information, such as the location of client computing device 102 or an identifier for the user of client computing device 102. In some embodiments, search server 152 may determine the expected location of client computing device 102 from the user identifier (e.g., if the user has enabled location services on client computing device 102 that associate information about the user's location with the user's account).

The digital supplement data store 166 stores information about digital supplements. In some embodiments, the digital supplement data store 166 includes an index of digital supplements. For example, the index may be generated by the content crawler 162. The digital supplement search engine 164 may use the index to respond to search queries.

Digital supplement server 172 is a computing device. Digital supplement server 172 provides digital supplements. In some embodiments, digital supplement server 172 includes memory 180, processor assembly 174, and communications module 176. Memory 180 may include digital supplements 182 and metadata 184. In some embodiments, memory 180 may also include other network-accessible resources, such as web pages, that are not necessarily digital supplements. For example, memory 180 may store web pages that include metadata for providing details about one or more digital supplements and how to access those digital supplements. Additionally, memory 180 may include a resource serving engine, such as a web server, that responds to requests, such as HTTP requests, with network-accessible resources, such as web pages and digital supplements.

Digital supplement 182 is any type of content that can be provided as a supplement to something in the user's surrounding physical environment. Digital supplement 182 can also include any type of content that can supplement a stored image (e.g., of a previous physical environment around the user). For example, a digital supplement can be associated with a supplemental anchor such as an image, an object or product identified in the image, or a location. Digital supplement 182 can include one or more images, audio content, text data, video, games, data files, applications, or structured text documents. Examples of structured text documents include HyperText Markup Language (HTML) documents, XML documents, and other types of structured text documents.

Digital supplement 182 can launch an application and define parameters for that application. Digital supplement 182 can also send requests to a server (e.g., HTTP requests) and define parameters for the requests. In some embodiments, digital supplement 182 initiates a workflow to complete an activity, such as a workflow to complete a purchase. For example, digital supplement 182 can add a particular product to a user's shopping cart, add a coupon code, and send an HTTP request to a server to retrieve a purchase confirmation page.

Metadata 184 is data that describes a digital supplement. Metadata 184 may describe one or more digital supplements provided by digital supplement server 172 or provided elsewhere. Metadata 184 for a digital supplement may include one or more of the following: a type indicator, an anchor indicator, a name, a description, a preview snippet or excerpt, an associated image, a link, such as a URL, to the digital supplement, and an identifier for an application associated with the digital supplement. The metadata may also include information about the publisher of the digital supplement, such as the publisher name, a description of the publisher, and an image or icon associated with the publisher. In some embodiments, the metadata also includes contextual information about the digital supplement or contextual information that must be satisfied to provide the digital supplement. For example, the metadata may include conditions that must be met to access the digital supplement (e.g., geographic conditions, client computing device requirements, required applications). Exemplary contextual information includes a location, an entity identified in an image, or multiple entities identified in an image (e.g., some digital supplements may require a combination of entities recognized in an image). A recognized entity may be a supplement anchor. In some embodiments, the recognized entities do not supplement anchors but instead provide contextual information. Metadata 184 may also include supplemental anchors (e.g., text, code, entities, or types of entities) associated with the digital supplement.

Metadata 184 may be stored in a variety of formats. In some embodiments, metadata 184 is stored in a database. Metadata 184 may be stored as an XML file, a JSON file, or other format file. In some embodiments, digital supplement server 172 retrieves metadata 184 from a database and formats metadata 184 as XML, JSON, or otherwise provides a response to a request from a client or search server 152. For example, search server 152 may access metadata 184 to generate data that is stored in digital supplement data store 166 and used to respond to search requests from client computing devices 102.

The communications module 106 includes one or more devices for communicating with other computing devices, such as the search server 152 or the digital supplemental server 172. The communications module 106 may communicate over a wireless or wired network, such as the network 190. The communications module 156 of the search server 152 and the communications module 176 of the digital supplemental server 172 may be similar to the communications module 106.

Display device 108 may include, for example, an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touch screen, or any other screen or display for displaying images or information to a user. In some embodiments, display device 108 includes a light projector configured to project light onto a portion of the user's eye.

Memory 120 may include one or more non-transitory computer-readable storage media. Memory 120 may store instructions and data usable by client computing device 102 to implement the techniques described herein, such as to generate image content queries based on acquired images, transmit image content queries, receive responses to the image content queries, and present digital supplements identified in the responses to the image content queries. Memory 160 of search server 152 and memory 180 of digital supplement server 172 may be similar to memory 120 and may store data and instructions usable to implement the techniques of search server 152 and digital supplement server 172, respectively.

Processor assembly 104 includes one or more devices capable of executing instructions, such as instructions stored by memory 120, to perform various tasks associated with digital supplement association and retrieval for visual search. For example, processor assembly 104 may include a central processing unit (CPU) and/or a graphics processor unit (GPU). For example, if a GPU is present, some image/video rendering tasks, such as generating and displaying a user interface or displaying portions of a digital supplement, may be offloaded from the CPU to the GPU. In some embodiments, some image recognition tasks may also be offloaded from the CPU to the GPU.

Although not shown in FIG. 1, some embodiments include a head-mounted display device (HMD). The HMD may be a separate device from the client computing device 102, or the client computing device 102 may include the HMD. In some embodiments, the client computing device 102 communicates with the HMD via a cable. For example, the client computing device 102 may send video and/or audio signals to the HMD for display to the user, and the HMD may send movement, position, and/or orientation information to the client computing device 102.

The client computing device 102 may also include various user input components (not shown), such as a controller that communicates with the client computing device 102 using a wireless communication protocol. In some embodiments, the client computing device 102 communicates with the HMD via a wired connection (e.g., a Universal Serial Bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any BlueTooth protocol, Zigbee, etc.).
(not shown). In an embodiment, the client computing device 102 may be a component of the HMD and may be housed within the housing of the HMD.

Network 190 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or any other network. Client computing device 102 may receive, for example, audio/video signals, which, in one exemplary embodiment, may be provided over the network as part of a digital supplement.

FIG. 2 is a third-person perspective of an exemplary physical space 200 in which an embodiment of a client computing device 102 is accessing a digital capture. In this example, the physical space 200 includes an object 222, where the object 222 is artwork on a wall of the physical space 200. The object 222 is included within the field of view 204 of the camera assembly 112 of the client computing device 102.

An exemplary user interface screen 206 is also shown. The user interface screen 206 may be generated, for example, by the user interface engine 130 of the client computing device 102. The user interface screen 206 includes an image display panel 208 and a digital capture selection panel 210. The image display panel 208 displays images. For example, the image display panel 208 may show images corresponding to a real-time feed from the camera assembly 112 of the client computing device 102. In some embodiments, the image display panel 208 shows previously acquired images or images retrieved from the memory 120 of the client computing device 102.

In some embodiments, the user interface screen 206 is presented to the user on a display device of the client computing device 102. In some embodiments, the user interface screen 206 may be so overlaid on an image of the physical space (or a video feed being acquired by a camera on the computing device). Additionally, the user interface screen 206 may be displayed as AR content across the user's field of view using an HMD worn by the user.

Image display panel 208 may also include annotations or user interface elements associated with the image. For example, image display panel 208 may include an indicator that an object in the image (e.g., object 222) has been recognized as a supplemental anchor. The indicator may include user-actuable controls for accessing or viewing information about the digital supplement associated with the identified supplemental anchor. In some situations, an image displayed on image display panel 208 may include multiple objects recognized as supplemental anchors, and image display panel 208 may include multiple annotations overlaying the image to identify those supplemental anchors.

Supplementary anchors may be recognized by a supplementary anchor identification engine of client computing device 102. In one embodiment, supplementary anchors are identified by sending an image to search server 152. Search server 152 may then analyze the image and identify supplementary anchors within the image. In one embodiment, search server 152 may send one or more of the locations (e.g., image coordinates) associated with the supplementary anchors or the dimensions of any identified objects to client computing device 102. Client computing device 102 may then update a user interface screen to show annotations identifying the supplementary anchors (or associated objects) within the image. In one embodiment, client computing device 102 may track the locations of the supplementary anchors (or associated objects) within the video stream (e.g., a sequence of sequentially acquired images) captured by camera assembly 112 (e.g., supplementary anchor identification engine 124 may track supplementary anchors identified by search server 152).

The digital supplement selection panel 210 allows a user to select a digital supplement for presentation. For example, the digital supplement selection panel 210 may include a menu including user-actuatable controls, each associated with a digital supplement. In this example, the digital supplement selection panel 210 includes user-actuatable controls 212 and 214, each including information about the associated digital supplement. For example, the user-actuatable controls may display one or more of a name (or title), a short description, and an image associated with the digital supplement, which may be received from the search server 152. When the user-actuatable control 212 or the user-actuatable control 214 is actuated, content of the associated digital supplement may be presented to the user. Presenting the digital supplement to the user may include causing the client computing device 102 to display a user interface screen including images, video, text, other content, or a combination thereof, from the digital supplement. In some embodiments, the digital supplement content is displayed as an overlay on the image display panel 208 over an image or camera feed. The digital supplement content may be three-dimensional augmented reality content.

In some embodiments, presenting the digital supplement includes launching an application (e.g., one of the other applications 140) installed on the client computing device 102. Presenting the digital supplement may also include sending a request to a URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as an identifier for a product or object identified in the image. In some embodiments, the image (or other content) from the image content query is passed with the parameters. The image may also be provided via an API associated with the digital supplement server 172. In some embodiments, the client computing device 102 sends the image to the digital supplement server 172. In some embodiments, the search server 152 may send the image to the digital supplement server 172. For example, in response to a user selecting a digital supplement, the client computing device 102 may send an indicator of the selection to the search server 152, which may then send the image to the corresponding digital supplement server. The client computing device 102 may also send a URL to a location on the search server 152 that the digital supplement server 172 can use to access the image. Beneficially, these implementations may reduce the amount of data that the client computing device needs to transmit.

A digital supplement associated with user-actuatable control 212 may cause information about object 222 to be displayed, such as information from the museum. A digital supplement associated with user-actuatable control 214 may cause information related to the museum to be displayed. For example, presenting a digital supplement may cause a stop on a museum tour to be marked as completed and display information for the next stop.

Figure 3 is a diagram of an example method 300 for enabling triggers for a digital supplement, consistent with embodiments described herein. This method 300 may be performed, for example, by a content crawler 162 of a search server 152 to enable a user to access a digital supplement based on an image content query.

In operation 302, data specifying a digital supplement is received. The data may identify the digital supplement and the context in which the digital supplement should be provided. The data specifying the digital supplement may be received in a variety of ways. For example, the data specifying the digital supplement may be received from a network-accessible resource, such as a web page, that includes metadata about the digital supplement. The data specifying the digital supplement may also be received, for example, via an API or form provided by search server 152. The data specifying the digital supplement may also be received from a memory location or data store.

Data about a digital supplement may include access data usable by a client computing device to access the digital supplement. For example, the access data may include the URL of the digital supplement and parameters to be passed to the URL. The access data may also include an application identifier and parameters for the application. Data about a digital supplement may also include descriptive data about the digital supplement. The descriptive data may be used by the client computing device to present information about the digital supplement to a user (e.g., on a menu from which the user can select the digital supplement). The descriptive data may include, for example, a name (or title), a description, the name of the publisher, and an image. Data about a digital supplement may also include an identifier for a supplement anchor.

At operation 304, a data structure instance based on the received data is generated. The data structure may be, for example, a record in a database. The database may be a relational database, and the data structure instance may be linked (e.g., via an external key) to one or more records associated with the supplemental anchor.

In operation 306, after generating the data structure instance, the digital supplement is enabled for searching by image content queries. For example, a database field associated with the data structure instance may be set to active to enable the digital supplement search engine 164 to access and return the associated digital supplement. In some embodiments, triggering the digital supplement may include saving or committing a database record. In some embodiments, enabling searching for the digital supplement includes enabling triggering of the digital supplement by a client computing device. For example, after the instance is generated, the digital supplement may be returned to the client computing device in response to a search and activated or presented by the client computing device.

FIG. 4 is a diagram of an example method 400 for enabling triggers for a digital supplement, consistent with embodiments described herein. This method 400 may be performed, for example, by the content crawler 162 of the search server 152 to enable users to access a digital supplement based on an image content query.

In operation 402, network-accessible resources are analyzed. In some embodiments, the network-accessible resources are, for example, web pages presented by digital supplemental server 172. In some embodiments, a set of network-accessible resources is analyzed. The set of network-accessible resources may be generated based on submissions via a form or an API. In some embodiments, the set of network-accessible resources may be generated by crawling other network-accessible resources to identify URLs. This crawling process may be performed recursively.

At operation 404, metadata associated with the digital supplement within the network-accessible resource is identified. In some embodiments, the network-accessible resource may include an indicator of the metadata associated with the digital supplement. For example, the network-accessible resource may include a tag that identifies a portion of the network-accessible resource that includes the metadata. The tag may be an XML tag with a particular type or attribute. The tag may be an HTML tag, such as a script tag, that includes a JSON data structure that includes the metadata.

In operation 406, a digital supplemental data structure instance based on the metadata is generated. Operation 406 may be similar to operation 304.

At operation 408, an image content query is received. The image content query may be sent by a client computing device, such as client computing device 102, for example. In some embodiments, the image content query includes an image. The image content query may also include text data describing the image. For example, the text data may include an identifier of a supplemental anchor in an image captured by the camera assembly of the client computing device. In some embodiments, the image content query also includes other information, such as the location of the client computing device or an identifier of a user account associated with the client computing device.

At operation 410, multiple digital supplemental data structure instances are identified based on the image content query. In some embodiments, supplemental anchors are identified within the image provided in the image content query. The supplemental anchors may then be used to query an index or database for associated digital supplements. In some embodiments, other data provided with the query may also be used to identify the digital supplement, such as the location of the client computing device or information associated with a user account. In some embodiments, multiple supplemental anchors may be used to identify the associated supplemental anchors.

In operation 412, an ordering of the plurality of digital supplement data structure instances is determined. The ordering may be based on various scores associated with the digital supplements or the relevance of the digital supplements to the image content query. In some embodiments, a relevance score corresponding to the relevance of the digital supplement to the image content query may be used to order the plurality of digital supplement data structure instances. The relevance score may be determined from multiple factors, such as one or more of the content of the digital supplement, the content of a network-accessible resource that links to the digital supplement (or a network-accessible resource associated with the digital supplement), link text, or content near links to the digital supplement on other network-accessible resources.

The score may also be based on a popularity metric. A prestige metric is one example of a popularity metric. The prestige metric may be based on how many other network resources link to the digital supplement in combination with the prestige scores of those other network-accessible resources. In some embodiments, the popularity score may be based on how frequently the digital resource is selected or has been selected. In some embodiments, the popularity score may correspond to how frequently the digital resource is selected for image content queries.

The scores may be determined or retrieved from a data store or API. In some embodiments, an API is accessed to retrieve the scores of the digital supplement. For example, the scores may be retrieved from a search engine that determined the relevance and/or popularity of the digital resource with respect to search terms based on the supplemental anchors.

The plurality of digital supplemental data structures may also be ordered based on frequency of use by a particular user (e.g., a user of a client computing device) or recency of use by that particular user. In some embodiments, the plurality of digital supplemental data structures are ordered randomly.

At operation 414, the image content query is answered based on the multiple digital supplement data structure instances. For example, information associated with the multiple digital supplement data structure instances may be sent to the client computing device in the order determined at operation 412. In some embodiments, the information includes descriptive data that may be presented within a menu or another type of user interface configured to receive a user selection of a digital supplement. The information may also include access data that may be used by the client computing device to access or present the digital supplement.

FIG. 5 is a diagram of an example method 500 for searching and presenting digital supplements according to embodiments described herein. The method 500 may be performed, for example, by application 122 of client computing device 102 to identify and access digital supplements based on an image content query.

At operation 502, an image-based image content query is sent to a server computing device (e.g., search server 152). For example, the image may be captured by camera assembly 112 of client computing device 102. The image may also be a stored image, such as an image previously captured by camera assembly 112. In some embodiments, the image content query includes only the image. In some embodiments, the image content query includes additional information. For example, the image content query may include information such as the location of client computing device 102 or an identifier for an account associated with the user of client computing device 102. Application 122 may also identify anchors within the image (e.g., using supplemental anchor identification engine 124). The image content query may include an identifier (e.g., text, number, or other type of identifier) for the identified anchor. In at least some embodiments, the image content query does not include an image.

In some embodiments, sending the image content query to the server includes invoking an API. In some embodiments, sending the image content query to the server includes invoking an API provided by the server. In some embodiments, sending the image content query to the server includes submitting a form using the HTTP protocol (e.g., by submitting a GET request or a POST request).

At operation 504, a response to the image content query identifying a digital supplement is received. The response may be received over network 190 from search server 152. The response may include one or more digital supplements identified by search server 152 based on the image content query. For example, the response may include an array of data associated with the digital supplement. In some embodiments, the data associated with the digital supplement may include descriptive data that can be used to present digital supplement options for a user to select. For example, the descriptive data may include a name, a short description, a publisher name, and an image. The data may also include access data, such as a URL and parameters to include in a request via a URL, or an application name and associated parameters. The data may also include the location, coordinates, or dimensions of a supplemental anchor in the image sent in the image content query (e.g., if a supplemental anchor is identified by search server 152).

At operation 506, a user interface screen containing information associated with the digital supplement is displayed. In some embodiments, the user interface screen includes an annotation overlaying the identified supplemental anchor (e.g., based on the provided coordinates). The annotation may provide information about an object within the image associated with the identified supplemental anchor. The annotation may include user-actuable controls that may be actuated to present or launch the digital supplement. The user interface screen may also include a digital supplement selection panel that may be used to select from multiple digital supplements identified in the response received at operation 504. In some embodiments, the user interface screen may be generated by a web browser opening a URL specified by the digital supplement. The user interface screen may also be generated by another application launched to provide the digital supplement.

FIG. 6 is a diagram of an example method 600 for identifying and presenting a digital supplement based on an image, consistent with embodiments described herein. The method 600 may be performed, for example, by application 122 of client computing device 102 to identify and access a digital supplement based on an image content query.

At operation 602, an image is captured. For example, the image may be captured by the camera assembly 112 of the client computing device 102. In some embodiments, a series of images (i.e., video) may be captured by the camera assembly 112.

In operation 604, the image-based image content query is sent to a server computing device, such as search server 152. Operation 604 may be similar to operation 502. In embodiments in which a sequence of images is obtained, the image content query may include multiple images or a sequence of images. In some embodiments, the sequence of images may be streamed to the server computing device.

In operation 606, a response to the image content query is received that identifies multiple digital captures. Operation 606 may be similar to operation 504 described above.

At operation 608, a user interface screen including user-actuatable controls for selecting a digital supplement from the plurality of digital supplements is displayed. For example, a digital supplement selection panel may be displayed. The digital supplement selection panel may include a plurality of user-actuatable controls, each associated with one of the plurality of digital supplements identified in the response. The digital supplement selection may provide the user-actuatable controls based on an ordering or ranking of the digital supplements provided by the server computing device. The digital supplement selection panel may arrange the user-actuatable controls vertically, horizontally, or in other manners. The user-actuatable controls may be associated with or include information about the associated digital supplement that the user can consider in determining whether to select the digital supplement. For example, the displayed information may include one or more of the name, description, image, and publisher name of the digital supplement.

In operation 610, user input for selecting a digital supplement is received. The user input may be a click with a mouse or other device. The user input may also be touch input from a stylus or finger. Another example of user input is near-touch input (e.g., holding a finger or pointing device close to the screen). In some embodiments, the user input may also include hand gestures, head movements, eye movements, or speech input.

In operation 612, information is provided to a resource associated with the selected digital supplement. For example, information about the user of the client computing device may be sent to a server providing the digital supplement (if permission to provide the information is provided). The information may also be provided to an application providing the digital supplement. Various types of information may be provided. For example, the information may include user information such as username, user preferences, or location.

The information may also include information associated with the image content query, such as an image or a sequence of images. The information may also include identifiers and/or locations of one or more supplemental anchors within the image. This information may be used to provide a digital supplement to the user. For example, AR content of the digital supplement may be formed and positioned based on the image.

The information may be sent by the client computing device 102 directly to the resource associated with the digital supplement (e.g., the digital supplement server 172). In some embodiments, the information is provided to the resource associated with the digital supplement by the search server 152 (e.g., the client computing device does not need to send as much data). In at least some of these implementations, the client computing device 102 may send selection information identifying the selected digital supplement to the search server 152. Upon receiving the selection and confirming that the user has authorized the information to be shared, the search server 152 may send the information to the resource providing the digital supplement. The client computing device 102 may also prompt the user to allow the information to be shared. In some embodiments, the search server 152 may determine the information to send to the resource based on the digital supplement data structure instance (which may be based on metadata associated with the digital supplement).

In operation 614, the user interface is updated based on the selected digital supplement. Operation 614 may be similar to operation 506.

7A-7C are schematic diagrams of user interface screens displayed by embodiments of client computing device 102 for conducting image content searches and displaying digital supplements. In FIG. 7A, user interface screen 700a is shown. User interface screen 700a includes an image display panel 708 and an information panel 730. In this example, image display panel 708 displays an image of a shelf full of wine bottles (e.g., as might be found in a store). Image display panel 708 also includes indicator 740 and indicator 742. Each of these indicators indicates that the wine bottle shown in the image below the indicator has been recognized as a supplement anchor (e.g., as a recognized product in this case). Indicator 740 and indicator 742 are examples of user-actuatable controls. Within information panel 730, an instruction is provided to "tap what interests you."

In FIG. 7B, user interface screen 700b is shown after the user activates indicator 740. After activation, annotations 744 from the digital supplement are displayed. Annotations 744 include information about the wine's rating, which may help the user select a bottle of wine to purchase.

In FIG. 7C, another user interface screen 700c is shown after a user activates indicator 740. User interface screen 700c is shown instead of or in addition to displaying user interface screen 700b (e.g., after activation of annotation 744 or when a user swipes up on information panel 730 of FIG. 7B). In FIG. 7C, an expanded information panel 732 is shown. The expanded information panel 732 occupies more of user interface screen 700c than the information panel 730 featured in FIGS. 7A and 7B.

The expanded information panel 732 includes a digital supplement selection panel 710 and a digital supplement content display panel 734. The digital supplement selection panel 710 includes a user-actuable control 712, a user-actuable control 714, and a (partially visible) user-actuable control 716. In some embodiments, when a user swipes on the digital supplement selection panel 710, additional user-actuable controls may be displayed. The user-actuable controls of the digital supplement selection panel 710 may be arranged in ranked order. The user-actuable control 712 is associated with a digital supplement for meal pairings. When the user-actuable control 712 is activated, a digital supplement displaying food and meal pairing information for the selected wine may be displayed. The user-actuable control 714 is associated with a digital supplement for saving photos. When activated, a photo saving application may be activated to provide the image. Additional information may be saved with the photo, such as an identified supplement anchor.

The digital supplemental content display panel 734 may display content from the digital supplement. The digital supplemental content display panel 734 may display the default digital supplement or the highest-ranked digital supplement associated with the identified supplemental anchor. In this example, the digital supplemental content display panel 734 includes product information about the product associated with the selected supplemental anchor. In this case, the wine name, rating, origin, image, and comments are provided.

Figures 8A-8C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing an image content search and displaying digital supplements. In this example, the image content search is based on an image of a receipt.

FIG. 8A illustrates user interface screen 800a. User interface screen 800a includes image display panel 808 and information panel 830. In this example, image display panel 808 displays an image of a receipt from a restaurant. Image display panel 808 also includes indicator 840, indicator 842, annotation 844, and highlight overlay 846. In this case, indicator 840 is associated with the receipt as a document, and indicator 842 is associated with the particular restaurant named on the receipt. The identified receipt document and the identified restaurant name are both examples of supplemental anchors.

Annotation 844 is associated with a digital supplement providing a tip calculator. In this example, an exemplary tip calculator is included in annotation 844 and overlaid in an appropriate location on image display panel 808. In some embodiments, the digital supplement may be selected by default or may be displayed upon identifying the appropriate supplement anchor. A highlight overlay 846 is overlaid on a portion of the receipt document containing information used by the tip calculator digital supplement.

In this example, the items displayed in information panel 830 relate to the receipt as a document, just as if indicator 840 had been activated. In some embodiments, the identified supplemental anchors are ranked based on the user's likely relevance or interest, for example, based on the user's past actions, other users' actions on similar images, the supplemental anchor's confidence score, or the location or size of the portion of the image to which the supplemental anchor is associated. Information panel 830 may then, in at least some embodiments, display items related to the highest-ranked supplemental anchor. If indicator 842 had instead been activated, information panel 830 may include items related to a particular restaurant.

Here, information panel 830 includes digital supplement selection panel 810. The digital supplement panel includes user-actuatable control 812, user-actuatable control 814, and user-actuatable control 816. In this example, user-actuatable control 812 is associated with a tip calculator digital supplement, user-actuatable control 814 is associated with a check split digital supplement, and user-actuatable control 816 is associated with an expense report digital supplement. For example, when user-actuatable control 812 is actuated, user interface controls for adjusting parameters of the tip calculator (e.g., to adjust a percentage) may be displayed.

In FIG. 8B, user interface screen 800b is shown after a user activates user-actuable control 814. After activation, an expanded information panel 832 is shown containing items to help the user calculate how to split a bill. For example, the number of people splitting a bill can be entered to determine the amount each person owes.

In FIG. 8C, user interface screen 800c is shown after a user activates user-actuable control 816. After activation, an expanded information panel 834 is shown containing items to help the user store the receipt in an expense report. For example, the user can select the expense report with which the receipt should be associated (e.g., "Sydney Trip 2018"). Once the expense report is selected, the image of the receipt can be uploaded to an expense report submission or management system. In some embodiments, the entire image displayed in image display panel 808 is uploaded. In some embodiments, a portion of the image is uploaded (e.g., the image is cropped to include only the receipt).

Figures 9A and 9B are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing an image content search and displaying a digital supplement. In this example, the image content search is based on images of faces.

FIG. 9A shows user interface screen 900a. User interface screen 900a includes an image display panel 908 and an information panel 930. In this example, image display panel 908 displays an image of a face, where the face is an example of a supplemental anchor. Information panel 930 includes user-actuatable controls 912 for a digital supplement identified for the supplemental anchor in the image (i.e., the face). User-actuatable controls 912 are associated with a digital supplement for binding to eyeglasses.

9B, user interface screen 900b is shown after a user activates user-actuatable control 912. After activation, an expanded information panel 932 is shown containing items to help the user visually try on eyeglasses on a face in an image. Here, multiple eyeglass styles are displayed, and the user can select a pair to try on. Upon selecting a pair of eyeglasses, AR content 960 is overlaid on image display panel 908, where the AR content 960 corresponds to the selected eyeglasses and is sized to fit the face in the image. In one embodiment, once a digital supplement for trying on eyeglasses is selected, image display panel 908 displays an AR content 960.
The image shown in 8 is sent to a server providing digital supplements so that the image can be analyzed to determine where and how to place the AR content 960, how large it should be, or whether to recommend trying on glasses.

Figures 10A-10C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for conducting an image content search and displaying a digital supplement. In this example, the image content search is based on images of furniture in a catalog.

FIG. 10A shows user interface screen 1000a. User interface screen 1000a includes image display panel 1008. In this example, image display panel 1008 displays an image of a portion of a page from a furniture catalog. The image display panel also includes indicator 1040, indicator 1042, and indicator 1044. In this example, indicator 1040 is associated with beds, indicator 1042 is associated with decorative items, and indicator 1044 is associated with rugs. The images of beds, decorative items, and rugs in the catalog are examples of supplemental anchors.

In FIG. 10B, user interface screen 1000b is shown after a user selects indicator 1040 (e.g., by touching the screen at or near where indicator 1040 appears). User interface screen 1000b includes a digital supplement selection panel 1010 and an information panel 1030. Information panel 1030 includes information (e.g., product name, description, image) about the supplement anchor associated with the selected indicator.

Digital supplement selection panel 1010 includes user-actuatable control 1012 and user-actuatable control 1014. User-actuatable control 1012 is associated with a digital supplement that provides an in-home view. User-actuatable control 1014 is associated with another digital supplement (e.g., a digital supplement for posting to a social media site).

In FIG. 10C, user interface screen 1000c is shown after activation of user-actuatable control 1012. User interface screen 1000c includes an image display panel 1008, a digital supplement selection panel 1010, and a reduced information panel 1032. The reduced information panel 1032 may include a user-actuatable control that, when activated, may cause an information panel to pop up and be displayed.

Here, the image display panel 1008 displays an image of the room and includes AR content 1060. The AR content 1060 includes a 3D model of a bed associated with the indicator 1040 overlaid on the image panel. The user may adjust the position of the AR content 1060 within the room to see how the bed fits within the room. In one embodiment, when a digital supplement for the home view is selected, the image shown on the image display panel 1008 is transmitted to a server providing the digital supplement, so that the image can be analyzed to determine where and how the AR content 1060 should be positioned and what size it should be. In one embodiment, the AR content 1060 may be provided after the image content query.

11A-11C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for conducting various image content searches in a store. In this example, the image content searches are based on images of products acquired in the store.

FIG. 11A illustrates user interface screen 1100a. User interface screen 1100a includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 displays an image taken in a store. Image display panel 1108 also includes an indicator 1140 associated with a vase. The vase displayed in image display panel 1108 is an example of a supplement anchor. Information panel 1130 displays a digital supplement that includes product information about the vase and functionality for purchasing the vase. The digital supplement may include, for example, a workflow for initiating a purchase of the vase. In this example, the digital supplement is identified based on the image content and the location of the client computing device, such that a digital supplement published by (or associated with) the store where the image was captured may be identified and provided as a high-ranking result in response to an image content query when the client computing device is in the store. In some embodiments, if the location of the client computing device is changed, a different digital supplement may be provided for the same image.

11B shows user interface screen 1100b. User interface screen 1100b includes image display panel 1108 and information panel 1130. In this example, image display panel 1108 displays another image captured in-store. Image display panel 1108 also includes indicator 1142 associated with a rug. The rug displayed in image display panel 1108 is an example of a supplement anchor. Information panel 1130 displays a digital supplement that includes product information about the rug and functionality for selecting a size and purchasing the rug. As in FIG. 11A, the digital supplement is identified based on the image content and the location of the client computing device.

11C illustrates user interface screen 1100c. User interface screen 1100c includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 displays another image captured in the store. Image display panel 1108 also includes an indicator 1144 associated with a vase. The vase displayed in image display panel 1108 is an example of a supplement anchor. Information panel 1130 displays a digital supplement including product information about the vase. Information panel 1130 also includes a coupon indicator 1132 and functionality for redeeming the coupon. Redeeming the coupon may include purchasing the item at a discounted price from a website associated with the retailer. In some embodiments, a coupon code is presented that can be used to secure the discount during checkout. Similar to FIGS. 11A and 11B, the digital supplement is identified based on the image content and the location of the client computing device.

Figures 12A-12C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device 102 during various image content searches. In this example, the image content search is based on images of movie posters (e.g., as might be obtained at a movie theater).

12A illustrates a user interface screen 1200a. User interface screen 1200a includes an image display panel 1208. In this example, image display panel 1208 displays an image of a movie poster. Image display panel 1208 also includes an indicator 1240 associated with the movie poster identified in the image. The movie poster is an example of a supplement anchor. Indicator 1240 may include a user-actuable control that, when activated, displays a digital supplement or a menu for selecting a digital supplement.

User interface screen 1200b is shown in FIG. 12B. Image display panel 1208 also includes a preview digital supplement 1242 associated with the movie poster identified in the image. For example, preview digital supplement 1242 may be shown after activation of indicator 1240 (of FIG. 12A). Preview digital supplement 1242 may overlay an image or footage from the movie associated with the identified movie poster over the image of the movie poster.

User interface screen 1200c is shown in FIG. 12C. Image display panel 1208 also includes rating indicators 1244 and 1246. Rating indicators 1244 and 1246 may be generated by one or more digital supplements in response to an image content query that includes a movie poster. The digital supplement may, for example, overlay rating information within the image for a movie associated with the movie poster. Rating indicators 1244 and 1246 may include user-actuable controls that, when activated, cause the rating and additional information about the associated movie to be revealed.

13 illustrates an example of a computing device 1300 and a mobile computing device 1350 that may be used with the technology described herein (e.g., to implement the client computing device 102, search server 152, and digital supplemental server 172). The computing device 1300 includes a processor 1302, memory 1304, storage device 1306, a high-speed interface 1308 for connecting to the memory 1304 and a high-speed expansion port 1310, and a low-speed interface 1312 for connecting to a low-speed bus 1314 and storage device 1306. Each of the components 1302, 1304, 1306, 1308, 1310, and 1312 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 1302 can process instructions for execution within computing device 1300, including instructions stored in memory 1304 or storage device 1306, to display graphical information for a GUI on an external input/output device, such as a display 1316 coupled to high-speed interface 1308. In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and types of memory, as needed. Multiple computing devices 1300 may also be connected, each providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

Memory 1304 stores information within computing device 1300. In one embodiment, memory 1304 is one or more volatile memory units. In another embodiment, memory 1304 is one or more non-volatile memory units. Memory 1304 may also be another form of computer-readable storage medium, such as a magnetic or optical disk.

The storage device 1306 may provide mass storage for the computing device 1300. In one embodiment, the storage device 1306 may be or include a computer-readable storage medium such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. A computer program product may be tangibly embodied on an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier may include the memory 1304.
, storage device 1306 , or memory on processor 1302 .

The high-speed controller 1308 manages bandwidth-intensive operations for the computing device 1300, while the low-speed controller 1312 manages less bandwidth-intensive operations. This allocation of functionality is exemplary only. In one embodiment, the high-speed controller 1308 is coupled to the memory 1304, to the display 1316 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 1310, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 1312 is coupled to the storage device 1306 and the low-speed expansion port 1314. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, pointing device, scanner, or networking device such as a switch or router, for example, via a network adapter.

Computing device 1300, as shown in the figure, may be implemented in several different forms. For example, it may be implemented as a standard server 1320, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 1324. Additionally, it may be implemented as a personal computer, such as a laptop computer 1322. Alternatively, components from computing device 1300 may be combined with other components in a mobile device (not shown), such as device 1350. Each such device may include one or more of computing devices 1300, 1350, or an entire system may be composed of multiple computing devices 1300, 1350 communicating with each other.

Computing device 1350 includes, among other components, a processor 1352, memory 1364, an input/output device such as a display 1354, a communications interface 1366, and a transceiver 1368. Device 1350 may also include a storage device such as a microdrive or other device to provide additional storage. Each of components 1350, 1352, 1364, 1354, 1366, and 1368 are interconnected using various buses, and some of the components may be located on a common motherboard or otherwise as desired.

Processor 1352 can execute instructions within computing device 1350, including instructions stored in memory 1364. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide, for example, control of a user interface, applications executed by device 1350, and coordination of other components of device 1350, such as wireless communication by device 1350.

The processor 1352 may communicate with a user via a control interface 1358 and a display interface 1356 coupled to a display 1354. The display 1354 may be, for example, a thin-film-transistor liquid crystal display (TFT LCD), a light emitting diode (LED), an organic light emitting diode (OLED), or a liquid crystal display (LCD).
) display, or other suitable display technology. Display interface 1356 may include appropriate circuitry for driving display 1354 to present graphics and other information to a user. Control interface 1358 may receive commands from a user and translate and submit them to processor 1352. Additionally, external interface 1362 may be provided in communication with processor 1352 to enable short-range communication between device 1350 and other devices. External interface 1362 may, for example, provide for wired communication in some embodiments or for wireless communication in other embodiments, and multiple interfaces may also be used.

Memory 1364 stores information within computing device 1350. Memory 1364 may be embodied as one or more computer-readable storage media, one or more volatile memory units, or one or more nonvolatile memory units. Expansion memory 1374 may also be provided and connected to device 1350 via expansion interface 1372, which may include, for example, a Single In-Line Memory Module (SIMM) card interface. Such expansion memory 1374 may provide additional storage space for device 1350 or store applications or other information for device 1350. Specifically, expansion memory 1374 may include instructions for performing or supplementing the processes described above and may also include secure information. Thus, for example, expansion memory 1374 may be provided as a security module for device 1350 and may be programmed with instructions that enable secure use of device 1350. Additionally, secure applications can be provided via the SIMM card with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable storage medium or machine-readable medium, such as memory 1364, expansion memory 1374, or memory on processor 1352, which may be received, for example, via transceiver 1368 or external interface 1362.

Device 1350 may communicate wirelessly via communication interface 1366, which may include digital signal processing circuitry as needed. Communication interface 1366 may provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via radio frequency transceiver 1368. Additionally, short-range communication may occur, such as with Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, GPS (Global Positioning System) receiver module 1370 may provide device 1350 with additional navigation- and location-related radio data that may be used as needed by applications executing on device 1350.

Device 1350 may also communicate audibly using voice codec 1360, which may receive speech information from the user and convert it into usable digital information. Voice codec 1360 may also generate audible sounds for the user, such as through a speaker in the handset of device 1350. Such sounds may include sounds from a voice call, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 1350.

The computing device 1350 may be implemented in several different forms, as shown in the figure. For example, it may be implemented as a mobile phone 1380. It may also be implemented as part of a smartphone 1382, personal digital assistant, or other similar mobile device.

Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementation in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and/or object-oriented programming language and/or in an assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable storage medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and/or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for user interaction, the systems and techniques described herein may be implemented on a computer having a display device (such as an LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor/screen) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user may provide input to the computer. Other types of devices may also be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a client computer having a graphical user interface or web browser through which a user may interact with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

In some embodiments, the computing device shown in FIG. 13 may include sensors to interface with the AR headset/HMD device 1390 to generate an augmented environment for viewing inserted content within a physical space. For example, one or more sensors included in the computing device 1350 or other computing devices shown in FIG. 13 may provide input to the AR headset 1390 or may provide input to the AR space generally. The sensors may include, but are not limited to, a touchscreen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 1350 may use the sensors to determine the computing device's absolute position and/or detected rotation in the AR space, which may then be used as input to the AR space. For example, the computing device 1350 may be integrated into the AR space as a virtual object such as a controller, laser pointer, keyboard, or weapon. When integrated into the AR space, the user's positioning of the computing device/virtual object may allow the user to position the computing device to view the virtual object in a particular way within the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were a real laser pointer. The user can move the computing device left and right, up and down, in a circular motion, and use the device in the same way as they would use a laser pointer. In some embodiments, the user may use the virtual laser pointer to aim at a target.

In some embodiments, one or more input devices included in or coupled to computing device 1350 may be used as input to the AR space. The input devices may include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or earbuds with input capabilities, a game controller, or other connectable input devices. When the computing device is integrated into the AR space, a user interacting with an input device included in computing device 1350 can cause specific actions in the AR space.

In some embodiments, the touchscreen of computing device 1350 may be rendered as a touchpad within the AR space. A user may interact with the touchscreen of computing device 1350. The interaction is rendered, for example, in AR headset 1390, as movements on a touchpad rendered within the AR space. The rendered movements may control virtual objects within the AR space.

In some embodiments, one or more output devices included in computing device 1350 can provide output and/or feedback to a user of AR headset 1390 within the AR space. The output and/or feedback can be visual, tactile, or audio. The output and/or feedback can include, but is not limited to, vibration, turning one or more lights or strobes on and off or flashing and/or emitting light, sounding an alarm, playing a chime, playing a song, and playing an audio file. Output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobes, and speakers.

In some embodiments, the computing device 1350 may appear as another object in the computer-generated 3D environment. A user's interaction with the computing device 1350 (e.g., rotating, shaking, touching the touchscreen, swiping a finger across the touchscreen) may be interpreted as an interaction with an object in the AR space. In the example of a laser pointer in the AR space, the computing device 1350 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 1350, the user in the AR space sees the movement of the laser pointer. The user receives feedback from their interaction with the computing device 1350 in the AR environment on the computing device 1350 or the AR headset 1390. The user's interaction with the computing device may be translated into an interaction with a user interface generated in the AR environment for the controllable device.

In some embodiments, computing device 1350 may include a touchscreen. For example, a user may interact with the touchscreen to interact with a user interface for the controllable device. For example, the touchscreen may include user interface elements such as sliders that can control characteristics of the controllable device.

Computing device 1300 is intended to represent various forms of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1350 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely exemplary and are not intended to limit the embodiments of the invention(s) described and/or claimed herein.

Several embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the present specification.

Additionally, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve desirable results. Additionally, other steps may be provided or steps may be eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

While several features of the described embodiments have been illustrated as set forth herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the embodiments. It should be understood that they have been presented by way of example only, and not by way of limitation, and that various changes in form and detail may be made. Any portions of the apparatus and/or methods described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein may include various combinations and/or subcombinations of the functions, components, and/or features of the different described embodiments.

Claims (4)

1. A computer-implemented method comprising:
a processor of the computing device executing an application;
acquiring an image of a physical object with a camera assembly of the computing device operating within the application;
a communication module of the computing device sending a query including the image and context information including location information to a search server;
and receiving, from the search server, a response to the query, the response identifying a digital supplement server and identifying a digital supplement provided by the search server from the identified digital supplement server, the digital supplement including information associated with the image, the computer-implemented method further comprising:
A computer-implemented method, wherein the application displays on a display device of the computing device a user interface including the information from the digital capture associated with the image, wherein in the user interface, the information from the digital capture is displayed overlaid on the image of the physical object as augmented reality content. The computer-implemented method of claim 1, wherein receiving the response identifying the digital capture includes receiving information related to at least one product identified in the image. Displaying the user interface includes :
purchasing information associated with the at least one product identified in the image ; or
The computer-implemented method of claim 2 , further comprising displaying at least one of characteristic information associated with the at least one product identified in the image. A computer program for causing a processor to execute the method according to any one of claims 1 to 3 .