AU2020370038B2

AU2020370038B2 - System and method of personalized navigation inside a business enterprise

Info

Publication number: AU2020370038B2
Application number: AU2020370038A
Authority: AU
Inventors: Edward L. Hill
Original assignee: Position Imaging Inc
Current assignee: Position Imaging Inc
Priority date: 2019-10-21
Filing date: 2020-10-20
Publication date: 2025-10-02
Anticipated expiration: 2040-10-20
Also published as: EP4118396A1; WO2021080933A1; AU2020370038A1; JP2022552526A; KR20220082011A; EP4118396A4; CA3154571A1

Abstract

Systems and methods for tracking movement of individuals through a building receive, by one or more RF nodes disposed near an entrance to the building, RF signals from RF-transmitting mobile devices carried by persons near the entrance, capture an image of the persons while they are near the entrance, determine an identity and relative distance of each RF-transmitting mobile device from each RF node based on information associated with the RF signals received by that RF node, detect humans in the image, determine a relative depth of each human in the image, and assign the identity of each RF-transmitting mobile device to one of the humans detected in the image based on the relative distance of each RF-transmitting mobile device from each RF node and the relative depth of each human in the image, thereby identifying each individual who to be tracked optically as that individual moves throughout the building.

Description

WO wo 2021/080933 PCT/US2020/056405

SYSTEM AND METHOD OF PERSONALIZED NAVIGATION INSIDE A BUSINESS ENTERPRISE

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part application of pending U.S. Patent Application No.

16/163,708, filed October 18, 2018, titled "System and Method of Personalized Navigation Inside a

Business Enterprise," which claims the benefit of and priority to continuation-in-part application of U.S.

Patent Application No. 15/839,298, filed December 12, 2017, titled "System and Method of Personalized

Navigation Inside a Business Enterprise," which claims the benefit of and priority to under 35 U.S.C. §

119(e) of U.S. Provisional Application No. 62/432,876 titled "System and Method of Personalized

Navigation Inside a Business Enterprise," filed on December 12, 2016, the entireties of which patent

applications and provisional applications are herein incorporated by reference for all purposes.

FIELD OF THE INVENTION The invention relates generally to systems and methods for providing personalized navigation for

people while inside certain business enterprises.

BACKGROUND A common complaint among shoppers is that they are often frustrated by not knowing where certain

items are located within the store. They wander about inefficiently through the aisles searching for items

on their shopping list, often retracing steps, taking the long path in their quest of the desired items.

SUMMARY According to one embodiment, the invention relates to a system for tracking locations of individuals

in a building. The system comprises at least one radiofrequency (RF) node disposed near an entrance to

the building. The at least one RF node has an RF receiver to receive RF signals from RF-transmitting

devices near the entrance to the building. At least one optical device disposed near the entrance to the

building, the at least one optical device capturing an image of a plurality of persons while the plurality of

persons is near the entrance to the building. A controller is in communication with the at least one RF

node to obtain therefrom information associated with the RF signals received by the RF receiver of that at

least one RF node and in communication with the at least one optical device to obtain therefrom the

captured image. The controller is configured to determine an identity of each RF-transmitting device and

an angular position of that RF-transmitting device with respect to each RF node of the at least one RF

node based on the information associated with the RF signals obtained by the controller from that RF

node of the at least one RF node. The controller is further configured to detect an orientation marker and

a plurality of humans in the image obtained by the controller from the at least one optical device and to

assign the identity of each RF-transmitting device to one of the plurality of humans detected in the image

WO wo 2021/080933 PCT/US2020/056405

based on a position of the orientation marker in the image relative to each human detected in the image

and on the determined angular position of that RF-transmitting device with respect to each RF node of the

at least one RF node. Thereby, each individual is identified who is to be located optically as that

individual moves throughout the building.

In one example embodiment, the at least one RF node is configured to determine a relative signal

strength indicator (RSSI) value for the RF signals received from each RF-transmitting device. The

information obtained by the controller from the at least one RF node includes the RSSI values. The

controller is configured to estimate a distance of each RF-transmitting device from each RF node based

on the RSSI values for the RF signals received by that RF node from that RF-transmitting device and to

use the estimated distance of each RF-transmitting device from each RF node when assigning the identity

of each RF-transmitting device to one of the plurality of humans detected in the image.

In another example embodiment, the controller is further configured to detect in the image a mobile

phone held by one of the plurality of humans detected in the image and to assign the identity of each RF-

transmitting device to one of the plurality of humans detected in the captured image based on the which of

the plurality of humans detected in the image is holding the detected mobile phone.

In another example embodiment, the controller is further configured to arrange, for each RF node, the

RF-transmitting mobile devices in an angular order based on the angular positions of the RF-transmitting

mobile devices from that RF node, arrange the humans captured in the image into an angular order based

on relative positions of the plurality of humans detected in the captured image, and compare the angular

order of humans detected in the captured image with the angular order of the RF-transmitting mobile

devices when assigning the identity of each RF-transmitting mobile device to one of the plurality of

humans detected in the captured image. The controller may be further configured to arrange the angular

order of the plurality of humans in the captured image to be in respect to the at least one RF node before

the angular order of humans detected in the image is compared with the angular order of the RF-

transmitting mobile devices.

In yet another example embodiment, the system further comprises a plurality of optical devices

disposed throughout the building, and wherein the controller is further configured to optically track each

identified individual through the building based on detecting that identified individual in images received

from at least some of the plurality of optical devices over time.

In yet another example embodiment, the information carried by RF signals transmitted by a given RF-

transmitting mobile device includes a shopping list. The controller may be further configured to

determine a route through the building based on items on the shopping list and transmit the route to the

given RF-transmitting mobile device for display on a screen of the given RF-transmitting mobile device.

WO wo 2021/080933 PCT/US2020/056405

As another example, the system further comprises a plurality of optical devices disposed throughout

the building, and the controller is further configured to optically track each identified individual through

the building based on detecting that identified individual in images received from at least some of the

plurality of optical devices over time.

In other examples, the information carried by RF signals transmitted by a given RF-transmitting

mobile device includes a shopping list. The controller may be further configured to determine a route

through the building based on items on the shopping list and transmit the route to the given RF-

transmitting mobile device for display on a screen of the given RF-transmitting mobile device. The at

least one RF node may comprise two or more RF nodes.

In another example, the controller is further configured to compute a location for each RF-

transmitting mobile device based on the angular positions of the RF-transmitting mobile devices received

from the at least one RF node, to arrange the RF-transmitting mobile devices in a depth order based on the

computed locations of the RF-transmitting mobile devices with respect to the at least one RF node, to

arrange the humans captured in the image into a depth order based on relative positions of the plurality of

humans detected in the captured image, and to assign the identity of each RF-transmitting mobile device

to one of the plurality of humans detected in the captured image by comparing the depth order of humans

detected in the image with the depth order of the RF-transmitting mobile devices.

According to another embodiment, the invention relates to a method for tracking locations of

individuals in a building. The method comprises receiving, by at least one radiofrequency (RF) node

disposed near an entrance to the building, RF signals from RF-transmitting mobile devices carried by a

plurality of persons near the entrance to the building. An image is captured of the plurality of persons

while the plurality of persons is near the entrance to the building. An identity of each RF-transmitting

mobile device and an angular position of each RF-transmitting mobile device from each RF node are

determined based on information associated with the RF signals received by the at least one RF node. A

plurality of humans and an orientation marker are detected in the captured image. The identity of each

RF-transmitting mobile device is assigned to one of the plurality of humans detected in the captured

image based on a position of the orientation marker in the image relative to each human detected in the

image and on the determined angular position of each RF-transmitting mobile device with respect to each

RF node, thereby identifying each individual who is to be located optically as that individual moves

throughout the building.

In one example embodiment, a relative signal strength indicator (RSSI) value is determined for the

RF signals received from each RF-transmitting device, an estimated distance of each RF-transmitting

device from each RF node is computed based on the RSSI values for the RF signals received by that RF

node from that RF-transmitting device, and the estimated distance of each RF-transmitting device from

WO wo 2021/080933 PCT/US2020/056405

each RF node is used when assigning the identity of each RF-transmitting device to one of the plurality of

humans detected in the image.

In one example embodiment, a mobile phone held by one of the plurality of humans is detected in the

image, and assigning the identity of each RF-transmitting device to one of the plurality of humans

detected in the captured image is based on the which human detected in the image is holding the detected

mobile mobile phone. phone.

In another example embodiment, the RF-transmitting mobile devices are arranged in an angular order

based on the angular positions of the RF-transmitting mobile devices with respect to the at least one RF

node, and the humans captured in the image are arranged into an angular order based on relative positions

of the plurality of humans detected in the captured image, and assigning the identity of each RF-

transmitting mobile device to one of the plurality of humans detected in the captured image includes

comparing the angular order of humans detected in the image with the angular order of the RF-

transmitting mobile devices. The angular order of humans in the captured image may be rearranged to be

in respect to the at least one RF node before the angular order of humans detected in the captured image is

compared with the angular order of the RF-transmitting mobile devices.

In another example embodiment, a location of each identified individual is optically tracked through

the building by comparing successive images that capture that identified individual.

In still another example embodiment, the information associated with the RF signals transmitted by a

given RF-transmitting mobile device includes a shopping list. A route through the building may be

determined based on items on the shopping list, and the route may be transmitted to the given RF-

transmitting mobile device for display on a screen of the given RF-transmitting mobile device.

In yet another example embodiment, radio signals carrying a current location of a given identified

individual within the building may are transmitted from the at least one RF node to the RF-transmitting

mobile device carried by that given identified individual.

In still another example embodiment, a location for each RF-transmitting mobile device is computed

based on the angular positions of the RF-transmitting mobile devices received from the at least one RF

node. The RF-transmitting mobile devices are arranged in a depth order based on the computed locations

of the RF-transmitting mobile devices with respect to the at least one RF node. The humans detected in

the captured image are arranged into a depth order based on relative positions of the humans detected in

the captured image. Assigning the identity of each RF-transmitting mobile device to one of the plurality

of humans detected in the captured image includes comparing the depth order of humans detected in the

image with the depth order of the RF-transmitting mobile devices.

According to another embodiment, the invention relates to a system for tracking locations of

individuals in a building. The system comprises at least one radiofrequency (RF) node disposed near an

WO wo 2021/080933 PCT/US2020/056405

entrance to the building. The at least one RF node has an RF receiver channel to receive RF signals from

a plurality of RF-transmitting devices near the entrance to the building. The system further comprises at

least one camera disposed near the entrance to the building. The at least one camera captures an image of

a plurality of persons while the plurality of persons is near the entrance to the building. A controller of

the system is in communication with the at least one RF node to obtain therefrom information associated

with the RF signals received by the RF receiver channel of that at least one RF node and in

communication with the at least one camera to obtain therefrom the captured image. The controller is

configured to determine an identity of each RF-transmitting device and a position of that RF-transmitting

device relative to each RF node of the at least one RF node based on the information associated with the

RF signals obtained by the controller from that RF node of the at least one RF node, to detect a plurality

of humans in the image obtained by the controller from the at least one camera, to detect an orientation

marker in the image, and to assign the identity of each RF-transmitting device to one of the plurality of

humans detected in the image based on a position of the orientation marker in the image relative to each

human detected in the image and on the determined position of that RF-transmitting device relative to

each RF node of the at least one RF node.

Still other aspects, embodiments, and advantages of these exemplary aspects and embodiments are

discussed in detail below. Embodiments disclosed herein may be combined with other embodiments in

any manner consistent with at least one of the principles disclosed herein, and references to "an example",

"an embodiment," "some embodiments," "an alternate embodiment," "various embodiments," "one

embodiment" or the like are not necessarily mutually exclusive and are intended to indicate that a

particular feature, structure, or characteristic described may be included in at least one embodiment. The

appearances of such terms herein are not necessarily all referring to the same embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS Various aspects of at least one embodiment are discussed below with reference to the accompanying

figures, which are not intended to be drawn to scale. The figures are included to provide illustration and a

further understanding of the various aspects and embodiments and are incorporated in and constitute a

part of this specification but are not intended as a definition of the limits of the invention. In the figures,

each identical or nearly identical component that is illustrated in various figures is represented by a like

numeral. For purposes of clarity, not every component may be labeled in every figure. In the figures:

FIG. 1 is a block diagram of an example of a personalized navigation system according to certain

aspects of the present invention;

FIG. 2 is a diagram of an example a floor plan of a business enterprise configured with a personalized

navigation system according to certain aspects of the present invention;

FIG. 3 is an example display of the floor plan of the business enterprise as it may appear on the screen

of the mobile device carried by the shopper according to certain aspects of the present invention; and

FIG. 4 is a diagram illustrating an example of a system including a shelf camera and an aisle camera

tracking a shopper in front of a retail shelf within the business enterprise, according to aspects of the

present invention.

FIG. 5 is a diagram illustrating an example of a personalized navigation system configured to

distinguish among multiple persons disposed at the entrance of the business enterprise.

FIG. 6 is a flow diagram of an embodiment of a process for distinguishing among multiple persons

located at the entrance of the business enterprise.

FIG. 7 is a flow diagram of another embodiment of a process for distinguishing among multiple

persons located at the entrance of the business enterprise.

DETAILED DESCRIPTION Personalized navigation systems according to certain aspects and embodiments use a combination of

radiofrequency (RF) technology and optical imaging technology to identify and track persons at business

enterprises, and software to provide the persons with individualized information and an individualized

navigation experience within the business enterprise. As discussed in more detail below, a person can

provide the personalized navigation system with information, such as a shopping list or service request,

for example, and receive personalized navigation or other information in response to aid the person in

efficiently completing their objective(s) at the business enterprise. In this manner, the person's experience

at the business enterprise and can be improved.

According to one embodiment, a personalized navigation system uses RF technology to initially

identify a shopper who approaches an entrance to the business enterprise and uses optical technology to

detect and track movement of the shopper after the shopper arrives at and enters the business enterprise.

To cooperate with the navigation system, the shopper carries a mobile device (e.g., a smartphone or smart

watch) with RF transmitting and RF receiving capability. In certain embodiments, the mobile device runs

certain application software that transmits RF signals containing an identity of the shopper and the

shopper's shopping list. The shopper can acquire the application software and download it to the mobile

device from an "app store". Many business enterprises are currently equipped with RF transmitters, RF

receivers, and video cameras, and advantageously, the navigation systems described herein do not require

any hardware modifications to this existing RF and video equipment.

FIG. 1 shows an example of a personalized navigation system 100 configured to operate in a business

enterprise 110 with at least one RF node 120. Any number of RF nodes, designated 120-1 to 120-n (n

being a positive integer number), may be included. Each RF node 120-1 to 120-n includes a

corresponding RF receiver antenna 130-1, 130-n (generally, 130). Optionally, each RF node 120 can also

WO wo 2021/080933 PCT/US2020/056405 PCT/US2020/056405

have an RF transmit antenna (not shown). Each RF node 120 can be placed at or near an entrance to the

business enterprise 110. Examples of the business enterprise 110 include, but are not limited to, grocery

stores, supermarkets, department stores, hardware stores, warehouses, and retail stores. In general, the

location of the RF node 120 near the entrance of the business enterprise 110 facilitates detection of an RF-

transmitting mobile device 140 associated with a person approaching the entrance, provided that the

mobile device 140 is running application software that provides personalized navigation of the business

enterprise, as described herein. The mobile device 140 engages in communication with one or more of RF

nodes 120 using a wireless communication technology, such as Bluetooth®, Wi-Fi, or Bluetooth, Wi-Fi, or Near Near Field Field

Communication (NFC). If the business enterprise 110 has multiple entrances, with at least one RF node

120 disposed near each entrance, the personalized navigation system 100 can be aware of which entrance

the person is entering based on which RF node 120 detects the mobile device 140.

During typical operation, a person with the mobile device 140 approaches an entrance to the business

enterprise (i.e., a building) 110. The mobile device 140 runs a personalized navigation app and transmits

RF signals. In certain examples the RF signals carry an identifier associated with the person by which an

operator of the business enterprise 110 knows the person. For example, the identifier may include the

person's name, a telephone number, a rewards program number connected with the business enterprise, or

other identifying information. The RF signals may also carry the person's shopping list identifying those

items that the person wishes to find upon visiting the business enterprise 110. Typically, the person may

prepare this shopping list before visiting the business enterprise 110; however, the shopping list can be

constructed or edited at any time before or after the person arrives at the business enterprise 110.

When the person comes into range of an RF receiver antenna 130, the mobile device 140 establishes

communications with the RF node 120. In particular, in certain examples the mobile device 140 may pass

to the RF node 120 the identifier and shopping list. The mobile device may also or alternatively pass other

data to the RF node 120, such as a set of instructions or other information for a technician or similar

repair staff performing certain services at the business enterprise 110, for example. In certain examples,

the RF node 120 forwards the identifier and shopping list or other data to a computer processing unit (also

called a controller) 150, which can use this identifier to access a database 160 where information relating

to the person associated with the identifier is kept. This information can include records of prior visits to

the business enterprise 110 by the person, for example. Although the computer processing unit 150 and

database 160 are shown in FIG. 1 to be within the business enterprise 110, they may reside instead at

other sites, such as on a third-party network of computers and servers referred to broadly as "the cloud."

FIG. 2 shows a simple example of a floor plan 200 of the business enterprise 100 with various aisles

210-1 210-1 through through 210-7 210-7 (generally, (generally, 210) 210) and and video video cameras cameras 220-1 220-1 through through 220-n 220-n (generally (generally 220) 220) placed placed

throughout the store, preferably to provide full coverage of the interior of the business enterprise 100,

WO wo 2021/080933 PCT/US2020/056405

wherever visitors may walk. The business enterprise 100 can also include one or more RF transmitters

230 and a networking device 240. Each video camera 220, each RF node 120, and each RF transmitter

230 is in communication with the networking device 240 by a wireless or wired communication link (not

shown). The networking device 240 is in communication with the computer processing unit 150 (shown

in FIG. 1 as discussed above), which in one embodiment resides on the cloud 250. A wired or wireless

communication link 260 connects the networking device 240 to the central processing unit 150.

As discussed above, tracking of persons within the business enterprise 110 can be accomplished using

optical technology; in particular, by capturing and processing images from the video cameras 220 located

throughout the business enterprise 110. According to one embodiment, during typical operation of the

personalized navigation system, the video cameras 220 continuously capture images within their fields of

view. At least one video camera 220 can be placed proximate each entrance of the business enterprise 110

to acquire images of persons entering the business enterprise. In some embodiments, multiple video

cameras 220 can be placed proximate each entrance in such a way as to provide a complete field of

view, or at least a functionally sufficient field of view, of the area around the entrance such that images

of all persons entering the business enterprise 110 can be acquired. As discussed above, when a person

having a mobile device 140 configured to run the application software to engage with the personalized

navigation system 110 (referred to as a tracked person) arrives at an entrance to the business enterprise

110 the RF node 120 at that entrance receives the identifier, and optionally other information (such as the

shopping list), from the mobile device 140. At the same time, the video camera(s) 220 proximate that

entrance capture images of the area around the entrance, and at least some of these images should contain

the tracked person. As discussed above, in certain examples, the computer processing unit 150 knows

which entrance a person used to enter the enterprise 110 based on which RF node 120 detected the person

and known locations of each RF node. Accordingly, the computer processing unit 150 knows which video

camera or cameras 220 are in position to capture images of the person. These video cameras 220 pass

their captured images to the networking device 240, which sends the captured images to the central

processing unit 150. The central processing unit 150 includes an image processor that performs image

processing techniques adapted to detect a person within the image and to associate the detected person

with the most recently acquired identifier and shopping list.

Techniques for processing images to identify a person within the images are known in the art, and any

such image processing techniques can be implemented by the central processing unit 150. For example,

the image processor can be adapted to examine images captured by the video camera 220-1 positioned at

the relevant entrance for a smartphone in the hand of an individual, which may indicate that the individual

is engaging with the personalized navigation system 100. Alternatively, or in conjunction, the image

processor can be adapted to examine the captured images for the head or hands of a person. Since the

WO wo 2021/080933 PCT/US2020/056405

central processing unit 150 expects the next person to fall within the field of view of the video camera

220-1 located at the entrance to be the same as the person who communicated with the RF node 120-1

located at that entrance, that detected person becomes associated with received identifier and shopping

list. Once a person has been identified in an image and associated with the received identifier, the

personalized navigation system 100 tracks and guides the person as he or she moves through the business

enterprise 110.

Tracking can be accomplished by collecting images from the various video cameras 220 located

amongst the aisles 210 and processing these images to follow the tracked person. In certain examples the

central processing unit 150 follows the movement of the tracked person as her or she moves from one

camera field of view to another, dynamically registering and updating the location of the person within

the business enterprise 110. In one example the video cameras 120 operate in parallel, with all or some of

the video cameras providing images to the central processing unit simultaneously. The images can be

merged into a map or layout of the business enterprise 110, such as shown in FIG. 2. In some examples

two or more of the video cameras 220 can have at least partially overlapping fields of view, and in other

examples different video cameras 220 are used to monitor different areas of the business enterprise 110.

The video cameras 220 may capture images with different perspectives. The central processing unit may

flatten the images by removing perspective distortion in each of them, and merges the resulting corrected

images into the map. Image stitching techniques can be used to merge the images.

In certain examples, an image stitching process first performs image alignment using algorithms that

can discover the relationships among images with varying degrees of overlap. These algorithms are suited suited

for applications such as video stabilization, summarization, and the creation of panoramic mosaics, which

can be used in the images taken from the cameras 220. After alignment is complete, image-stitching

algorithms take the estimates produced by such algorithms and blend the images in a seamless manner,

while taking care of potential problems, such as blurring or ghosting caused by parallax and scene

movement as well as varying image exposures inside the business enterprise 110. Various image stitching

algorithms and processes are known in the art, and any such image processing techniques can be

implemented by the central processing unit 150.

A handoff can be made when a tracked person moves from one viewpoint to another or is seen by one

camera 220 and not the others. These handoffs may be made using the images running in parallel on the

central processing unit 150, with the tracked person's location and movement determined by the central

processing unit using whichever camera 220 has the best view of the tracked person.

In certain examples, the video cameras 220 can include depth sensors. In such examples, the image

stitching operation can be omitted, and each camera stream data is processed independently for change,

person detection and recognition. Then, the resulting "areas of interest" are converted to individual point

WO wo 2021/080933 PCT/US2020/056405

clouds (described further below) and transformed in to a single common coordinate system. The

translation and rotation transformations used for this process are based on the position and orientation of

the the video video cameras cameras (and (and their their associated associated depth depth sensors) sensors) in in relation relation to to one one another. another. In In one one example, example, one one

camera is picked as the main sensor and all other camera data is transformed into the main coordinate

system, achieving the same end result as the image stitching procedure, namely, unification of the actual

location of the tracked person among sensors.

In some examples the central processing unit 150 may use known information about the floor plan of

the business enterprise to assist with identifying and tracking persons based on the images acquired from

the video cameras 220. For example, the central processing unit can use known shapes and positions of

shelving along the aisles 210 to provide reference points. At times, a tracked person may be occluded in a

camera's field of view, for example, by another person, equipment, or shelving. The personalized

navigation system 100 can be configured to store the tracked person's prior-determined position and

compare multiple image frames to re-locate the tracked person after a temporary occlusion. As discussed

further below, the personalized navigation system 100 can be configured to provide a proposed route for

the tracked person through the business enterprise 110, and therefore the central processing unit can use a

predicted future location of the tracked person to relocate the person after a temporary occlusion.

According According to to certain certain embodiments, embodiments, the the central central processing processing unit unit 150 150 can can run run an an image-processing image-processing

process, optionally supplemented with depth information, to track a person as discussed above. A two-

dimensional (2D) optical image capture device (i.e., a video camera 220) with a single aperture is capable

of capturing 2D image information on a plane (film, CCD, etc.). To acquire three-dimensional (3D)

information typically requires acquisition of additional data. Three-dimensional data can be acquired

using multiple video cameras 220 or by combining one or more video cameras with one or more depth

sensors. The video cameras 220 can utilize visible light, infrared light, or other optical wavelength ranges.

Depth sensors can be based on infrared, laser or other wavelength emitters that transmit light to an object.

Depth sensors typically determine the distance to the object from which the light that is reflected or

backscattered. Alternatively, depth sensors can utilize acoustic signals to determine distance. In one

embodiment, depth sensing is integrated into the video cameras 220.

Image frames are acquired from the video cameras 220. A video camera system with depth sensing

capability typically outputs video (e.g., RGB, CYMG) and depth field information. Video may optionally

be encoded to a well-known format, such as MPEG. The optical and depth information are stitched

together. Open libraries such as OpenCV or OpenNI (used to capture depth images) enable the optical and

depth information to be stitched together. Alternatively, a user of the personalized navigation system 100

may develop customized software for generating 3D information or object data generated by optical

images and depth sensors.

WO wo 2021/080933 PCT/US2020/056405

An initial calibration can be performed over multiple image frames to determine background

information both for 2D optical images and the depth sensing. During the calibration, any motion (e.g.,

people) is extracted or ignored during background extraction until stable background optical (RGB) and

depth information can be stored, for example, in the database 160. Calibration may be performed

periodically or may be initiated by the personalized navigation system 100, for example, if errors are

detected.

After calibration is complete, the resulting spatial filter masks can be used to extract an "area of

interest" for each video camera 220. For example, for a video camera located near an entrance to the

business enterprise 110, the area of interest may correspond to the area between the background and a a

foreground (area where a person is expected to be), SO so that everything that is not walls, doors, or other

infrastructure (for background) and also not a detected person, is ignored. This ignoring of the

background and foreground focuses on data within the depth threshold of the area of interest being

monitored. Alternatively, the "area of interest" can include a different part of the scene, for example, the

foreground in order to see where the person is in later recognition steps and can be expanded or

contracted as system requirements dictate. In general, the area of interest applies to any cut-out of a scene

that is to be the focus within which to perform person tracking.

According to certain embodiments, multiple image frames (e.g., N-1 and N) are obtained and

compared, and in certain examples the image frames can include depth information in addition to RGB

(color) data, as discussed above. Image and depth information can be filtered for noise and then processed

to determine if a difference between two frames exists. This can be done with edge detection, threshold

and difference algorithms, or other image processing techniques. In certain examples, information from

the depth sensors is also processed to compare image frames. The system can use changes between image

frames, in particular, changes in the position or orientation of a detected person, to track the movement of

the person. In some embodiments, change detection can be limited to the area of interest to increase

processing speed.

In one embodiment, when the area of interest is determined, a "point cloud" is generated using the

video camera's extrinsic and intrinsic parameters through algorithms for "2D to 3D" data representation

conversion preformed on the RGB and/or depth images obtained and processed through OpenNI and

OpenCV. In one embodiment, the Point Cloud Library may beused. The object shape and location

information generated from the Point Cloud Library are used to identify and track a person in three

dimensions using edge detection, color detection, object recognition and/or other algorithms for

determining the presence of a person within the scene. If objectinformation is in the shape of a

human, for example, then the process continues to track the person. However, if the size, shape or

other appearance information indicates that the object is not a person, subsequent image frames can

WO wo 2021/080933 PCT/US2020/056405

be analyzed until a person is detected. In some examples, images captured by the video cameras 220

may include more than one person. Accordingly, the process may compare expected features and/or

appearance attributes of the tracked person with persons detected in the image frames to continue to

track the correct person.

As discussed above, the central processing unit 150 can merge the acquired images from the video

cameras 220 into a map to be able to track identified persons as they moved through the business

enterprise. In certain examples, the application software running on the mobile device 140 can be

configured to display the map or a similar map view or virtual layout of the floor plan of the business

enterprise 110, such that the tracked person can view their location within the business enterprise. The

central processing unit 150 can send commands to the RF transmitters 230 - by way of the networking

device 240 - to transmit RF signals carrying the updated location of the tracked person, which can be

determined using image processing techniques as discussed above. The mobile device 140 - with its RF

receiver - receives these signals and registers the updated location of the person within the application

software, which can show the location of the person within the virtual layout of the business enterprise

110 displayed on the mobile device 140.

FIG. 3 shows an example display 300 of the floor plan of the business enterprise 110, with aisles 210,

as it may appear on the screen of the mobile device 140 (FIG. 1) carried by the tracked person. In this

example, an arrow 310 indicates the current location of the tracked person. As discussed above, the

personalized navigation system 100, and the application software running on the mobile device 140, can

be configured to guide the tracked person through the business enterprise based on the information

received along with the identifier. For example, where a shopping list is provided, the personalized

navigation system 100 can access information from the business enterprise 110 that identifies where in

the aisles 210 the items on the shopping list are located. In one example, the central processing unit 150

(FIG. 1, FIG. 2) examines the shopping list acquired by the initial RF communications with the mobile

device 140 and accesses a database that stores the locations of all items within the business enterprise

110. The central processing unit can be further configured to match descriptions of items on the shopping

list with product SKUs or other identifying information in the database to obtain a location of each item

within the business enterprise 110. The database can be part of the personalized navigation system 100,

obtained from an operator of the business enterprise 110 and regularly updated, or the database can be

maintained by the operator of the business enterprise and accessed by the personalized navigation system

(e.g., by the central processing unit 150) via the cloud 250.

In FIG. 3, a directional dashed line 320 identifies a proposed route through the business enterprise

110. After the central processing unit 150 has examined the shopping list and identified where the desired

items 330 on the shopping list are located within the business enterprise 110, a route 320 is proposed

WO wo 2021/080933 PCT/US2020/056405

through the business enterprise for efficiently obtaining the items. In one embodiment, the route is based

on the shortest distance to acquire all the items. Another route can be to direct the person to non-

perishable or unrefrigerated items initially, and leaving perishable or refrigerated items for the end of the

route. The central processing unit 150 sends the map and item location information to the mobile device

140 via the RF transmitters 230 (FIG. 2). The application software executing on the mobile device 140

displays the route 320 based on the items on the tracked person's shopping list. To supplement the

positioning information and calibration provided by the video cameras, the mobile device 140 can have

inertial sensors. Techniques for using inertial sensing to enhance positioning information are described in

U.S. Pat. No. 8,957,812, issued Feb. 17, 2015, titled "Position Tracking System and Method using Radio

Signals and Inertial Sensing," the entirety of which U.S. patent is incorporated by reference herein. In

other examples, additional RF nodes (similar to the RF nodes 120) with RF transmitting/receiving

capability can be used to supplement the positioning information provided by the video cameras and

enhance the tracking capability of the personalized navigation system 100.

In addition to identifying the desired items 330, the central processing unit 150 can notify the person

of an item that may be of interest, as the person's current location approaches the location of that item,

even if that item is not on the shopping list. Such an advertisement may be based on the shopping history

of the person, for example. As discussed above, in certain examples the information provided from the

mobile device 140 to the RF node 120 (and therefore to the central processing unit 150) can include a

service request. Accordingly, in such examples instead of or in addition to displaying the locations of the

desired items 330, the location of the service desk or other relevant information can be displayed on the

map, and the route 320 can be configured to guide the person to that location.

Referring to FIG. 4, another aspect of this disclosure is a product tracking and checkout system 400

that can keep track of products taken by customers as they shop in the business enterprise 110, integrating

with the person's shopping list and also eliminating the need for shoppers to wait in cashier lines. This

system 400 enables customers 410 to any or all of take products 412, 414 off of the shelves 416, put them

in their cart, register the products that are taken and who took the products, automatically pay for the

products and leave the store with the products purchases automatically logged by the retail system

without having to go through a checkout system. Aspects of this system could improve the speed and

experience of the shopping process for customers and reduce the cost of cashiers for the retailers.

For example, retailer's costs can be reduced with a system in place that automatically keeps track of

inventory on shelves and/or what is taken off these shelves by customers to automatically keep track of

what customers take from stores and to manage inventory on shelves. The ability to track inventory and

what products customers remove from shelves can improve the cost basis for retailers by eliminating the

need for cashiers or extra staff to constantly go to shelves to inspect what items need replacing and re-

WO wo 2021/080933 PCT/US2020/056405 PCT/US2020/056405

stocking. In addition, the system can update the shopping list received from a tracked person based on

items founds and taken by the tracked person and update the displayed route 320 based on the progress

made by the tracked person.

It is appreciated that variations of image processing can be used for shelf and product tracking. One

aspect of the system 400 includes a product recognition camera 420 facing the shelves to view what

products are on the shelf and what products are removed by customers. The system may have one or

more first shelf facing cameras 420 with a view angle 422 focused on the shelf to see what products are

there and what products are removed. However, there may also be situations where one or more shelf

focused product recognition cameras 420 may not be sufficient as there may be times that two people

reach for products in the same area, potentially even cross arms while reaching for their individual

products, and/or possibly blocking the view of the one or more product tracking cameras 420 when

reaching and grabbing the product on the shelf.

Thus, an embodiment of the system incorporates an additional outward looking (aisle facing) camera

430. Thus, an aspect of this embodiment of the system includes at least two cameras on an integrated arm

mount 440. At least one first product tracking camera 420 is oriented to focus on the products on the shelf

and at least one second aisle tracking camera 430 is oriented to focus on the aisle and the customers doing

the shopping. Both cameras can be a video camera, and both cameras can be a video camera of the video

cameras 220-1 through 220-n (generally 220) placed throughout the business enterprise, as discussed

above, to provide full coverage of the interior of the business enterprise 110. Thus, in this embodiment, at

least one camera 420 ("shelf tracking camera") may be used primarily for product recognition on the shelf

and at least one additional camera 430 ("aisle tracking camera") may be used primarily for customer

skeletal tracking to confirm where that customer is reaching.

Some advantages of this embodiment of the system 400 are that by using at least one aisle tracking

camera 430 to focus into the aisle and on the shopper, the system can eliminate any occlusion issues from

the shopper standing in front of the shelf-facing camera 420 or any of the other video cameras 220. In

addition, the combination of the first shelf facing camera 420 and second aisle facing cameras 430 can

also prevent the cameras from confusing what item was taken should two shoppers reach in the same area

for products and either cross arms or occlude the camera potentially causing the system to charge the

wrong customer for the item taken.

Aspects of this embodiment of the system 400 having the dual cameras can include accomplishing

multiple functions from at least one first camera 420 and at least one second camera 430 including

shopper registration, shopper movement tracking, and product identification on retail shelving, inventory

tracking, and monitoring the amount of products on the shelving.

WO wo 2021/080933 PCT/US2020/056405

Typically, multiple persons may be entering and/or leaving the entrance of an enterprise at a given

moment. Further, more than one of them may be operating an RF-transmitting mobile device at that

moment, and an RF node of the personalized navigation system, located near the entrance, may be in

communication with a plurality of them. In addition, others passing through or near the entrance at that

moment may not be carrying a mobile device or may have their mobile devices are turned off.

Personalized navigation systems described herein are configured to distinguish between people

participating in the personalized navigation and between participants and non-participants. Unlike

participants, non-participants are not interacting through RF communications with the personalized

navigation system, but like participants, non-participants may still be optically tracked through the

enterprise.

It is appreciated that aspects of the cameras and the system can also include color sensing,

comparison and depth sensing, which can be accomplished for example with infrared sensing. The first

shelf tracking camera 420 can use either or both of color and depth sensing to register the products'

position and to recognize the actual products on the shelf. The second aisle tracking camera 430 can use

depth sensing to perform skeletal tracking to confirm where that customer is reaching. Confirmation of

which customer selected which product on the shelf is achieved by the shelf camera 420 providing

product identification and removal (from shelf) detection and by the position of the person's arm in

relation to the item and, upon removal, the item actually in the hand of the customer provided by the aisle

camera 430. The fusing of the functions of these two cameras provides a much more robust method for

confirming what was taken off the shelf and by which shopper.

FIG. 5 shows an embodiment of a personalized navigation system 100 configured to distinguish

among individuals who pass through an entrance of the business enterprise 110 concurrently or next to

each other. Near the entrance are located one or more RF nodes 120-1, 120-2 (generally, 120) and one or

more cameras 220 (only one shown to simplify the drawing). Each RF node 120 has a corresponding RF

receiver antenna 130-1, 130-n (generally, 130) and an RF transmitter (not shown). In one embodiment,

each RF node 120 has an antenna array comprised of two or more antennae (or antenna elements) with

known spatial separation(s). Timing differences between the RF signal received at each of the at least

receiver antennae are used by the RF node to estimate angle of arrival of the RF signal. An example of a

multiplexing technique for multiplexing antenna of an antenna array is described in U.S. Pat. No.

8,749,433, issued June 10, 2014, titled "Multiplexing Receiver System", the entirety of which is

incorporated by reference herein.

Each RF node 120 and each camera 220 is in a fixed location and in communication with the

controller 150, which knows the relative locations of each such electronic equipment (i.e., the pan and tilt

angles and distance between cameras 220, between RF nodes 120, between each camera 220 and each RF

WO wo 2021/080933 PCT/US2020/056405

node). Each camera 220 has a field of view 510 that covers an area near the enterprise entrance. As

previously described, the controller 150 is configured to detect a person within an image captured by the

camera(s) 220 and associate the detected person with an identifier that was transmitted by the person's

mobile device and recently received by the RF node.

As illustrated in FIG. 5, multiple persons 500-1, 500-2, 500-3 (generally, 500) are concurrently near

the enterprise entrance (whether entering or leaving). Individuals 500-1 and 500-2 are carrying RF-

transmitting mobile devices 140-1, 140-2, respectively. Each mobile device 140-1, 140-2 is running

application software that provides personalized navigation of the business enterprise, as previously

described, and is in communication with one or more of RF nodes 120-1, 120-2 using a wireless

communication technology, such as Bluetooth® Bluetooth, Wi-Fi, or Near Field Communication (NFC). This

application software may be running in the background, that is, the individual does not need to

deliberately activate the application to handshake with an RF node; the application is in the background

broadcasting or listening for join requests. By operating the application in the background, the mobile

device 140 automatically connects with each RF node 120 upon coming into wireless communication

range of it. To synchronize communications, each mobile device 140 handshakes with each RF node 120

with which it is in communication. In this example, the individual 500-3 is not carrying or not running a

mobile device that is in RF communication with an RF node 120. The camera(s) 220 capture images that

include all these individuals 500-1, 500-2, 500-3; that is, the controller may detect all the individuals 500-

1, 500-2, 500-3 in a single captured image. As described herein, the controller 150 is configured to

determine which received identifier corresponds to which person detected in the captured image, and

which detected person does not have or is not using an RF-transmitting mobile device to communicate

with the RF node 120.

FIG. 6 shows an embodiment of a process 600 for distinguishing among multiple persons located at

the entrance of the business enterprise, at least one of which is carrying and operating an RF-transmitting

mobile device. For purposes of illustration, consider that the individuals 500-2 and 500-3 pass through

the entrance side-by-side and continue further indoors, whereas individual 500-1 pauses just within the

entrance. The mobile devices 140-1, 140-2 of the individuals 500-1, 500-2, respectively, communicate

(step 602) with the RF nodes 120-1, 120-2, sending them their identifiers, and, at about the same time or

momentarily after, the camera 220 captures (step 604) an image with all three individuals in it.

Each RF node 120 derives (step 606) a received signal strength indicator (RSSI) for the radio signal

received from each mobile device 140 and associates that RSSI measurement with the corresponding

identifier. In general, the closer the mobile device is to the RF node 120, the stronger is the received

signal. At step 608, the RF nodes send the RSSI measurements and associated identifiers, and the camera

sends the captured image, to the controller. The controller associates the captured image with the RSSI

WO wo 2021/080933 PCT/US2020/056405

measurements and associated identifiers because of the near synchronicity of when the camera captured

the image and the RF nodes received the RF signals. The controller also knows the relative locations of

the RF nodes to each other and to the camera and tracks which RF node sent which RSSI values.

At step 610, the controller detects the three individuals in the image and establishes a depth order of

the detected individuals with respect to the camera. This determination of depth order may use depth

information obtained by the camera or by a depth sensor calibrated to the camera. The controller can

rearrange the depth order of humans in the image from the perspective of or in respect to each RF node,

thus facilitating a comparison of that RF node's distance order, which is based on its computed RSSI

values, with the depth order derived from the image captured by the camera.

In this example, the controller receives the RSSIs and identifier information for only two mobile

devices. The task of the controller is to match each identifier with the proper mobile device, that is, with

the appropriate one of the individuals detected in the image.

From the received signal strength indicators, the controller can compute approximate distances to the

RF-transmitting sources (i.e., the mobile devices). The greater the number of RF nodes that provide RSSI

values, the more precisely the controller can estimate the location of each mobile device with respect to

those RF nodes (e.g., through triangulation or multilateration). Alternatively, the controller can deduce a

closest-to-farthest order for the mobile devices based on their corresponding RSSI values, without

computing relative location or distance estimates. The controller arranges (step 612) the RF-transmitting

mobile devices in distance order with respect to the RF node 120-1 and with respect to the RF node 120-

2.

Using the known locations of the RF nodes 120-1, 120-2 and the camera 220 and camera angles, the

controller compares (step 614) the depth order of the persons detected in the image with the distance

order(s) determined for the mobile devices based on the signal strength indicators provided by the RF

nodes 120. Based on the comparison, the controller matches (step 616) identifiers with the detected

individuals.

For example, based on the example locations of the individuals 500 in FIG. 5, the RF node 120-1

determines that the signal strength of the mobile device 140-1 is greater than that of the mobile device

140-2, whereas the RF node 120-2 finds the opposite is the case. For example, the controller may

compute from the RSSI values that the mobile device 140-1 is approximately 6 feet away from the RF

node 120-1 and approximately 9 feet away from the RF node 120-2 and that the mobile device 140-2 is

approximately 11 feet away from the RF node 120-1 and approximately 4 feet away from the RF nodes

120-2. Accordingly, from the computed estimated distances, a distance order for mobile devices with

respect to each RF node emerges; the controller 150 thus finds that the mobile device 140-1 is closer to

the RF node 120-1 than is the mobile device 140-2.

17

WO wo 2021/080933 PCT/US2020/056405

Further, in the captured image, the controller detects three people and can determine a relative

distance from the camera of each person. Depth information associated with the image can help with this

relative distance determination. Because the locations of the RF nodes 120-1, 120-2 relative to each

camera and to each other are known, the controller can then provisionally assign identifiers to individuals

(i.e., presumably, the mobile device carrying persons). The RSSI values, any signal-strength-based

distance calculations, distance orders for mobile devices, and the depth information associated with

humans detected in the image, guide these assignments. Other information can further enhance the

decision-making process. For instance, the entrance may be sixteen feet in width, which provides an

approximate upper bound for possible distances of people from the camera and from the RF nodes and

helps establish the locations of persons passing through entrance. The controller can also consider

detected gaps in the image between adjacent individuals. Other indicators, such as orientation markers

deliberately placed and calibrated within the camera(s)' field of view, such as on a wall or shelf that is

above or on either side of the entrance, can assist the determination. Fixed features inherent to the

enterprise, examples of which include but are not limited to a pillar, beam, airduct, ceiling tile, pipe,

window, door, shelf, and counter, can serve as orientation markers. As another example, the RF node(s)

can fall within the camera's field of view at the entrance, and the camera can thus capture the RF node in

the same image that captures the plurality of individuals 500. During the image processing, the controller

can then use the RF node's location in the image as an orientation point to help determine the location

each detected human relative to the RF node.

Further image processing of images captured by the camera can also be employed to resolve any

ambiguity. For instance, the controller can attempt to detect mobile devices in the hands of the detected

individuals. Detecting a cell phone in one's possession, however, is not conclusive, as the phone may be

turned off or it may not be communicating with the personalized navigation system. In addition, not

detecting the cellphone in one's possession is likewise inconclusive, for an operating RF-transmitting

mobile device phone may be concealed in a person's handbag or pocket. Such information,

notwithstanding its inconclusive nature, can be used to increase the confidence level of a correct matching

of mobile phones to humans detected in an image.

Based on all its available information, the controller can assign, for example, the mobile device

closest to the RF node 120-1 to the person 500-1 in the image closest to the camera, and the mobile

device farther from the RF node 120-1 to the person 500-2 next in the image closest to camera. If

ambiguity remains (i.e., a confidence level for correct matching does not exceed a threshold), for

example, because not all individuals are carrying RF-transmitting mobile devices, such as individual 500-

3, or because individuals are walking close to each other, the controller may collect (step 618) additional

RSSI information from multiple RF nodes 120-1, 120-2 to determine 2-dimensional range information to

WO wo 2021/080933 PCT/US2020/056405

each of the mobile phones. Again, the greater the number of RF nodes, the more accurate this range

determination. This approximate directional information can be used to calibrate the location of the

smartphone-toting individuals within the captured images and distinguish them from an individual who is

not carrying a smartphone.

The RSSI measurements from a single RF node (e.g., 120-1) may be enough for the controller to

determine relative distance among the mobile devices from that RF node and find the proper match to a

human detected in the image. Because the distances from the camera to the RF node, from the RF Node

to each mobile device based on RSSI values, and from the camera to each detected human based on depth

information, are pre-known or computed, the controller can compute approximate locations of each

mobile device and match them to the humans detected in the image. As previously described, other visual

orientation markers and/or subsequent visual and RF tracking (via subsequent RSSI information) can aid

in the determination.

FIG. 7 shows an embodiment of another process 700 for distinguishing among multiple persons

located at the entrance of the business enterprise. At least one of the people is carrying and operating an

RF-transmitting mobile device that runs the aforementioned application software (either as a foreground

or background service) to communicate with the RF nodes 120 of the business enterprise. For purposes

of illustration, consider that the individuals 500-1, 500-2, and 500-3 (FIG. 5) are near the entrance. The

mobile devices 140-1, 140-2 of the individuals 500-1, 500-2, respectively, communicate (step 702) with

the RF nodes 120-1, 120-2, sending them their identifiers, and, at about the same time or momentarily

after, the camera 220 captures (step 704) an image with all three individuals in it.

For this process, each RF node 120 has an antenna array and estimates (step 706) an angle of arrival

for the radio signal coming from each mobile device 140 and associates that angle of arrival measurement

with the corresponding identifier. Each RF node 120 can determine the direction of a RF signal incident

on the antenna array by measuring the time difference of arrival (TDOA) of the RF signal at individual

antenna of the antenna array. An RF node can make this TDOA measurement by measuring phase

difference in the RF signal received by each antenna or antenna element in the antenna array. From these

time differences, the RF node can calculate the RF signal's angle of arrival. As used herein, this angle of

arrival corresponds to an angular position or angle line for the RF signal source, namely, the mobile

device 140. Having angular position data for a given mobile device at two (or more) RF nodes 120

enables computation of an intersection point that corresponds to the location of that mobile device.

Approaches other than TDOA for calculating angle of arrival are known in the art and may be used

without departing from the principles described herein.

At step 708, the RF nodes send the calculated angle of arrival (i.e., angular position) measurements

and their associated identifiers, and the camera sends the captured image, to the controller. The controller

WO wo 2021/080933 PCT/US2020/056405

associates the captured image with the angular position measurements and associated identifiers because

of the near synchronicity of when the camera captured the image and the RF nodes received the RF

signals. The controller also knows the relative locations of the RF nodes with respect to each other and to

the camera and tracks which RF node sent which angular position measurements.

At step 710, the controller detects the three individuals 500-1, 500-2, 500-3 in the image and

determines an angular alignment and/or depth order of the detected individuals with respect to the camera.

The angular alignment corresponds to a left-to-right or right-to-left appearance of the individuals in the

image. For example, from the perspective of the camera, individual 500-2 is between individuals 500-1

and 500-3 in the image, with individual 500-3 being on the left of individual 500-2 and individual 500-1

on the right of individual 500-2. The determination of depth order may use depth information obtained by

the camera or by a depth sensor calibrated to the camera.

The controller can rearrange the angular alignment and/or depth order of humans in the image to be

from the perspective of or in respect to each RF node. This rearrangement facilitates comparing the

angular alignment of individuals in the image with the angular alignment of the mobile devices according

to the angular positions determined by that RF node.

In this example, the controller receives the angular positions and identifier information for only two

of the three mobile devices 140-1, 140-2 (FIG. 5) captured in the image. The task of the controller is to to

match each identifier with the proper mobile device, that is, with the appropriate one of the individuals

detected in the image.

From the received or computed angular positions, the controller can compute approximate locations

of the RF-transmitting sources (i.e., the mobile devices). The greater the number of RF nodes that

provide angle of arrival or angular positions, the more precisely the controller can estimate the locations

of each mobile device with respect to those RF nodes (e.g., through intersection of angle lines from two

RF nodes or through triangulation or multilateration with angle of arrival lines from three or more RF

nodes).

Alternatively, or in addition, the controller (step 712) can deduce a left-to-right (or right-to-left)

arrangement for the mobile devices based on their corresponding angular positions, without computing

relative location or distance estimates. For example, based on the angular positions computed by the RF

nodes 120-1, 120-2, the controller determines the mobile device 140-2 to be on the left of mobile device

140-1 from the perspective of the RF node 120-1 and determines the mobile device 140-2 to be on the

right of mobile device 140-1 from the perspective of the RF node 120-2.

controller can compare (step 714) the angular alignment of the persons detected in the image with the

angular arrangement or order determined for the mobile devices based on the angular positions provided

WO wo 2021/080933 PCT/US2020/056405

by the RF nodes 120. Based on the comparison, the controller matches (step 716) identifiers with the

detected individuals. For example, the controller can assign the mobile device 140-1 on the right

according to the RF node 120-1 to the rightmost person 500-1 in the image, and the mobile device 140-2

on the left according to the RF node 120-1 to the person 500-2 left of person 500-1 in the image. This

matching can operate without ambiguity when the number of mobile phones 140 matches the number of

persons detected in the image (e.g., if individual 500-3 were not in the captured image). It can also be

performed using the angular order determined for the mobile devices based on the angular positions

provided by only one of the RF nodes 120, although using more than one RF node improves precision.

Alternatively, or in addition, the controller can compute the location of each mobile device based on

the angular positions provided by the RF nodes 120-1, 120-2 (e.g., by computing an intersection of the

two angle lines). This location has a depth component (from a reference point, e.g., the camera). The

controller can rank the mobile devices in a depth order based on the depth components of the locations

computed for the mobile devices. The controller can then match this depth order to a depth order of the

individuals in the captured image.

As an aid to the correct matching of mobile devices to individuals (or individuals to mobile devices),

particularly when individuals, such as individual 500-3, who are not using a mobile device may cause

ambiguity, the controller can collect (step 718) RSSI information from multiple RF nodes 120-1, 120-2,

to determine 2-dimensional range information to each of the mobile phones and to match individuals to

mobile devices as previously described. Thus, the controller can use the signal strength information to

supplement the matching based on angle of arrival computations. Again, the greater the number of RF

nodes, the more accurate this RSSI-based range determination. This approximate distance information

can also be used to calibrate the location of the smartphone-toting individuals within the captured images

and distinguish them from an individual who is not carrying a smartphone (e.g., individual 500-3).

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a

system, method, and computer program product. Thus, aspects of the present invention may be embodied

entirely in hardware, entirely in software (including, but not limited to, firmware, program code, resident

software, microcode), or in a combination of hardware and software. In addition, aspects of the present

invention may be in the form of a computer program product embodied in one or more computer readable

media having computer readable program code stored thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer

readable medium may be a computer readable signal medium or a computer readable storage medium.

The computer readable medium may be a non-transitory computer readable storage medium, examples of

which which include, include, but but are are not not limited limited to, to, an an electronic, electronic, magnetic, magnetic, optical, optical, electromagnetic, electromagnetic, infrared, infrared, or or

semiconductor system, apparatus, or device, or any suitable combination thereof.

21

WO wo 2021/080933 PCT/US2020/056405

As used herein, a computer readable storage medium may be any tangible medium that can contain or

store a program for use by or in connection with an instruction execution system, apparatus, device,

computer, computing system, computer system, or any programmable machine or device that inputs,

processes, and outputs instructions, commands, or data. A non-exhaustive list of specific examples of a

computer readable storage medium include an electrical connection having one or more wires, a portable

computer diskette, a floppy disk, a hard disk, a random access memory (RAM), a read-only memory

(ROM), a USB flash drive, an non-volatile RAM (NVRAM or NOVRAM), an erasable programmable

read-only memory (EPROM or Flash memory), a flash memory card, an electrically erasable

programmable read-only memory (EEPROM), an optical fiber, a portable compact disc read-only

memory (CD-ROM), a DVD-ROM, an optical storage device, a magnetic storage device, or any suitable

combination thereof.

A computer readable signal medium may include a propagated data signal with computer readable

program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated

signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any

suitable combination thereof. A computer readable signal medium may be any computer readable medium

that is not a computer readable storage medium and that can communicate, propagate, or transport a

program for use by or in connection with an instruction execution system, apparatus, or device. As used

herein, a computer readable storage medium is not a computer readable propagating signal medium or a

propagated signal.

Program code may be embodied as computer-readable instructions stored on or in a computer

readable storage medium as, for example, source code, object code, interpretive code, executable code, or

combinations thereof. Any standard or proprietary, programming or interpretive language can be used to

produce the computer-executable instructions. Examples of such languages include C, C++, Pascal,

JAVA, BASIC, Smalltalk, Visual Basic, and Visual C++.

Transmission of program code embodied on a computer readable medium can occur using any

appropriate medium including, but not limited to, wireless, wired, optical fiber cable, radio frequency

(RF), or any suitable combination thereof.

The program code may execute entirely on a user's device, such as the mobile device 140, partly on

the user's device, as a stand-alone software package, partly on the user's device and partly on a remote

computer or entirely on a remote computer or server. Any such remote computer may be connected to the

user's device through any type of network, including a local area network (LAN) or a wide area network

(WAN), or the connection may be made to an external computer (for example, through the Internet, using

an Internet Service Provider).

WO wo 2021/080933 PCT/US2020/056405

Additionally, the methods of this invention can be implemented on a special purpose computer, a

programmed microprocessor or microcontroller and peripheral integrated circuit element(s), an ASIC or

other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit such as discrete

element circuit, a programmable logic device such as PLD, PLA, FPGA, PAL, or the like. In general, any

device capable of implementing a state machine that is in turn capable of implementing the proposed

methods herein can be used to implement the principles of this invention.

Furthermore, the disclosed methods may be readily implemented in software using object or object-

oriented software development environments that provide portable source code that can be used on a

variety of computer or workstation platforms. Alternatively, the disclosed system may be implemented

partially or fully in hardware using standard logic circuits or a VLSI design. Whether software or

hardware is used to implement the systems in accordance with this invention is dependent on the speed

and/or efficiency requirements of the system, the particular function, and the particular software or

hardware systems or microprocessor or microcomputer systems being utilized. The methods illustrated

herein however can be readily implemented in hardware and/or software using any known or later

developed systems or structures, devices and/or software by those of ordinary skill in the applicable art

from the functional description provided herein and with a general basic knowledge of the computer and

image processing arts.

Moreover, the disclosed methods may be readily implemented in software executed on programmed

general-purpose computer, a special purpose computer, a microprocessor, or the like. In these instances,

the systems and methods of this invention may be implemented as program embedded on personal

computer such as JAVA® or CGI script, as a resource residing on a server or graphics workstation, as a

plug-in, or the like. The system may also be implemented by physically incorporating the system and

method into a software and/or hardware system.

Having described above several aspects of at least one embodiment, it is to be appreciated various

alterations, modifications, and improvements will readily occur to those skilled in the art. Such

alterations, modifications, and improvements are intended to be part of this disclosure and are intended to

be within the scope of the invention. Embodiments of the methods and apparatuses discussed herein are

not limited in application to the details of construction and the arrangement of components set forth in the

foregoing description or illustrated in the accompanying drawings. The methods and apparatuses are

capable of implementation in other embodiments and of being practiced or of being carried out in various

ways. Examples of specific implementations are provided herein for illustrative purposes only and are not

intended to be limiting. Also, the phraseology and terminology used herein is for the purpose of

description and should not be regarded as limiting. The use herein of "including," "comprising,"

"having," "containing," "involving," and variations thereof is meant to encompass the items listed

WO wo 2021/080933 PCT/US2020/056405

thereafter and equivalents thereof as well as additional items. References to "or" may be construed as

inclusive SO so that any terms described using "or" may indicate any of a single, more than one, and all the

described terms. Any references to front and back, left and right, top and bottom, upper and lower, and

vertical and horizontal are intended for convenience of description, not to limit the present systems and

methods methods or or their their components components to to any any one one positional positional or or spatial spatial orientation. orientation. Accordingly, Accordingly, the the foregoing foregoing

description and drawings are by way of example only, and the scope of the invention should be

determined from proper construction of the appended claims, and their equivalents.

Claims

CLAIMS 14 May 2025 14 2025

2020370038 May 1. 1. A A system system forfor tracking tracking locations locations of of oneone or or more more persons persons in a in a building, building, the the system system comprising: comprising:

at least at leastone one radiofrequency nodedisposed radiofrequency node disposed near near an an entrance entrance areaarea of the of the building, building, each each RF RF 55 nodeofofthe node the at at least least one radiofrequency one radiofrequency node node having having anreceiver an RF RF receiver to receive to receive an RFan RF signal signal from from respective RF-transmitting respective RF-transmittingdevices devices when when in near in or or near the the entrance entrance area, area, eacheach RF node RF node of theofatthe at least least

one radiofrequency one radiofrequency node node configured configured for associating for associating an identifier an identifier of each of each of the of the RF-transmitting RF-transmitting

devices, contained containedininthe theRF RFsignals signalsreceived receivedfrom from thethe relevant oneone of the RF-transmitting 2020370038

devices, relevant of the RF-transmitting

devices, with devices, with an anangular angularposition positionofofthe theRF-transmitting RF-transmitting device device sending sending the the RF signal RF signal containing containing

10 0 the respective identifier; the respective identifier;

at least at leastone one optical optical device device disposed nearthe disposed near the entrance entrancearea areaand andarranged arranged forfor capturing capturing an an imageofofthe image theentrance entrancearea areawhile whilea aplurality pluralityofofpersons personsare arepresent presentininthe theentrance entrancearea; area;and and a controller a controller in in communication with communication with each each RF node RF node of at of the theleast at least one one radiofrequency radiofrequency node node and the and the at at least least one one optical optical device, device, the the controller controller being being configured: configured:

155 - to - to obtain obtain from eachRF from each RFnode nodeof of theatatleast the leastone oneradiofrequency radiofrequency node node the the received received

identifier of identifier of each each of of the the RF-transmitting devicesand RF-transmitting devices andthe therespectively respectivelyassociated associated angular angular position position

of the of the RF-transmitting devices, RF-transmitting devices,

- to - to obtain obtain from the at from the at least leastone one optical optical device device an an image capturedofofthe image captured theentrance entrancearea area with the with the persons personsinin the the entrance entrancearea; area; 20 !O - to - to detect detect one one or or more of the more of the persons personsinin the the captured capturedimage, image,andand - to - to assign assign the the respective respective individual individual identifiers identifiers of ofthe theRF-transmitting devicestoto RF-transmitting devices

respective individual respective individualpersons personsdetected detectedininthe thecaptured capturedimage image andand the the angular angular position position of of the the respective ones respective onesofofthe the RF-transmitting RF-transmittingdevices devices by by each each RF node RF node of at of the theleast at least one one radiofrequency radiofrequency

node, thereby node, therebyenabling enablingidentification identificationofofeach eachindividual individualperson person whowho is be is to to be subsequently subsequently tracked tracked

25 25 optically as optically as that that individual individual person movesthroughout person moves throughout thethe building. building.
2. 2. Thesystem The systemofofclaim claim 1,1, wherein wherein each each of the of the RF RF nodes nodes of at of the theleast at least one one radiofrequency radiofrequency

nodecomprises node comprisesan an antenna antenna array array and and is configured is configured to compute to compute the angular the angular position position of theof the RF- RF- transmitting devices transmitting devicesbybydetermining determiningan an angle angle of of arrival arrival of of each each of of thethe RF RF signals signals incident incident on the on the

30 30 antennaarray, antenna array, optionally optionallyusing usingtime timedifference differenceofofarrival arrival(TDOA) (TDOA) measurements measurements of the of the incident incident

RFsignals. RF signals.

25 25 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU
3. Thesystem systemofofclaim claim 1 or2,2,wherein whereinthethe at at leastone oneoptical opticaldevice device is is configured to 14 May 2025 2020370038 14 May 2025

3. The 1 or least configured to

capture the image capture the imageofofthe theentrance entrancearea areaatatabout aboutthe thesame same time time or or momentarily momentarily afterafter the identifiers the identifiers

transmitted by transmitted bythe theRF-transmitting RF-transmitting mobile mobile devices devices are are received received in the in the RF nodes RF nodes ofattheleast of the at least one one RF node. RF node. 5 5
4. 4. Thesystem The systemofofanyone anyone of of claims claims 1 3, 1 to to 3, wherein wherein the the at least at least oneone radiofrequency radiofrequency node node

comprisestwo comprises twoorormore more stationary stationary RF RF nodes nodes spaced spaced fromanother, from one one another, whereinwherein the controller the controller is is configuredtotostore store relative relative locations of the the stationary stationary RF nodeswith withrespect respecttotoeach eachother other and 2020370038

configured locations of RF nodes and

with respect with respect to to the the at at least leastone one optical optical device; device; and and wherein thecontroller wherein the controlleris is configured configuredtototrack track 10 0 whichofofthe which thestationary stationaryRFRFnodes nodes sent sent which which of the of the angular angular position position determinations. determinations.
5. 5. Thesystem The systemofofany any one one of of claims claims 1 to 1 to 4, 4, wherein wherein the the at at least least oneone optical optical device device is is stationary and stationary and configured configuredfor forcapturing capturingananorientation orientationmarker marker in in thethe entrance entrance area, area, andand wherein wherein

the controller the controller is is further furtherconfigured configured to to detect detect the the orientation orientation marker in the marker in the image imagecaptured capturedofofthe the 155 entrance area entrance area and andtotoassign assignthe therespective respectiveindividual individualidentifiers identifiers of of the the RF-transmitting RF-transmittingdevices devices to to

respective individual respective individualpersons personsdetected detectedininthe theimage image based based also also on on a position a position of of thethe orientation orientation

markerininthe marker theimage imagerelative relativetotoeach eachperson person detected detected in in the the captured captured image. image.
6. 6. Thesystem The systemofofany any one one of of claims claims 1 to 1 to 5, 5, wherein wherein the the RF-transmitting RF-transmitting devices devices are are RF- RF- 20 !O transmitting mobile transmitting mobiledevices devicescarried carriedbyby atat leastsome least someof of thepersons the persons detected detected in in thethe captured captured

image. image.
7. 7. Thesystem The systemofofclaim claim 2 oranyany 2 or oneone of of claims claims 3 to 3 to 6 when 6 when dependent dependent on claim on claim 2, wherein 2, wherein

the controller is further configured to: the controller is further configured to:

25 25 - arrange, - arrange, for for each each of of the the RF nodesofofthe RF nodes theatat least least one radiofrequencynods, one radiofrequency nods, the the RF- RF-

transmitting mobile transmitting mobiledevices devicesininananangular angular order order based based on on the the angles angles of arrival of arrival of of thethe respective respective RF RF signals received signals fromthe received from theRF-transmitting RF-transmitting devices devices by by a respective a respective one one of the of the RF nodes; RF nodes;

- arrange - the persons arrange the capturedininthe persons captured theimage imageinto intoananangular angular order order based based on relative on relative

positions of positions of the the persons detectedininthe persons detected the captured capturedimage; image;andand 30 30 - compare - theangular compare the angularorder orderofofthe thepersons persons detected detected in in thethe captured captured image image withwith the the angular order angular orderofof the the RF-transmitting RF-transmittingdevices devices when when assigning assigning the identifiers the identifiers of respective of respective onesones of of the RF-transmitting the devicestotorespective RF-transmitting devices respectiveones ones of of thepersons the persons detected detected in in thethe captured captured image. image.

26 26 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU
8. Thesystem systemofofclaim claim 2 oranyany oneone of of claims 3 to 7 when dependent on claim 2, wherein 14 May 2025 2020370038 14 May 2025

8. The 2 or claims 3 to 7 when dependent on claim 2, wherein

the controller is further configured to: the controller is further configured to:

-compute -compute a alocation locationfor foreach eachofofthe theRF-transmitting RF-transmitting devices devices based based on angle on the the angle of arrival of arrival

of the of the respective respective RF signalsreceived RF signals receivedfrom from therespective the respective ones ones of of thethe RF-transmitting RF-transmitting devices devices by by 5 5 each RF each RFnode nodeof of theatatleast the leastone oneradiofrequency radiofrequency nodes; nodes;

-arrange the RF-transmitting -arrange the RF-transmittingmobile mobile devices devices indepth in a a depth order order based based on computed on the the computed locations of locations of the the RF-transmitting devices; RF-transmitting devices;

-arrange the persons personscaptured capturedininthe theimage image intoa adepth depth order based on on relative positions 2020370038

-arrange the into order based relative positions

of the of the persons detectedinin the persons detected the captured capturedimage; image;andand 10 0 -compare thedepth -compare the depthorder order ofof thepersons the persons detected detected in in thethe captured captured image image withwith the depth the depth

order of order of the the RF-transmitting deviceswhen RF-transmitting devices when assigning assigning the the identifiers identifiers of of respective respective ones ones of the of the RF-RF-

transmitting devices transmitting devicestotorespective respectiveones onesofofthe thepersons personsdetected detected inin thecaptured the captured image. image.
9. 9. Thesystem The systemofofany any one one of of claims claims 1 to 1 to 8, 8, wherein wherein each each RF node RF node of theofat theleast at least one one 155 radiofrequencynode radiofrequency node is is configured configured to to determine determine a relative a relative signal signal strength strength indicator indicator (RSSI) (RSSI) value value

for the for the RF signals received RF signals receivedfrom fromeach each of of theRF-transmitting the RF-transmitting devices, devices, and and wherein wherein the controller the controller

is configured is to estimate configured to estimate aa distance distance of of each eachofofthe the RF-transmitting RF-transmittingdevices devices from from each each of the of the RF RF nodesofofthe nodes the at at least least one one radiofrequency node radiofrequency node using using thethe RSSI RSSI value value for for the the RF signal RF signal received received by by the respective the onesof respective ones of the the RF-transmitting RF-transmittingdevices devices by by thethe respective respective RF RF nodenode anduse and to to the use the 20 !O estimateddistance estimated distanceofofeach eachofofthe theRF-transmitting RF-transmitting devices devices from from eacheach of the of the RF nodes RF nodes when when assigning the assigning the identifiers identifiers of of respective respective ones of the ones of the RF-transmitting devicestotorespective RF-transmitting devices respectiveones ones of of

the persons the detectedininthe persons detected the captured capturedimage. image.
10. 10. Thesystem The systemofofany any one one of of claims claims 1 to 1 to 9, 9, wherein wherein the the controller controller is is further further configured configured to:to:

25 25 - detect - detect in in the thecaptured captured image mobilephones image mobile phones held held by by the the persons persons detected detected in the in the image image

and to and to assign assign the the identifiers identifiers of of respective respective ones of the ones of the RF-transmitting devicestotorespective RF-transmitting devices respectiveones ones of the of the persons detectedinin the persons detected the captured capturedimage image based based on on which which of the of the persons persons detected detected in image in the the image are holding are the detected holding the detectedmobile mobilephones. phones.

30 30 11.
11. Thesystem The systemofofany any one one of of claims claims 1 to 1 to 10,10, further further comprising comprising a plurality a plurality of of further further optical optical

devices disposed devices disposedthroughout throughoutthethe building, building, andand wherein wherein the the controller controller is further is further configured configured to to optically track optically track each identified individual each identified personthrough individual person throughthe thebuilding buildingbased based on on detecting detecting that that

27 27 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU identified individual individual in in images receivedfrom fromatat leastsome someof of theplurality pluralityofoffurther furtheroptical optical 14 May 2025 2020370038 14 May 2025 identified images received least the devices over devices overtime. time.
12. 12. Thesystem The systemofofclaim claim 1010 or or 11, 11, wherein wherein information information carried carried by RF by the thesignals RF signals transmitted transmitted

5 5 by aa given by givenone oneofofthe theRF-transmitting RF-transmitting mobile mobile devices devices includes includes a shopping a shopping list. list.
13. 13. Thesystem The systemofofclaim claim 12,wherein 12, wherein thethe controller controller is is furtherconfigured further configured to:to:

- determine determine aa route routethrough throughthe thebuilding buildingbased based on on items on on the the shopping list;list; andand 2020370038

- items shopping

- transmit - transmit the the route route to to the the given given one of the one of the RF-transmitting mobile RF-transmitting mobile devices devices forfor display display on on 10 0 a screen a of the screen of the given RF-transmittingdevice. given RF-transmitting device.
14. 14. Amethod A method fortracking for tracking locations locations of of individuals individuals in in a building,thethemethod a building, method comprising: comprising:

receiving, by receiving, by at at least least one one radiofrequency node radiofrequency node disposed disposed near near an entrance an entrance areaarea to the to the

building, an building, an RF RFsignal signalfrom fromRF-transmitting RF-transmitting devices devices whenwhen near near or in or inentrance the the entrance area area to to the the 155 building, the building, the respective RFsignals respective RF signalscontaining containingananidentifier identifierofofthe theRFRFtransmitting transmitting device device emitting emitting

the respective the RFsignal; respective RF signal; determining, by determining, by each eachRF RFnode nodeofofthe theatatleast least one radiofrequencynode, one radiofrequency node,ananangular angular position of each of the RF signal emitting RF-transmitting devices; position of each of the RF signal emitting RF-transmitting devices;

associating, by associating, eachRFRFnode by each node of of the the atat leastone least oneradiofrequency radiofrequency node, node, the the identifier identifier of of a a 20 20 respective one respective oneofofthe the RF-transmitting RF-transmittingdevices devices with with thethe determined determined angular angular position position of RF- of the the RF- transmitting device transmitting devicesending sendingthe theRFRF signal signal carrying carrying that that identifierand identifier andreceived received by by that that RF RF node; node;

capturingan capturing animage imageofofthe theentrance entrance area; area;

detecting one detecting oneoror more morepersons persons in in thecaptured the captured image; image; and and

assigningthe assigning the respective respectiveindividual individualidentifier identifier of of each each of of the the RF-transmitting RF-transmittingdevices devices to to

25 25 respective individual respective individualpersons personsdetected detectedininthe thecaptured capturedimage image based based on aon a position position of each of each of of the the personsdetected persons detectedininthe thecaptured capturedimage imageandand thethe determined determined angular angular position position of respective of the the respective RF- RF- transmitting devices transmitting devicessending sendingthetheRFRF signals signals received received by by each each RF node RF node of theofat theleast at least one one radiofrequencynode, radiofrequency node, thereby thereby enabling enabling identification identification of of each each individual individual person person who who is to is beto be tracked optically tracked optically as as that that individual individual person moves person moves through through thethe building. building.

30 30 15.
15. Themethod The methodof of claim claim 14,14, wherein wherein eacheach of the of the RF nodes RF nodes of theofatthe at least least one radiofrequency one radiofrequency

nodecomprises node comprisesan an antenna antenna array, array, thethe method method further further comprising comprising computing computing the angular the angular positionposition

of the of the RF-transmitting devicesbybydetermining RF-transmitting devices determining an angle an angle of arrival of arrival of of each each of the of the RF RF signals signals

28 28 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU incident on on the the antenna antennaarray, array,optionally optionallyusing usingtime timedifference differenceofofarrival arrival(TDOA) (TDOA) measurements 14 May 2025 2020370038 14 May 2025 incident measurements of the of the incident incident RF signals. RF signals.
16. 16. Themethod The methodof of claim claim 14 14 or or 15,15, further further comprising comprising the the stepstep of detecting of detecting an orientation an orientation

5 5 markerininthe marker thecaptured capturedimage, image, and and wherein wherein the the stepstep of assigning of assigning the the respective respective individual individual

identifiers of identifiers of each each of of the the RF-transmitting devicestotorespective RF-transmitting devices respectiveindividual individualpersons persons detected detected in in the the

captured imageisisfurther captured image furtherbased basedonona aposition positionofofthe theorientation orientationmarker markerin in thethe captured captured image image

relative to to each each of of the the persons detected in in the the image. 2020370038

relative persons detected image.

10 0 17.
17. Themethod The methodof of claim claim 14,14, 15 15 or or 16,16, wherein wherein the the RF-transmitting RF-transmitting devices devices are are RF- RF- transmitting mobile transmitting mobiledevices devicescarried carriedbyby atat leastsome least someof of thepersons the persons detected detected in in thethe captured captured

image. image.
18. 18. Themethod The methodof of claim claim 15 15 or or anyany one one of claims of claims 1617towhen 16 to 17 when dependent dependent on15, on claim claim 15, 155 further comprising: further comprising:

arranging, for arranging, for each each RF RFnode nodeof of theatatleast the leastone oneradiofrequency radiofrequency node, node, the the RF-transmitting RF-transmitting

devices in devices in an an angular angularorder orderbased basedononthetheangles angles of of arrivalofofthe arrival theRF-signals RF-signals respectively respectively received received

fromthe from theRF-transmitting RF-transmitting devices devices by by that that RF RF node; node;

arranging the persons arranging the personsininthe thecaptured capturedimage image into into an an angular angular order order based based on relative on relative

20 !O positions of positions of the the persons detectedininthe persons detected the captured capturedimage; image;andand comparingthetheangular comparing angular order order of of thethe persons persons detected detected in the in the captured captured image image with with the the angular order angular orderofof the the RF-transmitting RF-transmittingdevices devices when when assigning assigning the respective the respective individual individual identifiers identifiers

of the of the RF-transmitting devicestotothe RF-transmitting devices therespective respectiveindividual individualpersons persons detected detected in in thethe captured captured

image. image.

25 25
19. 19. Themethod The methodof of claim claim 15 15 or or anyany one one of claims of claims 1618towhen 16 to 18 when dependent dependent on15, on claim claim 15, further comprising: further comprising:

computinga alocation computing locationforforeach eachofofthetheRF-transmitting RF-transmitting devices devices based based on angle on the the angle of arrival of arrival

of the of the respective respective RF signalreceived RF signal receivedfrom from therespective the respective RF-transmitting RF-transmitting device device by each by each RF RF node node 30 30 of the of the at at least leastone one radiofrequency node; radiofrequency node;

arranging the RF-transmitting arranging the RF-transmittingdevices devices in in a depth a depth order order based based on the on the computed computed locations locations

of the of the RF-transmitting devices; RF-transmitting devices;

arranging the persons arranging the personscaptured capturedininthe theimage image into into a depth a depth order order based based on relative on relative positions positions

of the of the persons detectedinin the persons detected the captured capturedimage; image;andand

29 29 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU comparingthethedepth depth order of of thethe persons detected in the captured image with with the depth 14 May 2025 2020370038 14 May 2025 comparing order persons detected in the captured image the depth order of order of the the RF-transmitting mobile RF-transmitting mobile devices devices when when assigning assigning the respective the respective individual individual identifiers identifiers of the of the RF-transmitting mobile RF-transmitting mobile devices devices to to thethe respective respective individual individual persons persons detected detected in the in the capturedimage. captured image. 55 20.
20. Themethod The methodof of any any oneone of claims of claims 1419, 14 to to 19, further further comprising: comprising: determining,bybyeach determining, eachofofthe theRFRF nodes nodes of the of the at at leastoneone least radiofrequency radiofrequency node, node, a relative a relative signal strength indicator (RSSI) (RSSI)value valuefor forthe theRFRFsignal signalreceived received from each of the RF-transmitting 2020370038 signal strength indicator from each of the RF-transmitting devices at devices at the the RF nodes; RF nodes;

10 0 estimating aa distance estimating distanceof of each eachofofthe theRF-transmitting RF-transmitting devices devices from from eacheach of the of the RF nodes RF nodes of of the at the at least leastone one RF nodebased RF node basedononthetheRSSI RSSI value value for for thethe RF RF signal signal received received fromfrom the respective the respective

ones of ones of the the RF-transmitting RF-transmittingdevices devices byby thethe respective respective RF RF node; node; and and consideringthe considering theestimated estimateddistance distanceofofeach each ofof theRF-transmitting the RF-transmitting devices devices fromfrom each each of of the RF the nodesofofthe RF nodes theatatleast least one oneradiofrequency radiofrequency node node when when assigning assigning the individual the individual identifiers identifiers of of 155 respective ones respective onesofofthe the RF-transmitting RF-transmittingdevices devices to to therespective the respective ones ones of of thethe persons persons detected detected in in the captured the image. captured image.
21. 21. Themethod The methodof of any any oneone of claims of claims 1420, 14 to to 20, further further comprising comprising detecting detecting in captured in the the captured imagemobile image mobile phones phones held held by the by the persons persons detected detected in captured in the the captured image, image, and wherein and wherein theofstep the step of 20 !O assigningthe assigning the identifiers identifiers of of respective respective ones of the ones of theRF-transmitting RF-transmitting devices devices to to thethe respective respective ones ones

of the of the persons detectedinin the persons detected the captured capturedimage imageisisbased basedonon which which persons persons detected detected in captured in the the captured image areholding image are holdingthe thedetected detectedmobile mobile phones. phones.
22. 22. Themethod The methodof of any any oneone of claims of claims 1421, 14 to to 21, further further comprising comprising arranging arranging a plurality a plurality of of 25 25 further optical further optical devices throughoutthe devices throughout thebuilding, building and , andoptically opticallytracking trackinga alocation locationofofeach each identified individual identified individual through thebuilding through the buildingbybycomparing comparing successive successive images images offurther of the the further optical optical

devices that capture that identified individual. devices that capture that identified individual.
23. 23. Themethod The methodof of any any oneone of claims of claims 1422, 14 to to 22, wherein wherein information information carried carried by theby RFthe RF signals signals

30 30 transmitted by transmitted byaagiven givenone oneofofthe theRF-transmitting RF-transmitting mobile mobile devices devices includes includes a shopping a shopping list. list.
24. 24. Themethod The methodof of claim claim 23,23, further further comprising: comprising:

determininga aroute determining routethrough through thebuilding the building based based on on items items on the on the shopping shopping list;list; and and

30 30 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU transmitting the the route route to to the the given oneofof the the RF-transmitting RF-transmittingdevices devices forfor display on on a 14 May 2025

2025 transmitting given one display a

screen of the screen of the given given RF-transmitting RF-transmittingmobile mobile device. device.

2020370038 14 May
25. 25. Themethod The methodof of claim claim 19 19 andand any any one one of claims of claims 20 to20 24towhen 24 when dependent dependent on claimon claim 19, 19, 5 5 further further comprising, after persons comprising, after personsidentified identifiedas as carrying carryingaamobile mobilephone phone in in thethe captured captured image image and and

to whom to theidentifiers whom the identifiersofofrespective respectiveones onesofofthe theRF-transmitting RF-transmitting mobile mobile devices devices have have been been assignedenter assigned enterthe the building, transmitting, by building, transmitting, bythe theat at least least one RFnode, one RF node,radio radiosignals signalsconveying conveyinga a current location location of of those those persons personswithin withinthe thebuilding buildingtotothe therespective respectivemobile mobile phones carried by 2020370038

current phones carried by

the respective the persons. respective persons.

10 0

31 31 21750991_1(GHMatters) 21750991_1 (GHMatters)P118744.AU P118744.AU