JP7624282B2 - Device, program, and method for tracking terminal based on communication compatibility of detected object - Google Patents

Description

This relates to technology for obtaining information about a terminal as a communication target, in particular technology for tracking that terminal.

The fifth generation mobile communications system (5G), which is currently being introduced, uses millimeter wave radio waves (mm waves) to enable high-performance communications such as high speed, large capacity, low latency, and multi-terminal connection. Here, mmWaves have a high degree of directivity and are not prone to diffraction, making it highly likely that their propagation will be blocked or attenuated by objects between the terminal and the base station. For this reason, with 5G, a major problem is that the presence of such objects can cause a sudden drop in received power, which can result in, for example, the communication connection itself being cut off.

Therefore, if it were possible to predict the drop in received power due to such objects, it would be possible to maintain communication connections without interruption by controlling and securing the terminal's communication path. As a technique for predicting this received power, Non-Patent Document 1 discloses a technique for predicting the received power in a communication device through machine learning, using time-series three-dimensional environmental information acquired by an RGB-D camera that can generate depth images in addition to RGB images.

Specifically, this technology uses training data that links time-series environmental information resulting from capturing an image of the communication environment with time-series information on the received power measured (at the same time as capturing the image) at the communication terminal that receives radio waves from the transmitting station as correct answer data to construct a machine learning model, and then uses this constructed machine learning model to determine a predicted value for the received power at the communication terminal from the time-series environmental information captured by the RGB-D camera.

Here, in predicting the received power of millimeter waves, it is important to continuously grasp the position of the communication terminal. However, if the position is grasped using the positioning results from a sensor on the communication terminal side, a mechanism is required for the communication terminal to notify the base station of the positioning results, and furthermore, the large power consumption and calculation load on the communication terminal side become problems.

In contrast to this, Patent Document 1, which includes the present inventor as one of its inventors, discloses a technology that estimates the position of a communication terminal using time-series information on the received power of millimeter waves and time-series information on the occlusion relationship between objects estimated from, for example, image information of the environment, as a technology that does not depend on the positioning results from the communication terminal. Furthermore, Non-Patent Document 2 discloses a technology that continuously captures the position of an object in an image by utilizing the consistency of the detected position and appearance features of the object obtained from the image.

JP 2021-078022 A

Nishio, R., "Prediction and control of wireless communication quality using machine learning", IEICE Technical Report, Vol. 118, No. 57, SR2018-1, pp. 1-8, 2018. Nicolai Wojke et al., "Simple Online and Realtime Tracking with a Deep Association Metric", 2017 IEEE International Conference on Image Processing, pp. 3645-3649, 2017. <https://doi.org/10.1109/ICIP.2017.8296962 ＞

If it were possible to continue tracking a communications terminal using such conventional technology, it would be possible to continue the communications connection, or to temporarily suspend it and then immediately resume it, even if the reception power suddenly drops due to the presence of an object.

However, object tracking technology based on image information such as that disclosed in Non-Patent Document 2 may fail to track a target when a phenomenon known as occlusion occurs, in which an object in the foreground at least partially covers an object in the background. In other words, conventional object tracking technology based on image information has problems when occlusion occurs, such as the target being "lost" because it is hidden by another object and cannot be detected, causing tracking to cease, or an "ID switch" occurring when the target being tracked crosses another object and the other object is mistaken for the target being tracked, causing the tracking to stray from what should have been done.

Therefore, even if the target tracking technology disclosed in Non-Patent Document 2 is applied to tracking communication terminals, if such an occlusion problem occurs, it will not be possible to accurately grasp the position of the communication terminal. In addition, the communication terminal position estimation technology described in Patent Document 1 also assumes that the terminal has not been lost or an ID switch has not occurred, and does not propose any measures to be taken when such problems occur.

The present invention aims to provide a device tracking device, program, and method that can continue tracking a device more appropriately even if occlusion occurs for the device being tracked.

According to the present invention,
an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, the environmental information being generated by the sensor, and determining object position information which is information related to a detection position of the detected object;
a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information;
a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position coincidence degree, which is a degree of coincidence between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal;
A terminal tracking device is provided, characterized in that the tracking target determination means makes the judgment based on the degree of communication compatibility rather than on the degree of positional agreement when the degree of communication compatibility is large enough to satisfy a specified condition, or makes the judgment by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a specified condition.

As one embodiment of the terminal tracking device according to the present invention, the tracking target determination means preferably performs the determination without basing the communication compatibility degree on the degree of communication compatibility when the degree of positional agreement is small enough to satisfy a predetermined condition, or performs the determination by reducing the degree based on the communication compatibility degree as the degree of positional agreement is small.

In another embodiment of the terminal tracking device according to the present invention, the object detection information determination means also determines object appearance information, which is information related to the detected appearance of the detected object, from the environmental information;
When the tracking target determination means makes the determination based at least on the degree of positional consistency, it is also preferable that the tracking target determination means also makes the determination based on the degree of appearance consistency, which is the degree of consistency between the target appearance information and past target appearance information of the target determined to be related to the terminal being tracked.

Furthermore, in the embodiment relating to the above-mentioned target appearance information, the tracking target determination means
When the degree of positional agreement is smaller than a predetermined condition, the determination is made based on at least the degree of appearance agreement and the degree of positional agreement, and at that time, the degree based on the degree of appearance agreement is made larger, or
It is also preferable that the determination be made based on at least the degree of appearance match and the degree of position match, and that the smaller the degree of position match, the greater the degree based on the degree of appearance match.

In one embodiment of the target communication information determination means, it is also preferable that the target communication information determination means determines the target communication information by assuming that the target is present at a position interpolated using detection positions determined before and after the occlusion during a time period during which an occlusion occurs and the detection position of the target is not determined.

Furthermore, as an embodiment related to the degree of positional agreement, the environment information is information related to an image of the environment, and the target position information includes a position and a range within the image of an image area related to the detected target,
It is also preferable that the degree of positional agreement is the degree of overlap between the range of the detected target and the predicted range in the image of the target determined to be associated with the tracking terminal, or the degree of closeness between the position of the detected target and the predicted position in the image of the target determined to be associated with the tracking terminal.

In addition, as a preferred embodiment of the tracking target determination process according to the present invention, the tracking target determination means
determining a weighting factor that is a monotonically increasing function of the degree of communication capability, the weighting factor including a zero value;
It is also preferable to calculate a tracking cost of the target, which is a weighted sum of environmental information cost including location cost, which is a monotonically decreasing function of the degree of location agreement, and communication cost, which is a monotonically decreasing function of the degree of communication compatibility, where the communication cost is associated with the weighting coefficient and the environmental information cost is associated with another weighting coefficient which may take on a value of zero, and to determine the target for which the tracking cost is small enough to satisfy a specified condition as the target related to the terminal being tracked.

Furthermore, in the embodiment in which the weighting coefficient is determined as described above, it is also preferable that the tracking target determination means determines the weighting coefficient that is a monotonically increasing function for both the communication compatibility degree and the position matching degree.

According to the present invention, there is also provided a method for producing a medicament for a medicament comprising the steps of:
an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, generated by a sensor, determining object position information which is information related to a detected position of the detected object, and also determining object appearance information which is information related to a detected appearance of the detected object from the environmental information;
a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information;
a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position matching degree, which is a degree of matching between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, an appearance matching degree, which is a degree of matching between the target appearance information and past target appearance information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of compatibility between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal;
A terminal tracking device is provided in which the tracking target determination means makes the determination based on the degree of communication compatibility, not on the degree of positional agreement, when the degree of communication compatibility is large enough to satisfy a specified condition, at least when occlusion is resolved for the detected target, or makes the determination by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a specified condition.

According to the present invention, further comprising:
an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, the environmental information being generated by the sensor, and determining object position information which is information related to a detection position of the detected object;
a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information;
a position match degree, which is a degree of match between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the terminal being tracked, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the terminal being tracked and the target communication information, determining whether or not the terminal being tracked and the detected target are in a corresponding relationship, and causing the computer to function as a tracking target determination means for determining the target determined to be in a corresponding relationship as the target related to the terminal being tracked;
A terminal tracking program is provided in which the tracking target determination means makes the determination based on the degree of communication compatibility, not on the degree of positional agreement, when the degree of communication compatibility is large enough to satisfy a specified condition, at least when occlusion is resolved for the detected target, or makes the determination by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a specified condition.

According to the present invention, further,
a first step of detecting a predetermined target which may include the terminal from environmental information generated by a sensor and related to an environment in which the terminal may be present, determining target position information which is information related to a detection position of the detected target, and also determining terminal communication information which is information related to a communication state between the terminal and the detected target based on information related to communication acquired from a communication means for communicating with the terminal, and further determining target communication information from the environmental information which is information related to a communication state which will be realized between the detected target and the target if the detected target includes the terminal;
a second step of determining whether or not the tracked terminal and the detected target are in a corresponding relationship based on at least one of a position match degree, which is a degree of match between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracked terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracked terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracked terminal;
A computer-implemented terminal tracking method is provided, characterized in that in a second step, when occlusion is resolved at least for the detected target, the determination is made based on the degree of communication compatibility rather than on the degree of positional agreement when the degree of communication compatibility is large enough to satisfy a predetermined condition, or the determination is made by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a predetermined condition.

The device, program, and method of the present invention make it possible to continue tracking a terminal more appropriately even if occlusion occurs for the terminal being tracked.

1 is a functional block diagram showing a functional configuration of an embodiment of a terminal tracking device according to the present invention; 5A to 5C are schematic diagrams for explaining two aspects of occlusion determination processing in the occlusion determination unit according to the present invention. FIG. 11 is a schematic diagram for explaining a specific example of a calculation process of communication costs according to the present invention. 10A and 10B are schematic diagrams for explaining a specific example of a calculation process of a position cost and an appearance cost according to the present invention; 1A to 1C are schematic diagrams for explaining various embodiments of a tracking target determination process in a tracking target determination and management unit according to the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings.

[Terminal tracking device]
FIG. 1 is a functional block diagram showing the functional configuration of an embodiment of a terminal tracking device according to the present invention.

The base station 1 shown in FIG. 1 as one embodiment of a terminal tracking device according to the present invention is a communication relay device compatible with 5G (fifth generation mobile communication system) in this embodiment, and is a device capable of communicating with multiple terminals 2, which are also communication terminals compatible with 5G.

In addition, as one embodiment of a terminal tracking device, the base station 1 is also a device capable of detecting specific targets, including the terminal 2, from image data serving as "environmental information" acquired by the camera 103, which acts as a sensor that senses the environment in which the terminal 2 may be present, and which is information relating to the environment and which may include specific targets, and tracking the terminal 2.

In this embodiment, the camera 103 is an RGB camera capable of capturing images of the communication area with a predetermined angle of view. The image data (video data) generated by the camera 103 includes the following:
(a) A person who possesses or carries the terminal 2,
(b) Moving objects such as automobiles, motorcycles, railroad cars, robots, drones, etc. that are equipped with, contain, or have terminal 2 on board, and (c) "objects" such as equipment, facilities, buildings, and fixed objects on which terminal 2 is installed may be included (images may be captured). Furthermore, "objects" such as people, moving objects, equipment, facilities, buildings, and fixed objects (including plants such as trees) that are not related to terminal 2 may also be included.

Incidentally, while such an "object" can be something for which a correspondence with the terminal 2 being tracked needs to be determined (in other words, something for which it needs to be determined whether or not it is a tracking target that should be tracked), its presence between the terminal 2 and the base station 1 can also become an obstacle to communication radio waves.

Here, in particular, 5G, which is the communication method of this embodiment, uses millimeter wave band radio waves (millimeter waves) for communication, but these millimeter waves have a high degree of directivity and are not easily diffracted, so there is a high possibility that their propagation will be blocked or attenuated by "objects" that exist between the terminal 2 and the base station 1. Therefore, with 5G, the presence of such "objects" can cause a sudden drop in the received signal power at the base station 1, which can lead to, for example, a situation in which the communication connection itself is interrupted, posing a serious problem.

In contrast, even if "occlusion" occurs with respect to a tracked device 2, if tracking of this device 2 can be continued appropriately, it is possible to continue the communication connection with that device 2, or to temporarily suspend and then immediately resume the connection.

Incidentally, the above-mentioned "occlusion" hereinafter refers to a phenomenon in which an "object" in front of the base station 1 covers at least a part of an "object" in the back. For example, a phenomenon in which multiple "objects" intersect, or at least one "object" is blocked by at least one other "object", will also be treated as "occlusion" hereinafter. When such "occlusion" occurs, there is an increased possibility that a specific object (e.g. a "person") cannot be detected from "environmental information" (image data in this embodiment), or can only be detected inaccurately.

In order to solve the serious problem when occlusion occurs as described above, the base station 1 specifically:
(A) an object detection information determination unit 113 that detects a predetermined object (e.g., a “person”) that may include the terminal 2 from “environmental information” (image data in this embodiment) generated by the above-mentioned sensor (the camera 103 in this embodiment) and determines “object position information” that is information related to the detection position of the detected object (the “person”);
(B) a terminal communication information determination unit 111 that determines "terminal communication information" which is information related to the communication state with the terminal 2 based on information related to communication (e.g., communication history information) acquired from a communication means (in this embodiment, a communication interface 101, a communication control unit 116, and a communication history information storage unit 102 provided in the terminal 2 itself) that communicates with the terminal 2;
(C) a target communication information determination unit 112 that determines, from the "environment information,""target communication information" which is information related to a communication state that will be realized between the detected target (e.g., a "person") and the terminal 2 if the detected target includes the terminal 2;
(D) The tracking target determination and management unit 115 determines whether or not the tracking terminal 2 and the detected target ("person") correspond to each other based on at least one of (d1) "position agreement degree," which is the degree of agreement between the "target position information" and predicted target position information predicted from the "target position information" of the target (e.g., a "person") determined to be related to the tracking terminal 2, and (d2) "communication compatibility degree," which is the degree of correspondence between the "terminal communication information" of the tracking terminal 2 and the "target communication information," and determines the target ("person") determined to be related to the tracking terminal 2 as the target related to the tracking terminal 2.

Furthermore, the tracking target determination and management unit 115 of (D) performs the following when an occlusion occurs and is resolved for at least a detected target (e.g., a “person”):
(D1) When the "degree of communication compatibility" is large enough to satisfy a predetermined condition (in this embodiment, when a representative value (e.g., a maximum value) of the communication compatibility degrees of each detected target exceeds a predetermined threshold), the above determination is made based on the "degree of communication compatibility" and not on the "degree of positional agreement", or
(D2) The larger the "degree of communication compatibility" is (in this embodiment, the larger the representative value (e.g., the maximum value) of the degree of communication compatibility of each detected target is), or when it is large enough to satisfy a specified condition (in this embodiment, when the representative value (e.g., the maximum value) exceeds a specified threshold value), special processing is performed in which the degree based on the "degree of communication compatibility" is made larger and the above-mentioned judgment is made.

The "predetermined target" detected from the "environmental information" in (A) above can be a "person" or a "mobile object" that carries, is equipped with, or may contain the terminal 2, as will be explained in detail later. In some cases, the "predetermined target" may be an "equipment," "facility," "building," "fixed object," etc. that may contain the terminal 2. The "target position information" in (A) above is information related to the target detection position, and is position-related information that may also include the position and speed of the detected target (e.g., a "person").

Furthermore, the "terminal communication information" in (B) above can be, in a preferred embodiment, (α) information related to the strength of radio waves received from terminal 2 (e.g., received signal power, RSSI (Received Signal Strength Indicator)). Or, (β) information related to the presence or absence of a communication connection with terminal 2. In addition, the "target communication information" in (C) above can be, in accordance with the above "terminal communication information", (α') information related to the strength of radio waves received from a specific target ("person") if the detected target (e.g., a "person") includes terminal 2 (e.g., RSSI). Or, (β') information related to the presence or absence of a communication connection with the detected target ("person"), determined using the detection position of the detected target (e.g., a "person").

Furthermore, the "degree of communication compatibility" in (D) above can be specifically, for example, the degree of agreement between time series data on the quantity related to radio wave strength (e.g., RSSI value) as "terminal communication information" and time series data on the quantity related to radio wave strength (e.g., RSSI value) as "target communication information". Alternatively, it can be the degree of agreement between time series data on the presence or absence of a communication connection as "terminal communication information" and time series data on the possibility of a communication connection as "target communication information".

With such settings for "degree of communication compatibility" and "degree of positional agreement," the base station 1 as a terminal tracking device according to the present invention, at least when occlusion is resolved, makes the above judgment by increasing the degree based on the "degree of communication compatibility" (after checking the degree of communication compatibility of each detected target) as shown in (D1) and (D2) above, or uses only the "degree of communication compatibility" as the judgment criterion for the above judgment.

The "degree of communication compatibility" can be determined from the communication state of the terminal 2 and the estimated communication state of the target to be determined, which is obtained from the "environmental information", regardless of whether the target to be determined (e.g. a "person") is detected during the process. In other words, the "degree of communication compatibility" is likely to be used as a more reliable criterion even when occlusion occurs. Therefore, the terminal tracking device (base station 1) according to the present invention makes it possible to continue tracking the terminal 2 more appropriately, even when occlusion occurs for the terminal 2 being tracked.

As a result, in this embodiment, even if a situation occurs in which the base station 1 has a person or other object between itself and the terminal 2 and the reception power drops suddenly, the base station 1 can continue or immediately resume the communication connection with the terminal 2.

Incidentally, in this embodiment, during a period when no occlusion occurs, excluding a predetermined period immediately after the occlusion is resolved, the base station 1 (terminal tracking device) performs at least (a) the above-mentioned "position matching degree" and
(b) It is also preferable to determine whether or not there is a correspondence between the tracked device 2 and the detected object (e.g., a “person”) based on either or both of the “degree of appearance match” determined from “environmental information” (e.g., image data) and described in detail below, and further,
(c) The "degree of communication compatibility" mentioned above
It is also preferable to make the determination based on the following:

In addition, 5G is adopted as the communication method between the base station 1 and the terminal 2 in this embodiment, but of course other communication methods such as LTE or future mobile communication methods using shorter wavelength radio waves may be used, and further, the communication between the terminal tracking device according to the present invention and the communication terminal may be based on various other wireless communication standards. For example, even in a communication method in which the problem of obstruction by objects is not as significant as in 5G, there are many situations in which it is necessary to accurately identify and track subordinate communication terminals using objects detected from camera image data. In addition, there are needs for terminal identification and tracking in various fields, such as determining the relationship between the viewed pages of a user determined to be in correspondence with a certain communication terminal and the user's movement line, and using this for marketing and management. In contrast, the terminal tracking device according to the present invention makes it possible to perform such terminal identification and tracking processing with high accuracy.

The terminal tracking device according to the present invention is not limited to a communication relay device such as a base station, of course. For example, it may be provided as a dedicated device for terminal tracking processing, connected to a communication relay device or facility such as a base station. Also, as the terminal tracking device according to the present invention, it is possible to use a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, or a smartphone, etc., equipped with the terminal tracking program according to the present invention.

Moreover, it is not impossible for at least one of the components (A)-(D) of the terminal tracking device (base station 1) according to the present invention to be a separate device from the other components. For example, it is also possible for the functions of (A)-(D) to be realized by a set of multiple servers. Even in such a case, the set of all of these can be considered to be a terminal tracking device or system that implements the terminal tracking method according to the present invention and is operated by the terminal tracking program according to the present invention.

[Functional configuration of terminal tracking device, terminal tracking program]
1, the base station 1 has a communication interface 101, a communication history information storage unit 102, a camera 103, an environment information storage unit 104, a tracking information storage unit 105, and a processor memory (a calculation processing system with a memory function) as an embodiment of a terminal tracking device and a communication relay device according to the present invention. Here, the processor memory stores an embodiment of a communication relay program including a terminal tracking program according to the present invention, and has a computer function, and executes this communication relay program to perform terminal tracking processing and communication relay processing.

The processor memory also has, as functional components, a terminal communication information determination unit 111, a target communication information determination unit 112 including a position interpolation unit 112a, a target detection information determination unit 113, an occlusion determination unit 114, a tracking target determination and management unit 115 including a communication cost calculation unit 115a, a position cost calculation unit 115b, an appearance cost calculation unit 115c, and a weight determination unit 115d, and a communication control unit 116. These functional components can be considered as functions of a terminal tracking program and a communication relay program stored in the processor memory, and the process flow shown by connecting the functional components of the base station 1 in the functional block diagram of Figure 1 with arrows can also be understood as one embodiment of the terminal tracking method and communication relay method according to the present invention.

Also in the functional block diagram of FIG. 1, the communication control unit 116 functions as a base station by controlling wireless communication operations between the communication interface 101 and each terminal 2, and further acquires and records various information related to communication with each terminal 2, generates communication history information by organizing the information in chronological order, and stores and manages the information in the communication history information storage unit 102.

The camera 103 may be a visible light, near-infrared or infrared compatible imaging device equipped with a solid-state imaging element such as a CCD image sensor or a CMOS image sensor, and may also be an RGB camera, an RGB-D camera, a stereo camera, an all-sphere (omnidirectional) camera, etc. Of course, instead of the camera 103, it is also possible to employ a sensor capable of sensing the environment in which the terminal 2 may be present and generating environmental information, such as a LiDAR, a laser/infrared positioning device, a TOF camera, or a thermography device.

In this embodiment, the camera 103 is an RGB camera capable of generating RGB image data (RGB video data) as environmental information, and the generated RGB image data is stored and managed in the environmental information storage unit 104. Note that, although the camera 103 is installed within the base station 1 in this embodiment, it may be, for example, a camera installed at a location distant from the base station 1, such as a surveillance camera in a city, and communicatively connected to the base station 1.

In addition, multiple cameras 103 may be installed at different locations, whether inside or outside the base station 1. For example, the base station 1 may receive and acquire "target location information" and "target communication information" generated by the cameras 103 installed at each of multiple base stations 1 present around the base station 1, and perform the tracking target determination process. In this case, the multiple base stations 1 work together to more appropriately perform the terminal tracking process according to the present invention.

<Target Detection Information Determination Process>
Also in the functional block diagram of FIG. 1, the object detection information determination unit 113 determines, from image data (environmental information) that may include a “predetermined object” acquired from the environmental information accumulation unit 104,
At least one, and preferably both, of (a) target position information, which is information relating to the detected position of a "specified target," and (b) target appearance information, which is information relating to the detected appearance of a "specified target," is determined.

As already mentioned, the "prescribed subject" is, for example,
(a) A person (user) who owns or carries the terminal 2,
(b) An automobile equipped with a terminal 2 having a drive recorder function, a CAN information transfer function, an interface function for automatic driving control by a server, etc.,
(c) It can be a "person" or "moving object" that may include the terminal 2 that is the communication destination related to the communication history information, such as an autonomous mobile robot or an autonomous flying drone equipped with a terminal 2 that has an interface function for autonomous movement control by a server. In some cases, "equipment,""facility,""building,""fixedobject," etc. that may include the terminal 2 may also be a "predetermined target."

Note that the target detection information determination unit 113 of this embodiment can of course also simultaneously detect multiple types of targets, including a "predetermined target." Here, when one "predetermined target" is detected as a tracking target from one piece of image data, other objects (including other "predetermined targets") detected from this image data are treated as targets that may cause occlusion, i.e., as targets that may cause interference, with respect to the detected "predetermined target."

Here, the process of determining the target position information (a) and the target appearance information (b) will be described. In this embodiment, the target detection information determination unit 113 sets multiple image regions in the acquired image data, calculates a score representing the likelihood of each image region being a specific target using a corresponding object detector, and determines an image region having a score high enough to satisfy a specific condition as the detected image region in which the specific target has been detected. Next,
(a) The position (detection position) of a detection image area related to a detected object, the range of the detection image area (for example, coordinate values of the four corners), and further, the detection speed as a change in the detection position between successive image data are set as the object position information of (a) above,
(b) The image feature amount of the detected image area related to the detected object, calculated using a known image feature extractor (e.g., a CNN (Convolutional Neural Networks) image feature extractor), is used as the object appearance information described above in (i).

The object detector may be, for example, the one described in non-patent document: Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection”, arXiv:2004.10934v1, 2020. Furthermore, it is also possible to determine the target position information (a) and the target appearance information (b) using various other object detectors and image feature extractors known in the field of image recognition technology.

When a stereo camera is used as the camera 103, an object detector such as that described in the non-patent document Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, “SSD: single shot multibox detector”, European Conference on Computer Vision, Computer Vision-ECCV 2016, pp.21-37, 2016 may be applied to the stereo camera image data, which is the generated environmental information.

In yet another embodiment, when LiDAR is used instead of the camera 103 and three-dimensional point cloud data is used as environmental information, an object detector for detecting a specific target can be used, for example, as disclosed in non-patent document: Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp.77-85, 2017.

Furthermore, in this embodiment in which image data is used as environmental information, instead of the above image feature extractor, a feature extractor described in, for example, non-patent document: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep Residual Learning for Image Recognition”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp.770-778, 2016 may be used.

Incidentally, in this embodiment, the target detection information determination unit 113 stores the determined "target position information" and "target appearance information" for each detected specific target for a specific period of time in the past, and can provide this information to the tracking target determination and management unit 115 as appropriate.

<Terminal communication information determination process>
1, the terminal communication information determination unit 111 acquires communication history information from the communication history information storage unit 102, and determines "terminal communication information" related to communication with the terminal 2 based on the communication history information. In this embodiment, the "terminal communication information" is
The data is either one or both of (α) time series data of the received signal power (RSSI) of the radio signal received from terminal 2 and (β) time series data of information regarding the presence or absence of a communication connection with terminal 2.

For the above (α), the terminal communication information determination unit 111 may generate time series data of a received signal power variable E_t indicating the signal power value at time t of the radio signal received from the terminal 2. For the above (β), the terminal communication information determination unit 111 may also generate time series data of time series information on whether or not a communication connection with the terminal 2 is established at each time point, that is, for example, time series data of a binary communication connection presence/absence variable B_t (∈{-1, 1}) that takes a value of 1 if a communication connection is established at time t and a value of -1 if a communication connection is not established. Here, taking into account the accuracy of the degree of match (degree of communication compatibility) with the "target communication information" calculated later, it is preferable that the "terminal communication information" includes at least the time series data of the above (α) received signal power variable E_t.

Incidentally, in this embodiment, the terminal communication information determination unit 111 stores the "terminal communication information" determined for each terminal 2 being tracked for a predetermined period of time in the past, and can provide it to the tracking target determination and management unit 115 as appropriate. In addition, it is also preferable that the terminal communication information determination unit 111 acquires, for example, an IMSI (International Mobile Subscriber Identity) from each terminal 2 present under the control of the base station 1, compiles the determined "terminal communication information" for each terminal 2 identified by the IMSI, and performs subsequent processing separately for each terminal 2.

<Target communication information determination process>
Also in the functional block diagram of Figure 1, the target communication information determination unit 112 determines "target communication information", which is information related to the communication state that will be realized between the detected target and terminal 2, from image data (environmental information) that may include a specified target and is obtained from the environmental information accumulation unit 104.

(When using received signal strength data as "target communication information")
The “target communication information” determined by the target communication information determination unit 112 is
(α') Information on the strength of radio waves received from a detected object, if the detected object includes terminal 2 (received radio wave strength data)
In this case, the target communication information determination unit 112 determines, based on the detection position of the target detected from the image data (by the target detection information determination unit 113) and the detection positions of "other targets" (which may cause interference), as shown on the right side of FIG. 3, which will be used later in the description,
(a) determining a straight line segment connecting the detected object and the (antenna of) base station 1;
(b) calculating occlusion degree information relating to the degree to which the “other object” occludes a predetermined area including the line segment;
(c) From the calculated obstruction degree information, received radio wave intensity data that would be received from the detected target if the detected target includes terminal 2 is calculated as "target communication information."

Specifically, the above-mentioned (b) information on the degree of occlusion is calculated as follows: First, as shown on the right side of Fig. 3 (used in the following explanation), a predetermined size (a cross-sectional area at the detection position having a predetermined area) is preset for the "other object" that may cause an obstacle according to its type (class), and then a rotating body area of a predetermined size including the above-mentioned straight line segment as the rotation axis is set, and at a certain point in time t (= t0 - M, t0 - M + 1, ..., t0 - 1, t0),
(b1) When any of the "other objects" has a cross-sectional area that is not included in the above-mentioned rotating body area at its detection position (the cross-sectional area of the other object does not overlap with the rotating body area), the value is 1.
(b2) In the case where a part of the cross-sectional area of any of the "other objects" is included in the above-mentioned rotating body area at the detection position (the cross-sectional area of the other object and the rotating body area have an overlapping part), the following formula (1) R = (S _a ∩ _{S b} ^- ) / S _a
The time series data of the variable E'_t having an R value calculated by the above formula can be used as the occlusion degree information.

In the above formula (1), S _a is a set (e.g., a set of unit areas forming the area) representing the cross-sectional area (perpendicular to the axis of rotation) at the detection position of the "other object" in the above rotating body area, and S _b is a set (e.g., a set of unit areas forming the area) representing the cross-sectional area at the detection position of the "other object." Also, S _b ^- is the complement of S _b , and "/" is an operator indicating that the area value of the cross-sectional area related to the numerator set (e.g., the number of unit areas) is divided by the area value of the cross-sectional area related to the denominator (e.g., the number of unit areas).

Next, the target communication information determination unit 112 determines the time series data of this variable E'_t (or the variable E'_t multiplied by a predetermined constant) as "target communication information" as received radio wave intensity data. Hereinafter, the variable E'_t will be referred to as a received power equivalent variable to distinguish it from the above-mentioned received signal power variable E_t.

In addition, when there are two or more "other objects" that have the above R value, _Sb may be the area formed by the entire cross-sectional areas of these "other objects," that is, the area expressed as the union of these cross-sectional areas. Furthermore, the above-mentioned revolution body area may be a cylindrical area, or may be one that can be formed by rotating a specified curve around a rotation axis, such as an ellipsoid.

As a further modification, the target communication information determination unit 112 may use a trained "target radio wave information estimation model" that outputs information related to the strength of received radio waves in response to input image data to derive information related to the strength of radio waves that will be received from the detected target if the detected target includes terminal 2 from the acquired image data, and determine the information related to the strength of radio waves as the "target communication information."

Here, as the above-mentioned "target radio wave information estimation model", a publicly known trained model such as that disclosed in Non-Patent Document 1 can be adopted. Specifically, image data at a certain point in time t (= t0-M, t0-M+1, ..., t0-1, t0) is input to this trained model, which outputs an estimated value of the received signal power. This estimated value is set as the value of the received power equivalent variable E'_t, and the time series data of this received power equivalent variable E'_t is set as the "target communication information".

(When using communication connection availability data as the "target communication information")
The “target communication information” determined by the target communication information determination unit 112 is
(β') Information regarding whether or not a communication connection with a detected target is possible (communication connection possible data)
In this case, the target communication information determining unit 112 determines, based on the detection position of the target detected from the acquired image data and the detection positions of the detected "other targets" that may cause a problem,
(a) determining a straight line segment connecting the detected object and the (antenna of) base station 1;
(b) calculating communication connection availability data indicating whether or not the “other object” is located on the line segment or within a predetermined area including the line segment, or whether or not the other object is blocking the line segment;
(c) It is also preferable to determine the calculated communication connection availability data as the "target communication information."

More specifically, as the communication connection availability data of (b) above, a binary communication connection availability variable that takes a value of -1 if the detected position of any of the "other targets" is located on the above line segment or within a rotating body area of a predetermined size that includes this line segment as the rotation axis at a certain time point t (==t0-M, t0-M+1, ..., t0-1, t0), and takes a value of 1 otherwise:
(2) B'_t∈{－1, 1}
The time series data may be generated and used as the "target communication information." Here, the above-mentioned rotating body region may be a cylindrical region, or may be one that can be formed by rotating a predetermined curve around a rotation axis, such as an ellipsoid.

As a modification, the target communication information determination unit 112 may preset a predetermined size (cross-sectional area at the detection position) for the "other target" according to its type (class), and define B'_t in (2) above as a communication connection feasibility variable that takes a value of -1 if any of the "other targets" is blocking the above line segment with that size (cross-sectional area) at its detection position (the above line segment penetrates the cross section of the "other target"), and a value of 1 if this is not the case.

The above is the "target communication information"
(α') Time series data of the received power equivalent variable E'_t (received signal strength data), or (β') Time series data of the communication connection availability variable B'_t (communication connection availability data)
Here, when the "terminal communication information" includes both of the above (α) and (β), it is also preferable that the "target communication information" includes both of the above (α') and (β'). Furthermore, taking into consideration the accuracy of the degree of match (degree of communication compatibility) with the "terminal communication information" calculated later, it is preferable that the "target communication information" includes time series data of the above received power equivalent variable E'_t of (α').

Incidentally, in this embodiment, the target communication information determination unit 112 stores the determined "target communication information" for each detected target for a specified period of time in the past, and can provide it to the tracking target determination and management unit 115 as appropriate.

<Occlusion Determination Processing>
Also in the functional block diagram of Figure 1, the occlusion determination unit 114 determines whether or not occlusion has occurred (or is predicted to occur) at each point in time for a target (e.g., a "person") related to the terminal 2 being tracked, based on information from the target detection information determination unit 113, and outputs this determination result to the tracking target determination and management unit 115, which will be described in detail later.

Figure 2 is a schematic diagram illustrating two aspects of the occlusion determination process in the occlusion determination unit 114.

According to the determination mode first shown in FIG. 2(A), the occlusion determination unit 114 obtains from the target detection information determination unit 113 the detection position and detection speed (target position information) of a target (e.g., a "person") related to the tracked terminal 2 at a time point earlier than time point t0 (e.g., time point t0-1), and determines the predicted position of this target ("person") at time point t0 from this information.

Next, if the occlusion determination unit 114 detects another specified object (e.g., a "person") at time t0 within a specified range including the determined predicted position (e.g., an image area corresponding to a real-space circular range of a specified size centered on the predicted position), it can determine that occlusion has occurred at this time t0.

Next, in the determination mode shown in FIG. 2(B), the occlusion determination unit 114 first determines the predicted position of the target (e.g., a "person") associated with the tracked terminal 2 at time t0 in the same manner as in the determination mode shown in FIG. 2(A) described above.

Next, the occlusion determination unit 114 can determine that occlusion has occurred at time t0 if no object, including this object, is detected at time t0 within a predetermined range including the determined predicted position (for example, an image area corresponding to a real-space circular range of a predetermined size centered on the predicted position). In this case, it is highly likely that occlusion has occurred such that the tracked object is completely blocked.

As yet another mode of occlusion determination, when the target detection information determination unit 113 is unable to determine one or both of the target position information and target appearance information of a tracked target (e.g., a "person") from image data at a certain time t0, the occlusion determination unit 114 can receive a report to that effect and determine that occlusion has occurred at this time t0.

Furthermore, as yet another determination mode, the occlusion determination unit 114 can determine that occlusion has occurred at time t0 if the target detection information determination unit 113 detects a tracked target (e.g., a "person") and another specified target (e.g., a "person") from image data at a certain time t0 and the distance between their detected positions (e.g., the distance in image space equivalent to a specified distance in real space) is less than a specified threshold. In this case, it is highly likely that partial occlusion has occurred to the extent that the detection position can be determined.

<Tracking target determination process/terminal tracking process>
Returning to the functional block diagram of FIG. 1 , the tracking target determination and management unit 115 of this embodiment performs at least the following during a time period in which no occlusion occurs for a detected target (e.g., a “person”) related to tracking and target position information and target appearance information are determined:
(a) "Position matching degree" which is the degree to which the determined target position information matches the predicted target position information predicted from the past target position information, and (b) "Appearance matching degree" which is the degree to which the determined target appearance information matches the past target appearance information.
Based on either one or both (both in this embodiment), it is determined whether or not there is a correspondence between the tracked terminal 2 and the detected target ("person"), and the target ("person") that is determined to be in a corresponding relationship is determined to be the target related to the tracked terminal 2, and terminal tracking processing is performed.

In this embodiment, as will be described later, the weight (λ) is adjusted, but
(c) A "communication correspondence degree" which is the degree to which the terminal communication information determined for the tracked terminal 2 corresponds to the object communication information determined for the detected object ("person").
The correspondence relationship determination process may also be performed based on

In addition, in this embodiment, when an occlusion occurs for a detected target related to tracking, in which the detection position cannot be determined, and is then resolved, the tracking target determination and management unit 115 always uses the "degree of communication compatibility" described above in (c) to perform a correspondence relationship determination process (tracking target determination process) between the terminal 2 and the target ("person"), for example, based only on this "degree of communication compatibility."

This is because, in order to calculate the "degree of positional agreement" in (A) above or the "degree of appearance agreement" in (B) above at the time when an occlusion with an undetermined detection position occurs and is resolved, "(past) target position information" and "(past) target appearance information" for the time period in which this occlusion occurred are required, but this information cannot be obtained precisely because such an occlusion occurred. In other words, in this case, neither the "degree of positional agreement" in (A) above nor the "degree of appearance agreement" in (B) above can be calculated in the first place.

Below, we will explain the specific correspondence determination process (tracking target determination process) using the above-mentioned "degree of positional match," "degree of appearance match," and "degree of communication compatibility."

Similarly, in the functional block diagram of FIG. 1, the communication cost calculation unit 115a, the position cost calculation unit 115b, and the appearance cost calculation unit 115c of the tracking target determination and management unit 115 calculate the position match degree, the appearance match degree, and the communication compatibility degree of the detected target as described above, respectively, and then:
- "Location cost" Cgeo, which is a monotonically decreasing function of the degree of location agreement;
"Appearance cost" Capp, which is a monotonically decreasing function of the degree of appearance match, and "Communication cost" Cradio, which is a monotonically decreasing function of the degree of communication compatibility
Calculate.

The smaller these costs are, the higher the possibility that the terminal 2 and the detected object are in a corresponding relationship. Here, the monotonically decreasing function of each degree may be, for example, a function that reverses the sign of the degree (multiplying the degree by -1), or a function that takes the reciprocal of the degree. Note that below, the position cost Cgeo and the appearance cost Capp may be collectively referred to as the "environmental information cost."

Next, the tracking target determination and management unit 115 of this embodiment, which will be described in detail later,
(a) for each of the detected objects (e.g., a person), a tracking cost Cost is calculated, which is a weighted sum of the calculated location cost Cgeo, appearance cost Capp, and communication cost Cradio by a "weighting factor λ" that may be zero, where the "weighting factor λ" is applied to the communication cost Cradio;
(b) The target ("person") for which the tracking cost is small enough to satisfy a predetermined condition, for example the minimum tracking cost, is determined to be the target associated with the terminal 2 being tracked.

Below, we will explain a specific example of how to calculate the communication cost Cradio, the location cost Cgeo, and the appearance cost Capp required to calculate the tracking cost Cost.

(Communication cost calculation)
FIG. 3 is a schematic diagram for explaining a specific example of the communication cost calculation process according to the present invention.

In the specific example of calculation of the communication cost Cradio shown in FIG. 3, the communication cost calculation unit 115a first calculates:
(a) time series data of the received signal power variable E_t of the tracked terminal 2 (as terminal communication information) determined by the terminal communication information determination unit 111;
(b) The degree of coincidence Cr (degree of communication compatibility) with the time series data of the received power equivalent variable E'_t relating to the detected target (as target communication information) determined by the target communication information determination unit 112 is calculated as the cross-correlation between the two (Σ _t E_t * E'_t) in a specified time interval.

Incidentally, when time series data of the communication connection existence variable B_t and the communication connection availability variable B'_t are adopted as the terminal communication information and the target communication information, respectively, the degree of agreement Cr (degree of communication compatibility) is calculated as the cross-correlation ( _{ΣtB_t} *B'_t) between them in a predetermined time interval. The degree of agreement Cr can also be calculated as the weighted sum of the cross-correlation ( _{ΣtE_t} *E'_t) and the cross-correlation ( _{ΣtB_t} *B'_t).

In this specific example, the communication cost calculation unit 115a calculates the degree of match Cr at time t0 when it is determined that no occlusion has occurred such that the detection position cannot be determined (the detection position (object position information) of the object detected by the object detection information determination unit 113 is stored), as the cross-correlation between the received signal power variable E_t and the received power equivalent variable E'_t in a predetermined past time interval (the time interval of t=t0-M, t0-M+1, ..., t0-N), using the following equation (3): Cr=C0·Σ _t=t0-M ^t0-N E_t*E'_t
Here, in the above formula (3), Σ _t=t0-N ^t0-N is an operator indicating the summation for time t (=t0-M, t0-M+1, ..., t0-N). Furthermore, C0 is a coefficient for adjusting the Cr value to 1 in the case of a perfect match. Furthermore, M is an integer set so that time t0-M is a time in the past that is sufficient to obtain a cross-correlation (for example, a time 10 seconds ago).

Furthermore, N is an integer equal to or greater than 1 such that time t0-N is the latest (closest to time t0) time at which the detection position (target position information) of the target detected (in the target detection information determination unit 113) is stored as viewed from time t0. Therefore, when N=1 and the detection position of the target detected at time t0-1, immediately before time t0, is stored in the target detection information determination unit 113, that is, when no occlusion with an undetermined detection position has recently occurred, the degree of match Cr at time t0 is expressed by the following formula (3'): Cr=C0·Σ _t=t0-M ^t0-1 E_t*E'_t
It will be calculated using:

In contrast, when N>1, the detection position of the detected object in the time interval from time (t0-N+1) to time (t0-1) is not determined due to the occurrence of occlusion to an extent that detection processing is not possible, and as a result, the received power equivalent variable E'_t is not calculated in the time interval in which such occlusion occurred. Therefore, in this case, the degree of match Cr expressed by the above formula (3) cannot be calculated as it is.

Then, the position interpolation unit 112a (FIG. 1) of the target communication information determination unit 112, in response to the fact that the detection position in the time interval from time (t0-N+1) to time (t0-1) has not been determined and stored (the received power equivalent variable E'_t has not been calculated), determines the position of the detected target at each time point (time point t0-N+1, time point t0-N+2, ..., time point t0-1) in the time interval in which such occlusion occurred by interpolation processing. Specifically, the detection position at the acquired time point (t0-N) (or the detection position before the acquired time point t0-N) and the detection position at the acquired time point t0 are complemented by a known interpolation method, such as linear interpolation or spline interpolation, to determine the positions at time points t0-N+1, t0-N+2, ..., and t0-1.

Then, the target communication information determination unit 112 (Fig. 1) determines that the detected target is present at the position at each time point determined as the interpolation position (time point t0-N+1, time point t0-N+2, ..., time point t0-1), and determines the received power equivalent variable E'_t (target communication information) at each of the times. This enables the communication cost calculation unit 115a to calculate the degree of match Cr using the above formula (3) even at time point t0 when the occlusion that occurred is resolved.

Furthermore, in this specific example, the communication cost calculation unit 115a inverts the sign of the degree of match Cr calculated as described above (multiplies the degree of match Cr by -1) and determines this sign inversion result (-Cr) as the communication cost Cradio. Therefore, the larger this communication cost Cradio is, the more likely it is that the detected target does not correspond to the terminal 2 being tracked.

Incidentally, in Figure 3, the communication cost Cradio of target A is smaller than the communication cost Cradio of target B because the waveform of the received power equivalent variable E'_t is more similar to the waveform of the received signal power variable E_t. Therefore, looking at the communication cost Cradio alone, it can be said that target A is more likely to correspond to the tracked terminal 2.

(Location cost calculation, appearance cost calculation)
FIG. 4 is a schematic diagram for explaining a specific example of the calculation process of the position cost and the appearance cost according to the present invention.

First, in the specific example of calculating the location cost Cgeo shown in FIG. 4A, the location cost calculation unit 115b calculates the following for a detected object:
(a) The range (e.g., coordinate values of the four corners) of the determined detected image area (as target position information),
(b) A degree of match (degree of positional match) Cg between the range of the detected image area predicted (as predicted target position information) from the past detected position (as target position information) is calculated.

More specifically, at time t0 when it is determined that no occlusion has occurred such that the detection position cannot be determined (the detection position (target position information) of the target detected by the target detection information determination unit 113 is stored), the position cost calculation unit 115b
(a) the range S1 of the detected image area for the detected object at this time t0;
(b) From the detection position at the latest (closest to time t0) time (t0-N) for which the detection position (target position information) of the target detected from this time t0 (in the target detection information determination unit 113) is stored, the area of the overlapping portion with the range S2 of the predicted detection image area at time t0 predicted using the detection speed at the same time (t0-N) is taken as S1∩S2, and further, the area occupied by the entire range S1 of the detected image area and the range S2 of the predicted detection image area (a) and (b) is taken as S1∪S2, and the following equation (4) Cg=(S1∩S2)/(S1∪S2) is obtained.
The degree of match (degree of positional match) Cg at time t0 is calculated by the above formula.

As a modification, the position cost calculation unit 115b can also determine the degree of closeness between the detected position of the detected object at time t0 and the predicted position predicted from the detected position at the latest time (t0-N) (using the detection speed at time t0-N) as the degree of match (degree of position match) Cg at time t0, for example, the inverse of the distance between the two positions.

In any case, by using the calculation process for the degree of coincidence Cg described above, it is possible to calculate the degree of coincidence Cg at time t0 even when there is a time point where occlusion occurs and the detection position cannot be determined, i.e., when N>1.

Here, in this embodiment, the location cost calculation unit 115b inverts the sign of the degree of match Cg calculated in this manner (multiplies the degree of match Cg by -1) and determines this sign inversion result (-Cg) as the location cost Cgeo. Therefore, the larger this location cost Cgeo is, the more likely it is that the detected target does not correspond to the device 2 being tracked.

Next, in the specific example of the appearance cost Capp calculation shown in FIG. 4B, the appearance cost calculation unit 115c calculates the appearance cost Capp of the detected object as follows:
(a) a determined image feature amount of the detected image region (as object appearance information);
(b) The degree of match (degree of appearance match) Ca with the image feature amount of the past detected image area (as target appearance information) is calculated.

More specifically, at time t0 when it is determined that no occlusion has occurred such that the detection position cannot be determined (when the object appearance information of the object detected by the object detection information determination unit 113 is stored), the appearance cost calculation unit 115c
(a) Image feature amounts derived using a known CNN image feature extractor for the detected image region of the detected object at this time point t0;
(b) The similarity between the detected image area of the detected object at the latest (closest to time t0) time t0-N for which object appearance information of the detected object is saved (in the object detection information determination unit 113) from this time t0 and the image feature derived using the same CNN image feature extractor as in (a) above is calculated using a publicly known method for calculating similarity between features, for example the method described in Non-Patent Document 2, and the calculated similarity is regarded as the matching degree Ca at time t0.

By using this calculation process for the degree of coincidence Ca, it is possible to calculate the degree of coincidence Ca at time t0 even when there is a time point where an occlusion occurs and the detection position cannot be determined, i.e., when N>1.

Here, in this embodiment, the appearance cost calculation unit 115c inverts the sign of the matching degree Ca calculated as described above (multiplies the matching degree Ca by -1) and determines this sign inversion result (-Ca) as the appearance cost Capp. Therefore, the larger this appearance cost Ca is, the more likely it is that the detected target does not correspond to the device 2 being tracked.

The calculation process for the communication cost Cradio, the location cost Cgeo, and the appearance cost Capp has been explained above. Below, we will explain the calculation process for the tracking cost Cost using these costs, and the correspondence determination process (tracking target determination process) using the calculated tracking cost Cost.

(Tracking cost calculation, tracking target determination process)
FIG. 5 is a schematic diagram for explaining a specific example of the tracking cost calculation process and the tracking target determination process according to the present invention.

As shown in FIG. 5, in this embodiment, the tracking target determination and management unit 115:
(a) For each of the detected targets (targets A and B in FIG. 5 ), a tracking cost Cost is calculated, which is a weighted sum of the calculated location cost Cgeo, appearance cost Capp, and communication cost Cradio, using a “weighting factor λ” that may be zero, where the “weighting factor λ” is applied to the communication cost Cradio;
(b) The target ("person") for which the tracking cost is small enough to satisfy a predetermined condition, for example the minimum tracking cost, is determined to be the target associated with the terminal 2 being tracked.

In this embodiment, the tracking cost Cost is calculated by the following formula (5): Cost = (1 - λ) (α Cgeo + β Capp) + λ Cradio
In the above formula (5), the weighting coefficient λ is a value not less than 0 and not more than 1. In addition, α (≧0) and β (≧0) satisfy α+β=1, and are weighting coefficients that determine the ratio between the position cost Cgeo and the appearance cost Capp, and may take, for example, a preset value.

As a modification, the weighting coefficient (1-λ) of the first term on the right-hand side of the above formula (5) (environmental information cost) can be changed to another weighting coefficient, for example a constant. Furthermore, although this will result in accuracy problems, it is also possible to use only one of the position cost Cgeo and the appearance cost Capp in the above formula (5). For example, it is possible to use only the position cost Cgeo as the environmental information cost. Incidentally, when the tracking cost Cost described above is greater than a predetermined threshold C_th, the correspondence determination process may be set not to be performed.

Here, a correspondence relationship determination process (tracking target determination process) using the tracking cost Cost in the above formula (5) in the case where occlusion occurs for a detected target will be described. When such occlusion is resolved, in this embodiment, the weight determination unit 115d of the tracking target determination and management unit 115 determines a weighting factor λ that is a monotonically increasing function of the maximum value (maxCr) of the degree of match Cr (degree of communication compatibility) calculated for each detected target. That is, the value of the weighting factor λ that is preset as such a monotonically increasing function is determined by substituting the maximum value (maxCr) of the calculated degree of match Cr. For example, the following formula (6) λ=λ ₁ when maxCr>K.
When maxCr≦K, λ＝λ ₀ (λ ₀ ＜λ ₁ )
The weighting factor λ may be determined by:

Here, K is a predetermined threshold value that defines the boundary as to whether the communication cost Cradio is reflected more significantly in the tracking cost Cost. For example, in the above formula (6), if _λ1 = 1 and _λ0 = 0, the tracking cost Cost will be the communication cost Cradio itself or the environmental information cost (α·Cgeo + β·Capp) itself, depending on whether the maximum value (maxCr) of the matching degree Cr (degree of communication compatibility) exceeds the K value.

In another embodiment, the weight determining unit 115d uses the following equation (7) using a sigmoid function: λ=1/(1+e ^{−a(maxCr+k)} )
The weighting coefficient λ may be determined by the following equation: where a is the gain of the sigmoid function. Also, k is an offset value for the maximum value (maxCr) of the degree of agreement Cr, and is set in advance. When using the above equation (7), it is preferable that the communication cost calculation unit 115a rounds up a value less than 0 in the cross-correlation (e.g., Σ _t E_t*E'_t) to 0 as the degree of agreement Cr.

The degree of match Cr (degree of communication compatibility) can be determined from the communication state of the terminal 2 and the estimated communication state of the target to be determined (target A or target B) obtained from environmental information (e.g., image data), regardless of whether the target to be determined (target A or target B in FIG. 5) is detected during the process. In other words, the degree of match Cr (degree of communication compatibility) is likely to be used as a more reliable judgment criterion even when occlusion occurs. As a result, as described above, in this embodiment, by determining the weighting coefficient λ that is a monotonically increasing function of the maximum value (maxCr) of the degree of match Cr (degree of communication compatibility), a more appropriate (suitable for the actual situation) tracking cost Cost can be calculated and determined even when occlusion occurs. This also makes it possible to improve the accuracy of terminal tracking.

Of course, when the degree of matching Cr (degree of communication compatibility) is considerably low, if a tracking target determination process is performed that places emphasis on the communication cost Cradio or takes only the communication cost Cradio into consideration, there is a concern that the accuracy of tracking the terminal may decrease. In response to this, the tracking target determination and management unit 115 of this embodiment determines a smaller weighting coefficient λ in such a situation, and is able to avoid or minimize the decrease in the accuracy of tracking the terminal. On the other hand, when the degree of matching Cr (degree of communication compatibility) is higher, conversely, a larger weighting coefficient λ is determined and a tracking target determination process that is based more on the communication cost Cradio is performed, thereby making it possible to maintain and improve the accuracy of tracking the terminal even when occlusion occurs.

Here, the weighting factor λ, which is a monotonically increasing function of the maximum value (maxCr) of the degree of match Cr (degree of communication compatibility) for each detected target, as in the above formula (6) or formula (7), may be set to be used for a predetermined period of time immediately after the occlusion is resolved. In this case, outside this predetermined period, the weighting factor λ can be set to a predetermined value (e.g., a small value such as 0.1) or zero. For example, if the weighting factor λ is set to zero, in a situation where no occlusion occurs, the correspondence determination process (tracking target determination process) is performed based on the environmental information cost (Cgeo, Capp) that is determined to be highly reliable. Note that it is also possible to set the weighting factor λ as in the above formula (6) or formula (7) at all times, not just when the occlusion is resolved.

Next, an embodiment in which the degree of match Cg (degree of positional match) is also taken into consideration when determining the weighting coefficient λ will be described. In this case, when occlusion is resolved, the weighting determination unit 115d determines a weighting coefficient λ that is a monotonically increasing function of both the maximum value (maxCr) of the degree of match Cr (degree of communication compatibility) of each detected object and the maximum value (maxCg) of the degree of match Cg (degree of positional match) of each detected object (when one is a constant, the other is a monotonically increasing function). For example,
(a) When the maximum value (maxCg) of the degree of match Cg (degree of positional match) of each detected object is greater than a predetermined threshold Kg (maxCg>Kg), the above formula (6) or (7) is followed, but on the other hand,
(b) If the maximum value (maxCg) of the match degree Cg (degree of positional match) of each detected object is equal to or less than a predetermined threshold value Kg (maxCg≦Kg) and the maximum value (maxCr) of the match degree Cr (degree of communication compatibility) of each detected object is also equal to or less than a predetermined threshold value K'(maxCr≦K' (＞K)), the value is zero or a small value less than a predetermined value (here, even if maxCg≦Kg, if maxCr>K', the above formula (6) or (7) is followed).
In another embodiment, it is also possible to determine a weighting factor λ that is a sigmoid function for both the maximum value (maxCr) of the degree of coincidence Cr and the maximum value (maxCg) of the degree of coincidence Cg in accordance with the above formula (7) (when one is a constant, the other is a sigmoid function).

By determining such a weighting coefficient λ, the proportion of the communication cost Cradio in the tracking cost Cost can be set to zero or reduced in a situation where the degree of match Cg (degree of positional match) is considerably low. Here, in such a situation, even if the degree of match Cr (degree of communication compatibility) is high, since the target detection position is used to calculate the degree of match Cr in the first place, it is determined that the reliability of the communication cost Cradio calculated from the degree of match Cr (degree of communication compatibility) has decreased. Therefore, by determining the weighting coefficient λ as described above and setting to zero or reducing the proportion of the communication cost Cradio in the tracking cost Cost, it is possible to maintain and improve the accuracy of terminal tracking.

To summarize the correspondence determination process described above, when the degree of match Cr (degree of communication compatibility) is small enough to satisfy a predetermined condition (in this embodiment, when the maximum value maxCr of the degree of match Cr of each detected target is equal to or less than a predetermined threshold value), the tracking target determination and management unit 115 performs the correspondence determination process without being based on the degree of match Cr (degree of communication compatibility) when the degree of match Cg (degree of positional match) is small enough to satisfy a predetermined condition (in this embodiment, when the maximum value maxCg of the degree of match Cg of each detected target is equal to or less than a predetermined threshold value); alternatively, the smaller the degree of match Cg (degree of positional match) is (in this embodiment, the smaller the maximum value maxCg of the degree of match Cg of each detected target is), the smaller the degree based on the degree of match Cr (degree of communication compatibility) is to be used to perform the correspondence determination process.

As a preferred modification, the tracking target determination and management unit 115
(a) When the degree of match Cg (degree of positional match) is smaller than a predetermined condition (for example, when the maximum value maxCg of the degree of match Cg of each detected object is equal to or smaller than a predetermined threshold), a correspondence relationship determination process is performed based on at least the degree of match Cg (degree of positional match) and the degree of match Ca (degree of appearance match), and at that time, it is preferable to increase the degree based on the degree of match Ca (degree of appearance match), specifically, to increase the value of the weighting coefficient β in the above formula (5) (i.e., to decrease the value of the weighting coefficient α (= β-1)), or
(b) A correspondence determination process is performed based on at least the degree of match Cg (degree of positional match) and the degree of match Ca (degree of appearance match), and the smaller the degree of match Cg (degree of positional match) is (for example, the smaller the maximum value maxCg of the degrees of match Cg of each detected object is), the greater the degree based on the degree of match Ca (degree of appearance match); specifically, it is preferable to increase the value of the weighting coefficient β.

In addition, such adjustment of the weighting factor β based on the degree of match Cg (degree of positional match) can be performed when the correspondence determination process is performed based on at least the degree of match Cg (degree of positional match) and the degree of match Ca (degree of appearance match), including when it is also based on the degree of match Cr (degree of communication compatibility).

In any case, when the degree of match Cg (degree of position match) is considerably low, it is determined that the reliability of not only the communication cost Cradio calculated from the degree of match Cr (degree of communication compatibility) but also the value of the position cost Cgeo itself calculated from this degree of match Cg is reduced. Therefore, as described above, by increasing the value of the weighting coefficient β and lowering the weight of the position cost Cgeo (increasing the weight of the appearance cost Capp calculated from the degree of match Ca), it is possible to maintain and improve the accuracy of terminal tracking.

Also according to FIG. 5, the tracking target determination and management unit 115 calculates the position costs Cgeo_A and Cgeo_B, the appearance costs Capp_A and Capp_B, and the communication costs Cradio_A and Cradio_B of each of the target A ("person") and the target B ("person") at time t0, and calculates the tracking costs Cost_A and Cost_B at time t0 according to the following equation (8): Cost_A=(1-λ)·(α·Cgeo_A+β·Capp_A)+λ·Cradio_A
(9) Cost_B=(1−λ)・(α・Cgeo_B+β・Capp_B)+λ・Cradio_B
Here, the weighting coefficient λ and the weighting coefficients α and β are determined by the determination process described above.

Next, in this embodiment, the tracking target determination and management unit 115 determines target A ("person") with the smallest tracking cost (the smaller tracking cost in the case of Figure 5) as the target related to the terminal 2 being tracked, and continues tracking that terminal 2. Incidentally, in Figure 5, there is no difference between the appearance cost Capp for both, but the location cost Cgeo and communication cost Cradio are clearly smaller for target A ("person").

As described above, the weighting coefficient λ and the weighting coefficients α and β are set to more suitable values, particularly by taking into account the degree of coincidence Cr (degree of communication compatibility), even when occlusion occurs with respect to a detected object whose correspondence with the terminal 2 should be determined. As a result, even if occlusion occurs with respect to the terminal 2 being tracked, it is possible to continue tracking the terminal 2 more appropriately (in FIG. 5, of the objects A and B, the one that is holding or carrying the terminal 2 is determined to be the target to be tracked).

Returning to the functional block diagram of FIG. 1, the tracking target determination and management unit 115 stores and manages the results of the tracking target determination process (correspondence determination process) for each tracked terminal 2, which have been described above with reference to various embodiments and specific examples, in the tracking information storage unit 105, and outputs the results to the communication control unit 116 continuously, periodically, or as required. Here, the results of this tracking target determination process (correspondence determination process) can be time-series data in which the location and image features at each time point are linked for each terminal ID of the terminal 2, for example, and are advantageous in that the adverse effects of occlusion (for example, loss, ID switching, etc.) are avoided or reduced.

By using the results of the tracking target determination process (correspondence determination process) of such a suitable terminal 2, the communication control unit 116 can more accurately grasp the situation, including the current location and communication path, of the connected terminal 2. This also makes it possible to continue the communication connection with this terminal 2 or immediately resume it, even if a sudden drop in the reception power from the connected terminal 2 occurs.

As described above in detail, according to the present invention, when occlusion occurs, a process for determining whether or not the tracked terminal and the detected target are in a correspondence relationship can be performed based on the degree of communication correspondence between the terminal and the detected target. As a result, even when occlusion occurs for the tracked terminal, it is possible to continue tracking the terminal more appropriately.

Furthermore, the terminal tracking process according to the present invention can be applied to various fields and situations related to 5G, which is currently being introduced, and future mobile communication systems that use shorter wavelength radio waves. For example, it can solve the problem of communication failure caused by obstructions to the communication path, notify and warn automobiles equipped with terminals of approaching people or other vehicles, and even determine the relationship between the pages viewed by a user determined to be in correspondence with a certain terminal and the user's movement path, which can be used for marketing and management.

In addition, by popularizing information and communication systems including the terminal tracking device of the present invention, which can stably realize a huge number of high-capacity communication connections, it will be possible to widely deploy mobile and cell phone communication services that strengthen people's "connections" and empower communities. Furthermore, it is believed that information and communication systems including the terminal tracking device of the present invention will be particularly effective in urban areas where the global population is growing. In other words, the present invention can contribute to the achievement of Goal 9 "Develop resilient infrastructure, promote sustainable industrialization and foster innovation" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the United Nations-led Sustainable Development Goals (SDGs).

In the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical ideas and viewpoints of the present invention can be easily made by a person skilled in the art. The above description is merely an example and is not intended to restrict the present invention in any way. The present invention is limited only by the scope of the claims and their equivalents.

1. Base station (terminal tracking device, communication relay device)
101 Communication interface 102 Communication history information storage unit 103 Camera (sensor)
104 Environment information accumulation unit 105 Tracking information storage unit 111 Terminal communication information determination unit 112 Target communication information determination unit 112a Position interpolation unit 113 Target detection information determination unit 114 Occlusion determination unit 115 Tracking target determination and management unit 115a Communication cost calculation unit 115b Position cost calculation unit 115c Appearance cost calculation unit 115d Weight determination unit 116 Communication control unit 2 Terminal

Claims (11)

an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, the environmental information being generated by the sensor, and determining object position information which is information related to a detection position of the detected object;
a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information;
a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position coincidence degree, which is a degree of coincidence between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal;
The tracking target determination means, at least when occlusion is resolved for the detected target, makes the determination based on the degree of communication compatibility rather than on the degree of positional agreement when the degree of communication compatibility is large enough to satisfy a specified condition, or, the greater the degree of communication compatibility is or when the degree of communication compatibility is large enough to satisfy a specified condition, makes the determination by increasing the degree based on the degree of communication compatibility. The terminal tracking device according to claim 1, characterized in that the tracking target determination means performs the determination without basing the communication compatibility degree on the degree of communication compatibility when the degree of positional agreement is small enough to satisfy a predetermined condition, or performs the determination by reducing the degree based on the communication compatibility degree as the degree of positional agreement is small. The object detection information determination means also determines object appearance information, which is information related to a detected appearance of the detected object, from the environmental information;
The terminal tracking device described in claim 1 or 2, characterized in that when the tracking target determination means makes the determination based at least on the degree of positional agreement, it also makes the determination based on the degree of appearance agreement, which is the degree of agreement between the target appearance information and past target appearance information of the target determined to be related to the terminal being tracked. The tracking target determination means
When the degree of positional agreement is smaller than a predetermined condition, the determination is made based on at least the degree of appearance agreement and the degree of positional agreement, and at that time, the degree based on the degree of appearance agreement is made larger, or
4. The device according to claim 3, wherein the determination is made based on at least the degree of appearance match and the degree of position match, and the smaller the degree of position match, the greater the degree based on the degree of appearance match. The terminal tracking device according to any one of claims 1 to 4, characterized in that the target communication information determination means determines the target communication information by assuming that the target is present at a position interpolated using detection positions determined before and after the occlusion during a time period during which an occlusion occurs and the detection position of the target is not determined. the environmental information is information relating to an image of the environment, and the target position information includes a position and a range within the image of an image area relating to the detected target;
A terminal tracking device as described in any one of claims 1 to 5, characterized in that the degree of positional agreement is the degree of overlap between the range of the detected target and the predicted range in the image of the target determined to be related to the terminal being tracked, or the degree of closeness between the position of the detected target and the predicted position in the image of the target determined to be related to the terminal being tracked. The tracking target determination means
determining a weighting factor that is a monotonically increasing function of the degree of communication capability, the weighting factor including a zero value;
A terminal tracking device as described in any one of claims 1 to 6, characterized in that a tracking cost of the target is calculated as a weighted sum of an environmental information cost including a position cost which is a monotonically decreasing function of the degree of position agreement, and a communication cost which is a monotonically decreasing function of the degree of communication compatibility, wherein the communication cost is correlated with the weighting coefficient and the environmental information cost is correlated with another weighting coefficient which may take on a value of zero, and a target for which the tracking cost is small enough to satisfy a predetermined condition is determined to be the target related to the terminal being tracked. The terminal tracking device according to claim 7, characterized in that the tracking target determination means determines the weighting coefficients that are monotonically increasing functions for both the communication compatibility degree and the position matching degree. an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, generated by a sensor, determining object position information which is information related to a detected position of the detected object, and also determining object appearance information which is information related to a detected appearance of the detected object from the environmental information;
a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information;
a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position matching degree, which is a degree of matching between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, an appearance matching degree, which is a degree of matching between the target appearance information and past target appearance information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of compatibility between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal;
The tracking target determination means, at least when occlusion is resolved for the detected target, makes the determination based on the degree of communication compatibility rather than on the degree of positional agreement when the degree of communication compatibility is large enough to satisfy a specified condition, or, the greater the degree of communication compatibility is or when the degree of communication compatibility is large enough to satisfy a specified condition, makes the determination by increasing the degree based on the degree of communication compatibility. an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, the environmental information being generated by the sensor, and determining object position information which is information related to a detection position of the detected object;
a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information;
a position match degree, which is a degree of match between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the terminal being tracked, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the terminal being tracked and the target communication information, determining whether or not the terminal being tracked and the detected target are in a corresponding relationship, and causing the computer to function as a tracking target determination means for determining the target determined to be in a corresponding relationship as the target related to the terminal being tracked;
The tracking target determination means, when the occlusion is resolved for at least the detected target, makes the determination based on the degree of communication compatibility rather than on the degree of positional agreement when the degree of communication compatibility is large enough to satisfy a specified condition, or makes the determination by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a specified condition. a first step of detecting a predetermined target which may include the terminal from environmental information generated by a sensor and related to an environment in which the terminal may be present, determining target position information which is information related to a detection position of the detected target, and also determining terminal communication information which is information related to a communication state between the terminal and the detected target based on information related to communication acquired from a communication means for communicating with the terminal, and further determining target communication information from the environmental information which is information related to a communication state which will be realized between the detected target and the target if the detected target includes the terminal;
a second step of determining whether or not the tracked terminal and the detected target are in a corresponding relationship based on at least one of a position match degree, which is a degree of match between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracked terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracked terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracked terminal;
A terminal tracking method implemented by a computer, characterized in that in a second step, when occlusion is resolved at least for the detected target, the determination is made based on the degree of communication compatibility rather than on the degree of positional agreement when the degree of communication compatibility is large enough to satisfy a specified condition, or the determination is made by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a specified condition.

Images