JP7476563B2 - OBJECT TRACKING DEVICE, OBJECT TRACKING METHOD, AND OBJECT TRACKING PROGRAM - Google Patents

Description

This disclosure relates to an object tracking device, an object tracking method, and an object tracking program.

There is technology related to control and management of moving objects.

For example, there is a technology that enables both automated and manually driven vehicles to travel smoothly even on a mixed road where both automated and manually driven vehicles can travel (see Patent Document 1). This technology determines whether a manually driven vehicle has stopped in a stop zone based on monitoring information such as the number, driving positions, driving speeds, and stopping positions of vehicles traveling on the mixed road.

There is also technology that makes highly accurate predictions in the event of a communication interruption from a mobile object and shares the predicted dynamic data to enable a seamless handover to other control programs (see Patent Document 2). With this technology, a dynamic map is used to grasp the position of the mobile object, and in the event of a communication interruption, the relevant management control device predicts the vehicle's running state and transmits the predicted data to other management control devices that share the dynamic map.

JP 2017-220129 A JP 2018-106504 A

In Patent Document 1, vehicle identification is performed using infrastructure sensors around the intersection and vehicle-to-vehicle or road-to-vehicle communication. However, no consideration is given to cases where communication is interrupted.

In addition, although Patent Document 2 focuses on communication outages, it does not take into consideration the identification of vehicle types or the use of infrastructure sensors installed within the area.

The present disclosure has been made in consideration of the above circumstances, and aims to provide an object tracking device, method, and program that is robust against temporary interruptions in communication and enables tracking that is capable of identifying the type of object.

To achieve the above object, the object tracking device of the present disclosure includes a detection unit that detects the position of each object from the output of a sensor installed within an area and outputs the detection result, a tracking unit that outputs a first result which is a tracking result obtained by tracking the position of each of a plurality of objects from the detection result, a communication unit that receives, from each of the objects capable of estimating its own position, an estimation result of the self-position estimated by the object, and an integration unit that integrates the object of the first result and the object of the second result by a predetermined correspondence and outputs the estimation result received from each of the objects as a second result.

The object tracking method disclosed herein also causes a computer to execute the following processes: detect the position of each object from the output of a sensor installed within an area, output the detection result, output a first result which is a tracking result obtained by tracking the position of each of a plurality of objects from the detection result, receive an estimated result of a self-location estimated by each object capable of estimating its own location from among the objects, and output the estimated result received from each of the objects as a second result by integrating the objects of the first result and the objects of the second result through a predetermined correspondence.

The object tracking program of the present disclosure causes a computer to execute the following processes: detect the position of each object from the output of a sensor installed within an area; output the detection result; output a first result which is a tracking result obtained by tracking the position of each of a plurality of objects from the detection result; receive an estimated result of a self-location estimated by each object capable of estimating its own location from among the objects; and output the estimated result received from each of the objects as a second result by integrating the objects of the first result and the objects of the second result through a predetermined correspondence.

The object tracking device, object tracking method, and object tracking program disclosed herein can achieve tracking that is robust to temporary communication interruptions and can identify the type of object.

1 is a block diagram showing a configuration of an object tracking system according to a first embodiment. FIG. 2 is a block diagram showing a hardware configuration of the object tracking device. FIG. 1 is a diagram showing an example of an arrangement of infrastructure sensors in a controlled area. FIG. 13 is a diagram illustrating an example in which the number of detection units is the same as the number of sensors. FIG. 13 is a diagram illustrating an example in which the number of first tracking units is the same as the number of sensors. 5 is a flowchart showing a flow of an object tracking process performed by the object tracking device of the first embodiment. FIG. 11 is a block diagram showing a configuration of an object tracking system according to a second embodiment. FIG. 2 is a block diagram showing a configuration of an association unit; FIG. 2 is a diagram showing an example of a combination of a first tracker group and a second tracker group; 13 is a graph showing an example of a situation in which communication is lost at time t2 and both objects A and B decelerate. 11 is a diagram showing an example of time series transition and association of objects A and B on an actual driving lane when communication is interrupted; FIG. This is an example of association when there is no communication interruption and an incorrect first tracker is generated. 13 is a flowchart showing a flow of an object tracking process performed by the object tracking device of the second embodiment. 13 is a flowchart showing the flow of an association process. FIG. 13 is a block diagram showing an example of a case where the self-position is held in the modified example. FIG. 13 is a block diagram showing an example of a case where a moving body in a modified example has a detection unit.

Embodiments of the present disclosure are described in detail below with reference to the drawings.

This embodiment is a technology that takes into consideration the application of controlled autonomous driving. When a control center controls an autonomous vehicle in an environment where manually driven vehicles are also present, a control plan for the moving objects is required. However, in order to plan the control of moving objects via communication in an environment where moving objects that can communicate and moving objects that cannot communicate are both present, in addition to tracking the moving objects, it is necessary to identify the type of object, such as whether communication is possible and whether it is a control target. Furthermore, it is necessary to appropriately track and identify even when communication with some moving objects is temporarily interrupted.

Therefore, the technology of this embodiment uses a method of integrating the position of an object tracked by a sensor (hereinafter referred to as an infrastructure sensor or simply a sensor) installed in a certain area with the self-position estimated by the vehicle itself. Hereinafter, in this embodiment, a case will be described as an example in which the object to be tracked is a manually driven vehicle or an autonomously driven vehicle capable of self-position estimation. Note that the object is not limited to a vehicle, and may be another moving object that needs to be controlled. In that case, it is sufficient that the moving object capable of self-position estimation is included.
[First embodiment]
FIG. 1 is a block diagram showing the configuration of an object tracking system 100 according to the first embodiment. As shown in FIG. 1, in the object tracking system 100, each of a plurality of sensors 110, each of moving objects 112 to be tracked, each of moving objects 114 to be tracked and capable of self-location estimation, and an object tracking device 130 are connected via a network N. The network N is, for example, an Internet line or a public wireless LAN. In this embodiment, the moving object 112 is treated as object A, and the moving object 114 is treated as object B. When the moving object 112 and the moving object 114 are collectively referred to, they are simply referred to as a moving object or an object. Each functional configuration will be described later.

Figure 2 is a block diagram showing the hardware configuration of the object tracking device 130. As shown in Figure 2, the object tracking device 130 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I/F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or the storage 14. In this embodiment, an object tracking program is stored in the ROM 12 or the storage 14.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including an operating system, and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may also function as the input unit 15 by adopting a touch panel system.

The communication interface 17 is an interface for communicating with other devices such as terminals, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark). This concludes the description of the hardware configuration of the object tracking device 130.

Each functional configuration of the object tracking system 100 will be explained.

The mobile unit 114 includes a communication unit 116 and a self-position estimation unit 118.

The object tracking device 130 includes a communication unit 132, a detection unit 134, a first tracking unit 136, a second tracking unit 138, and an integration unit 140. The processing of the object tracking device 130 is repeated at preset time intervals, and the object tracking device 130 integrates and outputs the moving objects moving within the area in real time. In this way, the object tracking device 130 controls the area under its jurisdiction.

The sensors 110 are infrastructure sensors installed within the control area of the object tracking system 100, and in this embodiment, cameras are used. Each sensor 110 photographs an object, and the photographed camera image is transmitted to the object tracking device 130. The objects photographed here are assumed to be object A, which is a moving object 112, and object B, which is a moving object 114. Note that any sensor other than a camera may be used as long as it can provide an output capable of detecting the position of the object, and an infrared sensor, ultrasonic sensor, etc. may also be used.

Figure 3 is a diagram showing an example of the placement of infrastructure sensors in a controlled area. In the example of Figure 3, C1 to C12, CR1, and CR2 are infrastructure sensors equivalent to sensor 110, and are installed to capture images of the driving area.

The mobile object 114 transmits the result of the self-location estimation performed by the self-location estimation unit 118 from the communication unit 116 to the object tracking device 130. In this embodiment, the self-location estimation unit 118 uses a map matching method for estimating the self-location. Map matching is a method of matching external shape data of the surrounding environment obtained by an external sensor with external shape data of the environment included in a map for self-location estimation prepared in advance, and estimating the matching point as the self-location. In order to improve accuracy and robustness, the detection results of internal sensors such as various encoders are also used for the estimation. Although Monte Carlo localization is a commonly known method for estimating the self-location, the method is not limited to this method and may be a self-location estimation method using GPS or the like.

Next, each functional configuration of the object tracking device 130 will be described. Each functional configuration is realized by the CPU 11 reading out an object tracking program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

The communication unit 132 receives camera images from each of the sensors 110. The communication unit 132 receives the estimation result of the self-position from the mobile object 114.

The detection unit 134 detects the position of each object from the camera image which is the output of each sensor 110, and outputs the detection result. The objects detected here are, for example, each of the moving bodies 112 and each of the moving bodies 114. FIG. 4 is a diagram showing an example of a case where the number of detection units is the same as the number of sensors. For example, as shown in FIG. 4, the detection unit 134 can be configured as detection units 134 ₁ to 134 _N corresponding to the sensors 110 ₁ to 110 _N , as the number of detection units corresponding to the number of sensors. Using a general object detection method, the position of an object in a camera image acquired by a plurality of sensors 110 arranged as shown in FIG. 3 is detected.

The first tracking unit 136 outputs a first result, which is a tracking result obtained by tracking the positions of each of the multiple objects from the detection results. The first tracking unit 136 uses the outputs of the detection units 134 ₁ to 134 _N to track the positions of the objects in a common coordinate system. An example of the common coordinate system is map coordinates. For example, the method of Reference 1 may be used as a method for tracking multiple objects using multiple cameras.
[Reference 1] JP 2016-71830 A In this way, object tracking is performed using infrastructure sensors, and the position and speed of the object are tracked on a map regardless of whether communication is possible or not. In the above example, detection is performed for each sensor 110, and the detection results are integrated and tracked by the first tracking unit 136, but this is not limited to the above example. FIG. 5 is a diagram showing an example of a case where the number of first tracking units is the same as the number of sensors. For example, as shown in FIG. 5, the first tracking unit 136 may be first tracking units 136 ₁ to 136 _N corresponding to the sensors 110 ₁ to 110 _N , and after performing detection and tracking for each sensor 110, the first tracking integration unit 136 _A may integrate the results on a map coordinate system.

The second tracking unit 138 obtains a tracking result of tracking the position of the mobile unit 114 from the transition of the self-position estimation result received from the mobile unit 114, and outputs it as a second result. The second tracking unit 138 needs to output the self-position of each object in a coordinate system common to the first tracking unit 136, for example, map coordinates. The second tracking unit 138 may hold only the most recent received self-position estimation result, or may perform filtering using, for example, a Kalman filter. It may also manage the history of the self-position of each object.

The integration unit 140 integrates the object of the first result and the object of the second result by a predetermined correspondence and outputs the integrated result. For example, objects that are close in distance are associated according to the closeness of the distance of the objects included in each result. In this way, identical objects are integrated and output. In addition, for each object in the integrated result, the type of object, whether communication is possible or not, and the like may be output as object attributes. The type of object is, for example, the type of vehicle, such as whether it is a manually driven vehicle or an autonomous vehicle in the case of a vehicle.

Next, we will explain the operation of the object tracking device 130 according to the first embodiment.

Figure 6 is a flowchart showing the flow of the object tracking process by the object tracking device 130 of the first embodiment. The object tracking process is performed by the CPU 11 reading out the object tracking program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. The CPU 11 executes the processes as each part of the object tracking device 130.

In step S100, the CPU 11, acting as the communication unit 132, receives camera images from each of the sensors 110 and estimation results of the self-position from each of the mobile bodies 114. Reception here is performed at any time according to the transmission timing of each of the sensors 110 and each of the mobile bodies 114.

In step S102, the CPU 11, as the detection unit 134, detects each object from the camera images that are the output of each sensor 110, and outputs the detection results.

In step S104, the CPU 11, as the first tracking unit 136, outputs a first result, which is a tracking result obtained by tracking the positions of each of the multiple objects from the detection results of step S102.

In step S106, the CPU 11, as the second tracking unit 138, obtains a tracking result for tracking the position of the mobile unit 114 from the progress of the self-position estimation result received from the mobile unit 114, and outputs the result as the second result.

In step S108, the CPU 11, as the integration unit 140, integrates the first result object and the second result object using a predetermined correspondence and outputs the integrated result.

As described above, the object tracking system according to the first embodiment realizes tracking that is robust against temporary interruptions in communication and is capable of identifying the type of object.
[Second embodiment]
Next, a second embodiment will be described. In addition to the configuration of the first embodiment, the method of the second embodiment is configured to perform matching involving prediction to improve the accuracy of object tracking. Note that the same reference numerals are used for the same configurations and functions as those of the first embodiment, and descriptions thereof will be omitted.

Figure 7 is a block diagram showing the configuration of an object tracking system 200 of the second embodiment. In the second embodiment, the configuration of the object tracking device 230 differs from that of the first embodiment and includes a matching unit 242 and a matching history storage unit 244. Figure 8 is a block diagram showing the configuration of the matching unit 242. The matching unit 242 includes a prediction unit 250, a classification unit 252, a confirmation unit 254, and a calculation unit 256. The hardware configuration of the object tracking device 230 is similar to that of the first embodiment.

Through the processing of each of these units, the matching unit 242 outputs the matching result of a matching pair in which the first result object and the second result object are matched based on history and prediction.

As a premise for the processing of the association unit 242, in the first result and the second result, each object in each tracking result is a tracker group. Also, the trackers in the first result and the second result are the first tracker and the second tracker, respectively. The history is the history of the association pairs in the association history storage unit 244. The association pair is a pair of the first tracker and the second tracker that are associated. The history here refers to a pair that is processed in chronological order and most recently stored in the association history storage unit 244. The prediction is the prediction result of the transition of the state of each of the first tracker and the second tracker in chronological order by the prediction unit 250. The tracker is the object that is the target of tracking and is the target of the prediction of the transition. The association history storage unit 244 sequentially stores the association pairs of each of the first trackers and each of the second trackers.

The prediction unit 250 outputs a prediction result that predicts the transition of the state in a common time series for each of the tracker groups, that is, each of the first trackers that are the first result and each of the second trackers that are the second result. The time that is the reference for the common time series can be the current time or the latest time of the trackers. The state here is the position on the x-axis and y-axis. Here, for the second tracker, a likelihood is set according to the difference between the time used for the prediction and the time of the last reception, to reflect the instability of communication. The prediction process can use any method, such as a linear regression model or a Markov process.

The classification unit 252 classifies the tracker group into a tracker group that satisfies the first condition and a tracker group that does not satisfy the first condition based on the history of the association in the association history storage unit 244 and the prediction result, according to the first condition related to the history and the prediction, and outputs the classified tracker group. The first condition is a condition that the tracker focusing on the association pair associated in the history is included, and the first tracker and the second tracker of the association pair are included in the tracker predicted by the prediction unit 250. The tracker group that satisfies the first condition is composed of an association pair of a first tracker and a second tracker that satisfies the first condition. The tracker group that does not satisfy the first condition is the first tracker group and the second tracker group other than the tracker group that satisfies the condition. Here, it is sufficient to determine whether or not the first condition is satisfied for each tracker, classify, and output. The condition determination of the first condition is based on the presence or absence of an association pair, so it is sufficient to focus on either the first tracker or the second tracker. Here, for example, by focusing on the second tracker and judging the first condition for all of the second trackers, it is possible to output a matching pair that satisfies the first condition. As a result, the input first tracker group and second tracker group are classified into a tracker group that satisfies the first condition and a tracker group that does not satisfy the first condition. The tracker group that satisfies the first condition is output to the confirmation unit 254 as a matching pair that requires confirmation of the matching. The tracker group that does not satisfy the first condition is output to the calculation unit 256, and the matching is calculated for each set of trackers.

The confirmation unit 254 judges whether or not the tracker group consisting of the corresponding pair that satisfies the first condition is satisfied according to the second condition regarding the likelihood. Specifically, the second condition is at least one of the likelihood of the second tracker being equal to or less than a certain value and the Mahalanobis distance being equal to or less than a certain value. In this way, the judgment is made based on the likelihood of the second tracker, and the closeness between the first tracker and the second tracker of the corresponding pair is evaluated. If the second condition is satisfied, the corresponding pair is maintained and output to the integration unit 140 and the corresponding history storage unit 244. If the second condition is not satisfied, the corresponding pair is released and output to the calculation unit 256. In addition, as another method of the second condition, the distance threshold of the correspondence may be changed based on the likelihood of the second tracker. In addition, not limited to the Mahalanobis distance, a normal distance may be used for the judgment. Note that the likelihood of the second tracker is the same as the likelihood used by the prediction unit 250. This makes it possible to determine whether the predicted corresponding pair is likely as a corresponding pair. In addition, it is possible to deal with a temporary interruption of communication and a case where an erroneous first tracker is generated by the first tracking unit 136. In the first tracking unit 136, since a tracker is generated based on a camera image, there is a case where a first tracker that does not actually exist is generated by mistake. This process makes it possible to prevent association with such an erroneously generated first tracker.

The calculation unit 256 receives input of a tracker group that does not satisfy the first condition and a tracker group that does not satisfy the second condition. Note that the first condition or the second condition is an example of a predetermined condition of the present disclosure.

For a group of trackers that do not satisfy a predetermined condition, the calculation unit 256 calculates an optimal matching pair for each combination of the first tracker and the second tracker, and outputs the pair to the integration unit 140 and the matching history storage unit 244. The calculation unit 256 calculates one-to-one exclusive matching for all combinations of the first tracker and the second tracker. A cost is set according to the Mahalanobis distance, and the combination that minimizes the cost is calculated.

Figure 9 is a diagram showing an example of a combination of the first tracker group and the second tracker group. As shown in Figure 9, the first tracker group (tracker 1) has N items, the second tracker group (tracker 2) has M items, and N dummy nodes with costs set to the threshold value in the confirmation unit 254 are added to the second tracker side to calculate the correspondence. This is a bipartite graph matching problem, and can be easily solved by using the Hungarian method.

The association history storage unit 244 stores the history of the association pairs output from the confirmation unit 254 or the calculation unit 256. The length of the history can be set appropriately, for example, to store only the most recent result or the most recent few results.

An example of a scene in which the above-described processing of the matching unit 242 is effective is shown in FIG. 10. FIG. 10 is a graph showing an example of a situation in which communication is interrupted at time t2 and both objects A and B decelerate. In FIG. 10, the vertical axis represents the object position in the x-axis direction, and the horizontal axis represents time t in the time series. Object A corresponds to the first tracker, and object B corresponds to the second tracker. The communication interruption is a situation in which communication with object B is interrupted in the shaded area from time t. In this case, since the deceleration cannot be reflected in the prediction by the prediction unit 250, the prediction is a linear function transition according to the transitions at time t1 and time t2. However, in reality, object B decelerates, so the transition is logarithmic. There is a discrepancy between the actual movement of object B and the movement of object B predicted by the second tracker. In this way, the accuracy of the prediction by the second tracker at time t3 is greatly reduced.

Figure 11 is a diagram showing an example of the time series transition and correspondence of object A and object B on the actual driving lane when communication is interrupted. In Figure 11, for two objects, object A and object B shown in Figure 10, the left side shows the time series positions of object A and object B, and the right side shows the correspondence between the first tracker and the second tracker. Object A on the left side does not have a communication function, and object B has a communication function and performs self-location estimation. The ellipse on the right side indicates the first tracker, and the triangle indicates the second tracker. In the example of Figure 11, it is assumed that the transition of the second tracker is predicted for each time t, and the first tracker and the second tracker that are simply closer to the prediction are associated. The dashed line is the predicted transition. As shown in Figure 11, at times t1 and t2, the second tracker is associated with the rear first tracker in the prediction. This is correct as a pair of associations of object A and object B on the left side. On the other hand, looking at time t3, the second tracker is erroneously associated with the front first tracker in the prediction. In this way, if the association is predicted each time, the association may be incorrect due to a communication interruption. Therefore, assuming such a communication interruption, the association pairs associated at time t1 and time t2 when the communication conditions are good are stored as history in the association history storage unit 244, and are processed by the confirmation unit 254 and the calculation unit 256 as described above. In this case, if the association pair at the previous time t2 remains as history at the time t3 and a prediction is made for the association pair, the first condition is satisfied, so the confirmation unit 254 confirms the association. Then, if the confirmation unit 254 determines that the association pair does not satisfy the second condition, the association is released and a new association is performed by the calculation unit 256. As a result, an association pair in which the second tracker is correctly associated with the subsequent first tracker at time t3 can be generated. In this way, even if communication is interrupted, incorrect association can be prevented.

Also, FIG. 12 is an example of correspondence when there is no communication interruption and an incorrect first tracker is generated. As shown on the right side of FIG. 12, an incorrect first tracker is generated at time t1, resulting in three first trackers being generated. In reality, as shown on the left side, since there are two objects, it is desirable to generate two first trackers correctly, but this is a case in which some obstacle or the like is mistakenly detected. Therefore, at time t1, the incorrectly generated first tracker and the second tracker are associated with each other. Next, at time t2, the distance between the incorrectly generated first generator and the first generator in front is far. Therefore, the association pair at time t1 is determined not to satisfy the second condition by the confirmation unit 254, and a new association is made by the calculation unit 256. As a result, the correct association pair is stored in the association history storage unit 114, and the correct association pair can be maintained even at time t3.

The integration unit 140 integrates the object of the first result and the object of the second result based on the matching result and outputs the result. For trackers that are not matched, the result may be output as is, and for trackers that are matched, either the first tracker or the second tracker may be selected and output. The selection may be made, for example, according to the likelihood of the second tracker. The likelihood of the first tracker may also be calculated and compared with that of the second tracker. The variance of the position of the second tracker may also be estimated, and the selection may be made based on the variance matrix.

Next, we will explain the operation of the object tracking device 230 according to the second embodiment.

Figure 13 is a flowchart showing the flow of the object tracking process by the object tracking device 230 of the second embodiment. After the process of outputting the second result of the process of step S106, the process of step S200 is performed.

In step S200, the CPU 11, functioning as the matching unit 242, outputs a matching result of a matching pair in which the first result object and the second result object are matched based on history and prediction. Details of the matching process will be described later.

In step S202, the first result object and the second result object are integrated based on the matching result of step S200 and output.

The function of the matching process in step S200 will now be described. Figure 14 is a flowchart showing the flow of the matching process.

In step S1200, the CPU 11, as the prediction unit 250, outputs a prediction result that predicts the state transition in a common time series for each of the tracker groups, that is, each of the first trackers that are the first result and each of the second trackers that are the second result.

In step S1202, the CPU 11, as the classification unit 252, selects a tracker for which a condition judgment is to be made from the group of trackers. Here, it is sufficient to sequentially select the second trackers for which a condition judgment has not been made.

In step S1204, the CPU 11, functioning as the classification unit 252, determines whether the selected tracker satisfies the first condition based on the association history in the association history storage unit 244 and the prediction result. If it does, the process proceeds to step S1206, and if it does not, the process proceeds to step S1208.

In step S1206, the CPU 11, as the classification unit 252, classifies the trackers into a group that satisfies the first condition and outputs the group to the confirmation unit 254.

In step S1208, the CPU 11, as the classification unit 252, classifies the trackers into a group that does not satisfy the first condition, and outputs the result to the calculation unit 256.

In step S1210, the CPU 11, as the classification unit 252, determines whether the condition determination of the first condition has been completed for all tracker groups, and if so, proceeds to step S1212, and if not, returns to step S1202.

In step S1212, the CPU 11, as the confirmation unit 254, selects a matching pair to be subjected to condition judgment from the group of trackers consisting of matching pairs that satisfy the first condition.

In step S1214, the CPU 11, functioning as the confirmation unit 254, determines whether the selected matching pair satisfies the second condition regarding likelihood, in accordance with the second condition. If it does, the process proceeds to step S1216, and if it does not, the process proceeds to step S1218.

In step S1216, the CPU 11, functioning as the confirmation unit 254, determines that the associated pair satisfies the second condition and maintains the association.

In step S1218, the CPU 11, functioning as the confirmation unit 254, determines that the associated pair does not satisfy the second condition, cancels the association, and outputs the result to the calculation unit 256.

In step S1220, the CPU 11, functioning as the confirmation unit 254, determines whether the condition determination of the second condition has been completed for all matching pairs, and if so, proceeds to step S1222, and if not, returns to step S1212.

In step S1222, the CPU 11, as the calculation unit 256, calculates the optimal matching pair for each combination of the first tracking unit and the second tracking unit for the tracker group that does not satisfy the specified condition.

In step S1224, the CPU 11, functioning as the calculation unit 256, outputs the association pairs whose association was maintained in step S1216 and the association pairs calculated in step S1222 as association results to the integration unit 140 and the association history storage unit 244.

As described above, the object tracking system according to the second embodiment realizes tracking that is robust against temporary communication interruptions and can identify the type of object by matching based on history and prediction.

Note that this disclosure is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit of the invention. Modifications are described below.

For example, in each of the above-described embodiments, the second tracking unit 138 is used to track the result of self-location estimation, but the present invention is not limited to this. For example, as shown in FIG. 15, the second tracking unit 138 may be configured to hold the received self-location as a self-location holding unit 238. In this case, the received self-location may be held in chronological order, and the self-location held in chronological order may be used as the second result instead of the tracking result.

Also, the moving body 114 may have the detection functions of the sensor 120 and the detection unit 122 as shown in FIG. 16. In the example shown in FIG. 16, the moving body 114 is divided into a moving body 114a that does not have a detection function and a moving body 114b that has a detection function. The sensor 120 uses a camera to capture a camera image, similar to the sensor 110 that is an infrastructure sensor. The sensor 120 may of course be another infrared sensor, ultrasonic sensor, etc. The detection unit 122 detects surrounding objects by processing similar to that of the detection unit 134 of the object tracking device 130 (230). In this way, when the moving body 114 has the detection unit 122, the object tracking device 130 (230) can receive information detected by the moving body 114, not limited to the output of the infrastructure sensor, and use it for tracking processing. If the detected information is converted into map coordinates, tracking similar to the information of the sensor 110 can be realized. In this way, the detection result of the surrounding objects may be received from the moving body 114 and used as the detection result in the first tracking unit 136.

In addition, in the above-mentioned second embodiment, the case where the confirmation unit 254 is included has been described, but this is not limited to this. For example, the processing of the confirmation unit 254 may be omitted. In other words, a simple configuration may be used in which the determination based on the second condition is omitted and the corresponding pairs that satisfy the first condition are output while maintaining the correspondence. Even in this case, the calculation unit 256 calculates the corresponding pairs for the tracker groups that do not satisfy the first condition.

In addition, the object tracking process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) that are processors having a circuit configuration designed exclusively to execute specific processes. The object tracking process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

In addition, in each of the above embodiments, the object tracking processing program is described as being pre-stored (installed) in ROM or storage, but this is not limiting. The program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

100, 200 Object tracking system 110 Sensor 112 Moving object (object A)
114 Moving object (Object B)
116 Communication unit 118 Self-position estimation unit 120 Sensor 122 Detection unit 130, 230 Object tracking device 132 Communication unit 134 Detection unit 136 First tracking unit 136A First tracking integration unit 138 Second tracking unit 140 Integration unit 242 Correspondence unit 244 Correspondence history storage unit 250 Prediction unit 252 Classification unit 254 Confirmation unit 256 Calculation unit

Claims (7)

a detection unit that detects the position of each object from the output of a sensor installed within the area and outputs the detection result;
a tracking unit that outputs a first result, which is a tracking result obtained by tracking the positions of each of a plurality of objects based on the detection result;
a communication unit that receives, from each of the objects capable of estimating a self-location among the objects, an estimation result of a self-location estimated by the object;
a matching unit that outputs a matching result in which the estimation result received from each of the objects is used as a second result and the object of the first result is matched with the object of the second result based on a predetermined history and prediction;
an integration unit that integrates and outputs the estimation results received from each of the objects as a second result, and integrates the objects of the first result and the objects of the second result by associating, as a predetermined correspondence, an object included in the first result and an object that is close in distance to the object included in the second result as the same object according to a distance between the object and the object included in the second result,
The association unit includes a prediction unit, a classification unit, and a calculation unit;
The prediction unit outputs a prediction result that predicts a transition of a state in a common time series for each of the tracker groups, which are the first trackers that are the first results and the second trackers that are the second results;
The classification unit classifies the tracker group into a tracker group consisting of a corresponding pair of the corresponding pairs that satisfies the first condition among the corresponding pairs and a tracker group that does not satisfy the first condition, according to a first condition related to the history and prediction, the first condition being a condition that a tracker focusing on the corresponding pair in the history is included and the first tracker and the second tracker of the corresponding pair are included in a tracker that is a prediction target, based on the association history of the corresponding pair, which is a pair of the first tracker and the second tracker,
the calculation unit sets a cost according to a distance for each combination of the first tracker and the second tracker for the tracker group not satisfying the predetermined condition, and calculates an association pair with an optimal cost;
the matching unit outputs, as a matching result, the matching pair that satisfies the first condition and the matching pair calculated by the calculation unit;
the integration unit integrates the first result object and the second result object based on the association result and outputs the integrated object.
Object tracking device. the tracking unit includes a first tracking unit that outputs the first result and a second tracking unit;
The object tracking device according to claim 1 , wherein the second tracking unit obtains a tracking result by tracking the position of the object from a transition of the estimation result, and outputs the result as the second result. The association unit further includes a confirmation unit,
2. The object tracking device according to claim 1, wherein the confirmation unit, in accordance with a second condition regarding likelihood for a group of trackers consisting of a corresponding pair that satisfies the first condition, at least one of a likelihood according to the time of reception by the second tracker being a certain value or less and a distance being a certain value or less, maintains the correspondence and outputs the corresponding pair if the second condition is satisfied, and releases the corresponding pair as not satisfying the predetermined condition and outputs it to the calculation unit if the second condition is not satisfied. Among the objects, there is an object having a function of detecting surrounding objects,
4. The object tracking device according to claim 1 , wherein the communication unit receives a detection result of a surrounding object from the object, and uses the result as the detection result in the tracking unit. 5. The object tracking device according to claim 1, wherein the integration unit outputs an integration result including attributes of the object. Detecting the position of each object from the output of sensors installed within the area and outputting the detection results;
outputting a first result which is a tracking result obtained by tracking the positions of each of the plurality of objects from the detection result;
receiving, from each of the objects capable of estimating a self-location, an estimation result of a self-location estimated by the object;
In an aspect of outputting an association result in which the estimation result received from each of the objects is associated with the object of the first result based on a predetermined history and prediction, the method includes a prediction process, a classification process, and a calculation process,
outputting the estimation results received from each of the objects as a second result, integrating the objects of the first result and the objects of the second result by associating, as a predetermined correspondence, an object included in the first result and an object that is close to the object included in the second result as the same object according to a distance between the object and the object included in the second result ;
In the prediction process, a prediction result is outputted, which predicts a state transition in a common time series for each of the tracker groups, which are the first trackers as the first results and the second trackers as the second results;
In the classification process, based on the association history of the association pair, which is a pair of the first tracker and the second tracker, and the prediction result, the tracker group is classified into a tracker group consisting of an association pair that satisfies the first condition among the association pairs and a tracker group that does not satisfy the first condition according to a first condition related to the history and prediction, the first condition being a condition that a tracker that focuses on the association pair that is associated in the history is included, and the first tracker and the second tracker of the association pair are included in a tracker that is a prediction target;
The calculating process includes: setting a cost according to a distance for each combination of the first tracker and the second tracker for the tracker group that does not satisfy the predetermined condition; and calculating a matching pair with an optimal cost;
outputting, as a matching result, a matching pair that satisfies the first condition and the calculated matching pair;
In the integration, the first result object and the second result object are integrated based on the matching result and output.
A method for tracking an object, the method comprising the steps of: causing a computer to execute a process; Detecting the position of each object from the output of sensors installed within the area and outputting the detection results;
outputting a first result which is a tracking result obtained by tracking the positions of each of the plurality of objects from the detection result;
receiving, from each of the objects capable of estimating a self-location, an estimation result of a self-location estimated by the object;
In an aspect of outputting an association result in which the estimation result received from each of the objects is associated with the object of the first result based on a predetermined history and prediction, the method includes a prediction process, a classification process, and a calculation process,
outputting the estimation results received from each of the objects as a second result, integrating the objects of the first result and the objects of the second result by associating, as a predetermined correspondence, an object included in the first result and an object that is close to the object included in the second result as the same object according to a distance between the object and the object included in the second result ;
In the prediction process, a prediction result is outputted, which predicts a state transition in a common time series for each of the tracker groups, which are the first trackers as the first results and the second trackers as the second results;
In the classification process, based on the association history of the association pair, which is a pair of the first tracker and the second tracker, and the prediction result, the tracker group is classified into a tracker group consisting of an association pair that satisfies the first condition among the association pairs and a tracker group that does not satisfy the first condition according to a first condition related to the history and prediction, the first condition being a condition that a tracker that focuses on the association pair that is associated in the history is included, and the first tracker and the second tracker of the association pair are included in a tracker that is a prediction target;
The calculating process includes: setting a cost according to a distance for each combination of the first tracker and the second tracker for the tracker group that does not satisfy the predetermined condition; and calculating a matching pair with an optimal cost;
outputting, as a matching result, the matching pair that satisfies the first condition and the calculated matching pair;
In the integration, the first result object and the second result object are integrated based on the matching result and output.
An object tracking program that causes a computer to execute the process.