JP7849255B2 - Tracking device - Google Patents

Description

This disclosure relates to a tracking device for tracking objects.

Patent Document 1 describes an autonomous driving system configured to receive detection information from each of several sensors indicating detection results in a common detection area across the multiple sensors, and to generate multiple hypotheses regarding objects present within that common detection area based on the received detection information.

U.S. Patent Application Publication No. 2020/0377086

In tracking devices that track objects based on sensor detection information, a problem arose where continuously generating multiple hypotheses to track an object resulted in a significant increase in the number of hypotheses over time, leading to a heavy computational load.

This disclosure aims to reduce the computational load required for tracking objects.

One aspect of this disclosure is a tracking device (4) that periodically acquires at least one piece of observational information indicating an observation result from at least one sensor (2, 3) configured to observe the surrounding conditions, and estimates and tracks the object state, which is the state of an object present in the surroundings, based on the at least one piece of observational information, and comprises a state update unit (S10 to S100).

The state update unit is configured to re-estimate the object state based on at least one observational information obtained from at least one sensor each time it acquires at least one observational information.

The state update unit comprises a probability calculation unit (S10 to S30, S60 to S80) and a state estimation unit (S40 to S50, S90 to S100).
The probability calculation unit is configured to calculate the probability of identity, which indicates the probability that the acquired at least one observation is information obtained from an object corresponding to the previously estimated object state, based on at least one observation from at least one sensor and the object state estimated by the state update unit immediately before acquiring the at least one observation.

The state estimation unit is configured to re-estimate the object state by generating object state estimation information indicating the estimated object state based on at least one observation information from at least one sensor, the object state estimated by the state update unit immediately before acquiring at least one observation information, and the probability of identical objects calculated by the probability calculation unit.

The probability calculation unit includes a likelihood calculation unit (S10-S20, S60-S70). The likelihood calculation unit is configured to calculate at least one likelihood for each of at least one observational information, indicating the likelihood that the information corresponds to the object state estimated by the state update unit immediately before acquiring at least one observational information. The probability calculation unit then calculates the probability of identical objects using the at least one likelihood calculated by the likelihood calculation unit.

The likelihood calculation unit includes a similarity calculation unit (S10, S60). The similarity calculation unit is configured to calculate the similarity between a multi-value observation (which is an observation information represented by multiple values from at least one set of observation information) and object state estimation information (which indicates the object state estimated by the state update unit immediately before acquiring the multi-value observation information).

The tracking device of this disclosure, configured in this manner, can convert multi-value observation information, represented by multiple values, into a similarity score represented by a single value. Therefore, it can estimate the object state while suppressing the increase in hypotheses. As a result, the tracking device of this disclosure can reduce the computational load required to track an object.

This is a diagram showing the configuration of the driver assistance system. This is a flowchart showing the tracking process of the first embodiment. This figure shows vectors A and B. This is a diagram showing the likelihood function. This figure shows the method for calculating the first type determination information. This figure shows the method for calculating the type determination information after the first observation. This graph shows the processing time for each frame. This is a flowchart showing the tracking process of the second embodiment.

[First Embodiment]
A first embodiment of this disclosure is described below with reference to the drawings.
The driver assistance system 1 of this embodiment is mounted on a vehicle and, as shown in Figure 1, comprises a camera sensor 2, a radar sensor 3, a tracking device 4, and a support execution unit 5. Hereinafter, the vehicle on which the driver assistance system 1 is mounted will be referred to as "the vehicle."

The camera sensor 2 is configured, for example, as a stereo camera capable of detecting the distance to an object. Based on the captured image, it detects the position (i.e., relative position to the vehicle), size, and type of the object in the image. In this embodiment, the object types are set to pedestrians, motorcycles, and automobiles. The camera sensor 2 generates first detection information indicating the detection result and transmits the generated first detection information to the tracking device 4. The first detection information includes first position detection information, first size detection information, and first type detection information. The first position detection information is information indicating the position of the object detected by the camera sensor 2. The first size detection information is information indicating the size of the object detected by the camera sensor 2. The first type detection information is information indicating the type of object detected by the camera sensor 2.

The radar sensor 3 transmits radar waves such as millimeter waves or microwaves and receives the reflected radar waves to detect the position (i.e., relative position to the vehicle), size, and type of an object. In this embodiment, the object types are set to pedestrians, motorcycles, and automobiles. The radar sensor 3 generates second detection information indicating the detection results and transmits the generated second detection information to the tracking device 4. The second detection information includes second position detection information, second size detection information, and second type detection information. The second position detection information is information indicating the position of the object detected by the radar sensor 3. The second size detection information is information indicating the size of the object detected by the radar sensor 3. The second type detection information is information indicating the type of object detected by the radar sensor 3.

The tracking device 4 is an electronic control unit centered around a microcomputer equipped with a CPU 11, ROM 12, RAM 13, etc. The various functions of the microcomputer are realized by the CPU 11 executing a program stored in a non-transitional physical recording medium. In this example, ROM 12 corresponds to the non-transitional physical recording medium storing the program. Furthermore, the execution of this program executes the method corresponding to the program. Note that some or all of the functions executed by the CPU 11 may be configured hardware-wise using one or more ICs, etc. Also, the number of microcomputers constituting the tracking device 4 may be one or more.

The tracking device 4 tracks objects around the vehicle based on the first detection information generated by the camera sensor 2 and the second detection information generated by the radar sensor 3, and generates object information indicating the state of the tracked objects.

The tracking device 4 uses the position of the object indicated by the first position detection information as the observed position value (hereinafter, the first observed position value) and uses a Kalman filter to calculate the estimated position value (hereinafter, the first estimated position value), the position standard deviation (hereinafter, the first standard deviation) indicating the uncertainty of the estimated position value, and the Kalman gain (hereinafter, the first Kalman gain).

The tracking device 4 uses the size of the object indicated by the first size detection information as the observed size (hereinafter, the first size observed value), and uses a Kalman filter to calculate the estimated size (hereinafter, the first size estimated value), the size standard deviation (hereinafter, the first size standard deviation) indicating the uncertainty of the size estimated value, and the Kalman gain (hereinafter, the first size Kalman gain).

The tracking device 4 uses the type of object indicated by the first type detection information as the observed type value (hereinafter, the first type observed value), and uses a Kalman filter to calculate the estimated type value (hereinafter, the first type estimated value), the type standard deviation (hereinafter, the first type standard deviation) indicating the uncertainty of the estimated type value, and the Kalman gain (hereinafter, the first type Kalman gain).

The tracking device 4 uses the position of the object indicated by the second position detection information as the observed position value (hereinafter, second position observation value), and uses a Kalman filter to calculate the estimated position value (hereinafter, second position estimate value), the position standard deviation (hereinafter, second position standard deviation) indicating the uncertainty of the estimated position value, and the Kalman gain (hereinafter, second position Kalman gain).

The tracking device 4 uses the size of the object indicated by the second size detection information as the observed size value (hereinafter referred to as the second size observation value), and uses a Kalman filter to calculate the estimated size value (hereinafter referred to as the second size estimate value), the size standard deviation (hereinafter referred to as the second size standard deviation) indicating the uncertainty of the size estimate value, and the Kalman gain (hereinafter referred to as the second size Kalman gain).

The tracking device 4 uses the type of object indicated by the second type detection information as the observed type value (hereinafter, second type observed value) and calculates the estimated type value (hereinafter, second type estimated value), the type standard deviation (hereinafter, second type standard deviation) indicating the uncertainty of the estimated type value, and the Kalman gain (hereinafter, second type Kalman gain) using a Kalman filter.

The support execution unit 5 includes, for example, an actuator, an audio device, and a display device. Based on the object information generated by the tracking device 4, the support execution unit 5 controls the vehicle's behavior and provides notifications to the vehicle's driver.

Next, the procedure for the tracking process executed by the CPU 11 of the tracking device 4 will be explained. The tracking process is a process that is repeatedly executed at predetermined processing cycles during the operation of the tracking device 4. The repetition period of the processing cycle is denoted as the execution cycle ΔT.

When the tracking process is executed, the CPU 11 first calculates the first similarity s1 in S10, as shown in Figure 2.
The first similarity score s1 is calculated using the first type detection information contained in the most recent first detection information transmitted from the camera sensor 2 and the second post-observation type determination information, described later, which was calculated in S100 of the tracking process one cycle prior to the execution cycle ΔT. Note that in S10 of the initial tracking process executed immediately after the tracking device 4 is started, the first similarity score s1 is not calculated because the second post-observation type determination information from the tracking process one cycle prior does not exist.

The first type detection information is represented by a vector showing the detection probability of pedestrians, motorcycles, and automobiles. The pedestrian detection probability is the probability that the object detected by camera sensor 2 is a pedestrian. Similarly, the motorcycle detection probability is the probability that the object detected by camera sensor 2 is a motorcycle. The automobile detection probability is the probability that the object detected by camera sensor 2 is an automobile. For example, if the first type information is [0.1, 0.2, 0.7], the detection probabilities for pedestrians, motorcycles, and automobiles are 10%, 20%, and 70%, respectively. The first type information is generated such that the sum of the detection probabilities for pedestrians, motorcycles, and automobiles equals 100%.

Furthermore, the second observation type determination information, similar to the first type detection information, is represented by vectors showing the detection probabilities for pedestrians, motorcycles, and automobiles, respectively.
As shown in Figure 3, the CPU 11 calculates the first similarity s1 using equation (1), where vector A = [ _A1 , _A2 , _A3 ] represents the first type detection information and vector B = [ _B1 , _B2 , _B3 ] represents the second type determination information after observation. If vector A and vector B are identical, the first similarity s1 is 1.

Next, as shown in Figure 2, the CPU 11 calculates the first type likelihood q11 in S20. The first type likelihood q11 represents the degree of likelihood that the object detected by the camera sensor 2 (i.e., the object corresponding to the first type detection information) and the predicted object (i.e., the object corresponding to the second post-observation type determination information) are the same object.

The first type likelihood q11 is calculated using the first similarity s1, the type determination accuracy of the camera sensor 2, and the prediction type determination accuracy. Note that in S20 of the initial tracking process, which is executed immediately after the tracking device 4 is started, the first type likelihood q11 is not calculated because the first similarity s1 from the previous tracking process does not exist.

The type determination accuracy of camera sensor 2 is a value between 0 and 1, and is a preset fixed value (for example, 0.2). A smaller value indicates better accuracy.

The accuracy of the predicted type determination is the degree to which the type indicated by the second post-observation type determination information matches the type of the detected object. Specifically, the prediction type determination accuracy is the standard deviation calculated using the multiple second post-observation type determination information calculated in S100 (described below).

The CPU 11 calculates the first type likelihood q11 by substituting the first similarity s1, the type determination accuracy of the camera sensor 2, and the prediction type determination accuracy into a pre-set first type likelihood function. The first type likelihood function is a function for calculating the first type likelihood q11 using the first similarity s1, the type determination accuracy of the camera sensor 2, and the prediction type determination accuracy as variables. Alternatively, the first type likelihood q11 may be calculated by referring to a first type likelihood table in which the correspondence between the first similarity s1, the type determination accuracy of the camera sensor 2, the prediction type determination accuracy, and the first type likelihood q11 is defined.

Graph G1 in Figure 4 shows the first type likelihood function. However, for the sake of simplicity in the illustration, the first type likelihood function shown in Graph G1 only includes the first similarity s1 as a variable.
Next, as shown in Figure 2, the CPU 11 calculates the first identical object probability r1 in S30. The first identical object probability r1 is the probability that the object detected by the camera sensor 2 (i.e., the object corresponding to the first type detection information) and the predicted object (i.e., the object corresponding to the second post-observation type determination information) are the same object.

The first identical object probability r1 is calculated using the first type likelihood q11, the first position likelihood q12, and the first size likelihood q13. Note that in S30 of the initial tracking process, which is executed immediately after the tracking device 4 is started, the first type likelihood q11 from the previous tracking process does not exist, and therefore the first identical object probability r1 is not calculated.

The first position likelihood q12 represents the degree of plausibility of the first position observation. The CPU 11 calculates the first position likelihood q12 by substituting the first position observation into the first position likelihood function, which is set based on the first position estimate and the first position standard deviation. The first position likelihood function, as shown in graph G2 of Figure 4, is a function for calculating the first position likelihood q12 using the first position observation as the variable. The first position likelihood function exhibits a normal distribution with the first position estimate as the mean and the square of the first position standard deviation as the variance.

The first size likelihood q13 represents the degree of plausibility of the first size observation. The CPU 11 calculates the first size likelihood q13 by substituting the first size observation into the first size likelihood function, which is set based on the first size estimate and the first size standard deviation. The first size likelihood function, as shown in graph G3 of Figure 4, is a function for calculating the first size likelihood q13 using the first size observation as the variable. The first size likelihood function exhibits a normal distribution with the first size estimate as the mean and the square of the first size standard deviation as the variance.

The CPU 11 then calculates the average of the first type likelihood q11, the first position likelihood q12, and the first size likelihood q13. Furthermore, it normalizes this average so that the maximum value of the average is 1, and this normalized value is defined as the first identical object probability r1.

Next, as shown in Figure 2, the CPU 11 calculates the first type determination information in S40. Specifically, as shown in Figure 5, the CPU 11 calculates the first type determination information using equation (2), with vector A' representing the first type determination information and K1 being the first type Kalman gain.

Next, as shown in Figure 2, the CPU 11 calculates the first post-observation classification information and the standard deviation of the first post-observation classification information in S50, and stores the calculated first post-observation classification information and standard deviation in the RAM 13.

Specifically, as shown in Figure 6, the CPU 11 calculates the first post-observation classification information using equation (3), with vector A'' representing the first post-observation classification information. Furthermore, the CPU 11 calculates the standard deviation of the first post-observation classification information using the multiple pieces of first post-observation classification information stored in the RAM 13.

Next, as shown in Figure 2, the CPU 11 calculates the second similarity score s2 in S60.
The second similarity score s2 is calculated using the second type detection information included in the most recent second detection information transmitted from the radar sensor 3 and the most recent first observation post-type determination information calculated in S50.

Let vector C = [ _C1 , _C2 , _C3 ] represent the second type detection information, and vector D = [ _D1 , _D2 , _D3 ] represent the first observation-based type determination information. The CPU 11 calculates the second similarity s2 using equation (4). If vector C and vector D are identical, the second similarity s2 becomes 1.

Next, in S70, the CPU 11 calculates the second type likelihood q21. The second type likelihood q21 represents the degree of likelihood that the object detected by the radar sensor 3 (i.e., the object corresponding to the second type detection information) and the predicted object (i.e., the object corresponding to the first observation type determination information) are the same object.

The second type likelihood q21 is calculated using the second similarity s2, the type determination accuracy of the radar sensor 3, and the type determination accuracy of the prediction.
The type determination accuracy of the radar sensor 3 is a value between 0 and 1, and is a preset fixed value (for example, 0.4).

The accuracy of the predicted type determination is the degree to which the type indicated by the first post-observation type determination information matches the type of the detected object. Specifically, the prediction type determination accuracy is the standard deviation calculated using the multiple first post-observation type determination information calculated in S50.

The CPU 11 calculates the second type likelihood q21 by substituting the second similarity s2, the type determination accuracy of the radar sensor 3, and the prediction type determination accuracy into a pre-set second type likelihood function. The second type likelihood function is a function for calculating the second type likelihood q21 using the second similarity s2, the type determination accuracy of the radar sensor 3, and the prediction type determination accuracy as variables. Alternatively, the second type likelihood q21 may be calculated by referring to a second type likelihood table in which the correspondence between the second similarity s2, the type determination accuracy of the radar sensor 3, the prediction type determination accuracy, and the second type likelihood q21 is defined.

Next, the CPU 11 calculates the second identical object probability r2 in S80. The second identical object probability r2 is the probability that the object detected by the radar sensor 3 (i.e., the object corresponding to the second type detection information) and the predicted object (i.e., the object corresponding to the first observation-based type determination information) are identical.

The second identical object probability r² is calculated using the second type likelihood q²1, the second position likelihood q²2, and the second size likelihood q²3.
The second position likelihood q22 is the degree of plausibility of the second position observation. The CPU 11 calculates the second position likelihood q22 by substituting the second position observation into the second position likelihood function, which is set based on the second position estimate and the second position standard deviation. The second position likelihood function is a function for calculating the second position likelihood q22 with the second position observation as the variable. The second position likelihood function exhibits a normal distribution with the second position estimate as the mean and the square of the second position standard deviation as the variance.

The second size likelihood q23 represents the degree of plausibility of the second size observation. The CPU 11 calculates the second size likelihood q23 by substituting the second size observation into the second size likelihood function, which is set based on the second size estimate and the second size standard deviation. The second size likelihood function is a function for calculating the second size likelihood q23 using the second size observation as the variable. The second size likelihood function exhibits a normal distribution with the second size estimate as the mean and the square of the second size standard deviation as the variance.

The CPU 11 then calculates the average of the second type likelihood q21, the second position likelihood q22, and the second size likelihood q23. Furthermore, it normalizes this average so that the maximum value of the average is 1, and this normalized value is defined as the second identical object probability r2.

Next, the CPU 11 calculates the second type determination information in S90. Specifically, the CPU 11 uses vector C' to represent the second type determination information and K2 to represent the second type Kalman gain, and calculates the second type determination information using equation (5).

Next, in S100, the CPU 11 calculates the second observation-based classification information and the standard deviation of the second observation-based classification information, and stores the calculated second observation-based classification information and standard deviation in the RAM 13.

Specifically, the CPU 11 calculates the second post-observation classification information using equation (6), with vector C'' representing the second post-observation classification information. Furthermore, the CPU 11 calculates the standard deviation of the second post-observation classification information using the multiple second post-observation classification information stored in the RAM 13.

Furthermore, the tracking device 4 determines, based on the second observation type determination information, whether the object detected by the camera sensor 2 and radar sensor 3 is a pedestrian, a motorcycle, or a motor vehicle.

Figure 7 is a graph comparing the time required to perform object state estimation calculations in each frame between the method of this disclosure and the conventional method. Each frame corresponds to a process that is repeatedly executed after the execution period ΔT described above has elapsed. The conventional method uses a PMB filter to perform object state estimation calculations. PMB stands for Poisson Multi-Bernoulli.

As shown in Figure 7, the maximum computation time in the conventional method is 47 ms, while the maximum computation time in the method of this disclosure is 2.7 ms, demonstrating that the computation time can be reduced by the method of this disclosure.

The tracking device 4, configured in this way, periodically acquires first and second detection information from the camera sensor 2 and radar sensor 3, which are configured to observe the surrounding conditions. Based on the first and second detection information, it estimates and tracks the state of objects in the surrounding area (hereinafter referred to as "object state").

The tracking device 4 is configured to re-estimate the object state based on the acquired first and second detection information each time it acquires first and second detection information from the camera sensor 2 and radar sensor 3.
The tracking device 4 is configured to calculate a first identical object probability r1, which indicates the probability that the acquired first detection information is information obtained from an object corresponding to the previously estimated object state, based on the first detection information from the camera sensor 2 and the object state estimated immediately before acquiring the first detection information (i.e., second post-observation type determination information).

The tracking device 4 is configured to calculate a second identical object probability r2, which indicates the probability that the acquired second detection information is information obtained from an object corresponding to the previously estimated object state, based on the second detection information from the radar sensor 3 and the object state estimated immediately before acquiring the second detection information (i.e., the first post-observation type determination information).

The tracking device 4 is configured to re-estimate the object state by generating first post-observation classification information indicating the estimated object state, based on the first detection information from the camera sensor 2, the object state estimated immediately before acquiring the first detection information (i.e., second post-observation classification information), and the calculated first identical object probability r1.

The tracking device 4 is configured to re-estimate the object state by generating second post-observation classification information indicating the estimated object state, based on the second detection information from the radar sensor 3, the object state estimated immediately before acquiring the second detection information (i.e., the first post-observation classification information), and the calculated second identical object probability r2.

The tracking device 4 calculates the first identical object probabilities r1 and r2 for each of the first and second detection information, using the first type likelihoods q11 and q21, the first and second position likelihoods q12 and q22, and the first and second size likelihoods q13 and q23, which indicate the likelihood that the information corresponds to the object state estimated immediately before acquiring the first and second detection information. Then, the tracking device 4 calculates the first and second identical object probabilities r1 and r2 using the calculated first and second type likelihoods q11 and q21, the first and second position likelihoods q12 and q22, and the first and second size likelihoods q13 and q23.

The tracking device 4 is configured to calculate the first and second similarity s1 and s2 between the first and second type detection information, which is represented by three values from the first and second detection information, and the second and first post-observation type determination information, which indicates the object state estimated immediately before acquiring the first and second type detection information.

This tracking device 4 can convert the first and second type detection information, which is represented by multiple values, into first and second similarity values s1 and s2, which are represented by a single value. Therefore, it can estimate the object state while suppressing the increase in hypotheses. As a result, the tracking device 4 can reduce the computational load required to track the object.

Furthermore, the tracking device 4 generates first and second post-observation type determination information by integrating the first assumed object state (i.e., vectors A', C'), which is the object state assuming that the first type detection information corresponds to the object state estimated immediately before acquiring the first type detection information, and the second assumed object state (i.e., vectors B, D), which is the object state assuming that the first type detection information does not correspond to the object state estimated immediately before acquiring the first type detection information, using the first and second identical object probabilities r1 and r2.

Such a tracking device 4 can improve the robustness of object state estimation by considering two types of states: a first hypothetical object state and a second hypothetical object state.

Furthermore, the tracking device 4 uses the first detection information acquired from the camera sensor 2 to estimate the object state as the first object state. Then, the tracking device 4 uses the second detection information acquired from the radar sensor 3 and the estimated first object state to estimate the object state as the second object state.

Such a tracking device 4 can estimate the state of an object by integrating detection information from two sensors.
In the embodiments described above, the camera sensor 2 and the radar sensor 3 correspond to sensors, the first and second detection information correspond to observation information, S10 to S100 correspond to processing as a state update unit, S10 to S30 and S60 to S80 correspond to processing as a probability calculation unit, and the identical object probabilities correspond to the first and second identical object probabilities r1 and r2.

Furthermore, steps S40-S50 and S90-S100 correspond to processing as a state estimation unit, the first and second observation-based type determination information corresponds to object state estimation information, steps S10-S20 and S60-S70 correspond to processing as a likelihood calculation unit, and likelihoods q11, q21, q12, q22, q13, and q23 correspond to likelihoods.

Furthermore, S10 and S60 correspond to processing as a similarity calculation unit, and the first and second type detection information corresponds to multiple value observation information.
Furthermore, the camera sensor 2 corresponds to the first sensor, the radar sensor 3 corresponds to the second sensor, S10 to S50 correspond to processing as the first estimation unit, and S60 to S100 correspond to processing as the second estimation unit.

[Second Embodiment]
A second embodiment of this disclosure is described below with reference to the drawings. In the second embodiment, the parts that differ from the first embodiment will be described. Common components are denoted by the same reference numerals.

The driving assistance system 1 of the second embodiment differs from the first embodiment in that the tracking process has been modified.
The tracking process in the second embodiment differs from that of the first embodiment in that the processes S22, S24, S26, S72, S74, and S76 are added, as shown in Figure 8.

In other words, when the processing in S20 is completed, the CPU 11 calculates the first Mahalanobis distance in S22 between vector A used in S10 and multiple second observation-based classification information (i.e., vectors B and C'') stored in RAM 13.

Specifically, the CPU 11 calculates the mean vectors and covariance matrices of multiple vectors B (or vector C'), and then calculates the first Mahalanobis distance by substituting vector A and the mean vectors and covariance matrices of the multiple vectors B into a well-known formula for calculating the Mahalanobis distance.

Then, in S24, the CPU 11 determines whether the first Mahalanobis distance calculated in S22 is below a predetermined threshold. If the first Mahalanobis distance is below the threshold, the CPU 11 proceeds to S30. On the other hand, if the first Mahalanobis distance exceeds the threshold, the CPU 11, in S26, changes the second post-observation classification information (i.e., vector B) used in S10 to the first post-observation classification information (i.e., vector A'') and proceeds to S60.

Furthermore, once the processing in S70 is complete, the CPU 11 calculates the second Mahalanobis distance in S72 between the vector C used in S60 and the multiple first observation-based type determination information (i.e., vector A'') stored in the RAM 13.

Specifically, the CPU 11 calculates the mean vector and covariance matrix of multiple vectors A'', and then calculates the second Mahalanobis distance by substituting vector C and the mean vector and covariance matrix of the multiple vectors A'' into a well-known formula for calculating the Mahalanobis distance.

Then, in S74, the CPU 11 determines whether the second Mahalanobis distance calculated in S72 is below a predetermined threshold. If the second Mahalanobis distance is below the threshold, the CPU 11 proceeds to S80. On the other hand, if the second Mahalanobis distance exceeds the threshold, in S76, the CPU 11 changes the first post-observation classification information (i.e., vector D) used in S60 to the second post-observation classification information (i.e., vector C''), and terminates the tracking process.

The tracking device 4, configured in this way, calculates the first Mahalanobis distance between the first type determination information (i.e., vector A) acquired from the camera sensor 2 and the generated second post-observation type determination information (i.e., vector B). If the first Mahalanobis distance is greater than a preset threshold, the tracking device 4 determines that the object corresponding to the second post-observation type determination information was not observed by the camera sensor 2.

Furthermore, the tracking device 4 calculates the second Mahalanobis distance between the first type determination information (i.e., vector C) acquired from the radar sensor 3 and the generated first post-observation type determination information (i.e., vector A''). If the second Mahalanobis distance is greater than a preset threshold, the tracking device 4 determines that the object corresponding to the first post-observation type determination information was not observed by the radar sensor 3.

This tracking device 4 can omit calculations using detection results with large Mahalanobis distances, thereby suppressing a decrease in the accuracy of object state estimation and speeding up the calculation of object state estimation.

In the embodiment described above, S22 and S72 correspond to processing as a Mahalanobis distance calculation unit, the first and second type detection information corresponds to observation information, the first and second Mahalanobis distances correspond to Mahalanobis distances, and S24 and S74 correspond to processing as an observation judgment unit.

Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment and can be implemented in various modified forms.
[Variation 1]
In the above embodiment, the object state is estimated based on first and second detection information acquired from the camera sensor 2 and the radar sensor 3. However, the tracking device 4 may also estimate the object state based on the detection information from either the camera sensor 2 or the radar sensor 3. Furthermore, if the tracking device 4 acquires detection information from three or more sensors, it may also estimate the object state based on the detection information acquired from three or more sensors.

[Modified example 2]
In the above embodiment, the tracking device 4 was shown to estimate the state of an object. However, the tracking device 4 may also calculate the probability that an object exists (hereinafter referred to as the object existence probability). This allows the tracking device 4 to update parameters related to the occurrence/disappearance of an object (i.e., the object existence probability) during object tracking. The tracking device 4 calculates the object existence probability using, for example, a PMBM filter or a PMB filter. PMBM stands for Poisson Multi-Bernoulli Mixture.

[Modification 3]
In the above embodiment, a method was shown in which the first identical object probabilities r1 and r2 are calculated using the first type likelihoods q11 and q21, the first and second position likelihoods q12 and q22, and the first and second size likelihoods q13 and q23. However, the tracking device 4 may also calculate the first identical object probabilities r1 and r2 using random finite set theory, such as a PHD filter, LMB filter, PMBM filter, or PMB filter. By using random finite set theory, such a tracking device 4 can improve the robustness of object estimation. PHD stands for Probability Hypothesis Density. LMB stands for Labeled Multi-Bernoulli.

[Modification 4]
In the above embodiment, a method was shown in which the first and second similarity scores s1 and s2 are calculated using first and second type detection information. However, it is also possible to calculate the similarity using multidimensional feature information of an object obtained particularly through image learning as detection information. Such a tracking device 4 can estimate the state of an object using information that does not have a physical interpretation for each individual numerical value, such as feature information output by AI. AI stands for Artificial Intelligence.

The tracking device 4 and its method described in this disclosure may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the tracking device 4 and its method described in this disclosure may be implemented by a dedicated computer provided by configuring a processor using one or more dedicated hardware logic circuits. Alternatively, the tracking device 4 and its method described in this disclosure may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor composed of one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. The methods for implementing the functions of each part included in the tracking device 4 do not necessarily need to include software; all of its functions may be implemented using one or more hardware components.

Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Furthermore, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, some parts of the configuration of the above embodiment may be omitted. Furthermore, at least some parts of the configuration of the above embodiment may be added to or replaced with the configuration of other above embodiments.

In addition to the tracking device 4 described above, this disclosure can also be implemented in various forms, such as a system comprising the tracking device 4, a program for causing the computer to function as the tracking device 4, a non-transitional physical recording medium such as semiconductor memory on which this program is recorded, and a tracking method.

2…Camera sensor, 3…Radar sensor, 4…Tracking device

Claims (7)

A tracking device (4) periodically acquires at least one piece of observational information indicating the observation result from at least one sensor (2, 3) configured to observe the surrounding conditions, and estimates and tracks the state of an object, which is the state of an object present in the surroundings, based on the at least one piece of observational information,
The system includes a state update unit (S10 to S100) configured to re-estimate the state of the object based on the acquired observation information each time at least one observation information is acquired from the at least one sensor,
The aforementioned state update unit,
A probability calculation unit (S10 to S30, S60 to S80) is configured to calculate the probability that the acquired at least one observation information is information obtained from the object corresponding to the immediately estimated object state, based on the at least one observation information from the at least one sensor and the object state estimated by the state update unit immediately before acquiring the at least one observation information,
The system includes a state estimation unit (S40-S50, S90-S100) configured to re-estimate the object state by generating object state estimation information indicating the estimated object state based on the at least one observation information from the at least one sensor, the object state estimated by the state update unit immediately before acquiring the at least one observation information, and the probability of identical objects calculated by the probability calculation unit,
The probability calculation unit,
The system includes a likelihood calculation unit (S10-S20, S60-S70) configured to calculate at least one likelihood for each of the at least one observational pieces of information that indicates the likelihood that it is information about the object corresponding to the object state estimated by the state update unit immediately before acquiring the at least one observational piece of information, and calculates the probability of identical objects using the at least one likelihood calculated by the likelihood calculation unit.
The likelihood calculation unit is a tracking device comprising a similarity calculation unit (S10, S60) configured to calculate the similarity between the multi-value observation information, which is observation information represented by multiple values from the at least one piece of observation information, and the object state estimation information, which indicates the object state estimated by the state update unit immediately before acquiring the multi-value observation information. A tracking device according to claim 1,
The state estimation unit,
A tracking device that generates object state estimation information by integrating, using the same-object probability, a first hypothetical object state which is the object state when it is assumed that the at least one piece of observational information is information about the object corresponding to the object state estimated by the state update unit immediately before acquiring the at least one piece of observational information, and a second hypothetical object state which is the object state when it is assumed that the at least one piece of observational information is not information about the object corresponding to the object state estimated by the state update unit immediately before acquiring the at least one piece of observational information. A tracking device according to claim 1 or claim 2,
The aforementioned at least one sensor is a plurality of sensors,
The plurality of sensors include a first sensor and a second sensor that are different from each other.
The aforementioned state update unit,
A first estimation unit (S10 to S50) estimates the object state as a first object state using the at least one observation information obtained from the first sensor,
A tracking device comprising: a second estimation unit (S60 to S100) that estimates the object state as a second object state using the at least one observation information obtained from the second sensor and the first object state estimated by the first estimation unit. A tracking device according to claim 1 or claim 2,
The state estimation unit is a tracking device configured to calculate the probability of the object's existence, in addition to the object's state. A tracking device according to claim 1 or claim 2,
The state update unit is a tracking device that updates the object state using random finite set theory, such as a PHD filter, LMB filter, PMBM filter, or PMB filter. A tracking device according to claim 1 or claim 2,
The aforementioned multi-value observation information is a tracking device that provides multidimensional feature information of the object. A tracking device according to claim 1 or claim 2,
A Mahalanobis distance calculation unit (S22, S72) is configured to calculate the Mahalanobis distance between the at least one observation information obtained from the at least one sensor and the object state estimation information generated by the state estimation unit,
A tracking device comprising: an observation determination unit (S24, S74) configured to determine that the object corresponding to the object state estimation information was not observed by the at least one sensor when the Mahalanobis distance is greater than a preset threshold.