JP7488673B2 - MOVING OBJECT TRACKING DEVICE, MOVING OBJECT TRACKING METHOD, AND MOVING OBJECT TRACKING PROGRAM - Google Patents

Description

This invention relates to a technology for tracking a moving object based on sequentially captured images.

For security purposes, moving objects such as people are tracked by analyzing images captured sequentially by a camera, and a particle filter is known as one of the techniques used in tracking processing. With a particle filter, multiple hypotheses are set that represent candidate destinations (candidate positions) for each moving object being tracked, and the position of the moving object is determined based on the degree to which the characteristics of the moving object appear at each candidate position in the captured images.

For example, Patent Document 1 describes a moving object tracking device that sets candidate positions for each part that constitutes a moving object. In this moving object tracking device, the total number of candidate positions to be set for each moving object is determined in advance, and the number of candidate positions to be assigned to each part is determined according to the past tracking reliability of that part, thereby allocating hypotheses for parts that have become difficult to track due to occlusion caused by pose change to other parts, preventing a decrease in hypotheses that are effective for tracking.

JP 2016-170603 A

However, in conventional technology, the number of candidate positions per moving object was a fixed value. Therefore, when applied to tracking in crowded areas such as event venues, as the level of congestion increases, the number of candidate positions per captured image increases too much, resulting in significant delays in the tracking process.

To prevent delays, it is also possible to determine the number of candidate positions per captured image and divide this number equally among the moving objects being tracked. However, in that case, as the level of congestion increases, the number of candidate positions per moving object becomes too small, which can lead to a problem of reduced tracking accuracy.

The present invention has been made in consideration of the above problems, and aims to provide a moving object tracking technology that can suppress delays in tracking processing and reductions in accuracy even when the degree of congestion of moving objects becomes high.

(1) The moving object tracking device according to the present invention is a moving object tracking device that tracks one or more moving objects based on sequentially captured images, and includes a congestion degree estimation means that estimates the congestion degree of the moving objects captured in the captured images, a candidate position number allocation means that allocates a number of candidate positions for each moving object according to the congestion degree, a candidate position setting means that determines candidate positions of the moving object at the current time from past positions of the moving object, the same number as the number of candidate positions assigned to each moving object, and an object position determination means that determines the object position of the moving object at the current time based on the multiple candidate positions of each moving object.

(2) In the moving object tracking device according to the present invention described in (1) above, the congestion degree estimation means estimates the congestion degree of an arbitrary position in the captured image by inputting the captured image to an estimator that has been trained in advance to output the congestion degree of an arbitrary position in the captured image when the captured image is input, and the candidate position number allocation means allocates the number of candidate positions for each moving object according to the congestion degree of the object position or candidate positions of the moving object in the captured image.

(3) In the moving object tracking device according to the present invention described in (1) or (2) above, the candidate position number allocation means allocates a larger number of candidate positions for the moving object as the congestion degree increases.

(4) In the moving object tracking device according to the present invention described in (3) above, the candidate position number allocation means compresses the number of candidate positions of each moving object so that the sum of the candidate positions of all moving objects to be tracked at one time falls within the upper limit of the total number.

(5) In the moving object tracking device according to the present invention described in (1) to (4) above, the candidate position setting means sets the range of the candidate positions of the moving object to a wider range as the congestion degree becomes smaller.

(6) In the moving object tracking device according to the present invention described in (1) above, in an area where the degree of congestion is equal to or higher than a certain level, the candidate position number allocation means, the candidate position setting means, and the object position determination means do not perform processing, and the positions of the group of moving objects on the sequentially captured images are stored.

According to the present invention, it is possible to suppress delays in tracking processing and deterioration in accuracy even when the degree of congestion of moving objects increases.

1 is a block diagram showing a schematic configuration of a moving object tracking device. 1 is a functional block diagram showing an outline of a moving object tracking device. FIG. 13 is a schematic diagram showing a process performed by a congestion degree estimation unit. FIG. 13 is a diagram showing a function of the number of candidate locations proportional to the degree of congestion. 1 is a schematic flow diagram showing the overall processing of a moving object tracking device according to an embodiment. FIG. 13 is a diagram illustrating a function of the number of candidate locations according to the degree of congestion in a modified example.

The following describes an embodiment of the present invention (hereinafter referred to as "embodiment") with reference to the drawings. In this embodiment, a moving object tracking device has a camera unit and a display unit connected to a computer, and the computer analyzes images captured sequentially by the camera unit to track a person, which is a moving object.

The moving object tracking device 1 shown as an example in this embodiment determines, for each moving object, a candidate position at the current time from the past position, and determines the current position based on the degree to which the image features of the moving object appear at the candidate position in the captured image at the current time. Hereinafter, the candidate positions of the moving object are referred to as candidate positions. It is assumed that multiple candidate positions are set per moving object at one time, and the number of candidate positions is referred to as the number of candidate positions. In addition, the position of the moving object is referred to as the object position. It is assumed that one object position is set per moving object at one time. In other words, the candidate positions are candidates for the object position, and one object position is determined by integrating multiple candidate positions. It should be noted that the moving object tracking device 1 is characterized by assigning the number of candidate positions to each moving object according to the degree of congestion.

Figure 1 is a block diagram showing the general configuration of a moving object tracking device 1. The moving object tracking device 1 is composed of a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a display unit 6.

The photographing unit 2 is a so-called surveillance camera. The photographing unit 2 is connected to the image processing unit 5 via the communication unit 3, photographs the monitored space at a predetermined time interval to generate images, and sequentially inputs the generated images to the image processing unit 5. For example, the photographing unit 2 is installed on a pole erected in an event venue, which is the monitored space, with a fixed field of view that overlooks people present in the monitored space, photographs the monitored space at time intervals of 1/5 seconds, and generates color images. Note that the photographing unit 2 may generate monochrome images instead of color images.

The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5 and the other end of which is connected to the photographing unit 2 and the display unit 6. The communication unit 3 acquires images from the photographing unit 2 and inputs them to the image processing unit 5. The communication unit 3 also outputs the tracking results of the moving object from the image processing unit 5 to the display unit 6.

The imaging unit 2, communication unit 3, storage unit 4, image processing unit 5, and display unit 6 are appropriately connected in a manner appropriate to the installation location of each unit. For example, if the imaging unit 2, communication unit 3, and image processing unit 5 are installed remotely, the imaging unit 2 and communication unit 3 can be connected via an Internet line. Also, the communication unit 3 and image processing unit 5 can be connected via a bus. Other connection means that can be used include a LAN (Local Area Network), various cables, etc.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. For example, the storage unit 4 stores data for learning and information on an estimator, which is a trained model, and inputs and outputs this information between the image processing unit 5. In other words, information used to train the estimator and information generated during the process are input and output between the storage unit 4 and the image processing unit 5.

The image processing unit 5 is composed of arithmetic devices such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit), and a GPU (Graphics Processing Unit). The image processing unit 5 operates as various processing means and control means by reading and executing programs from the storage unit 4, reading various data from the storage unit 4 as necessary, and storing the generated data in the storage unit 4. For example, the image processing unit 5 learns and generates an estimator, and stores the generated estimator in the storage unit 4 via the communication unit 3.

The display unit 6 is a liquid crystal display or an organic EL (Electro-Luminescence) display, etc., and displays the tracking results of the moving object input from the image processing unit 5 via the communication unit 3.

Figure 2 is a schematic functional block diagram of the moving object tracking device 1, in which the memory unit 4 functions as a congestion level storage means 40 and an object information storage means 41, and the image processing unit 5 functions as a congestion level estimation means 50, a candidate position number allocation means 51, a candidate position setting means 52, and an object position determination means 53. In addition, the communication unit 3 cooperates with the image processing unit 5 and functions as an image acquisition means 30 and a tracking result output means 31.

The image acquisition means 30 acquires images captured sequentially from the image capture unit 2 and outputs them to the congestion degree estimation means 50 and the object position determination means 53.

The congestion degree estimation means 50 estimates the congestion degree of moving objects captured in the captured image. The congestion degree estimation means 50 may estimate (an approximation of) the congestion degree at the current time, for example, by counting the number of object positions one time before in the entire captured image or for each block obtained by dividing the captured image into multiple blocks. However, if the congestion degree is calculated from the tracking result and used for tracking, a decrease in tracking accuracy will also decrease the congestion degree estimation accuracy based on the tracking result, doubling the decrease in tracking accuracy. For this reason, it is desirable to estimate the congestion degree by a process independent of tracking, so as to prevent the errors in tracking and congestion degree estimation from doubling.

That is, preferably, the congestion degree estimation means 50 estimates the congestion degree of a moving object photographed at an arbitrary position on the photographed image at the current time from the photographed image at the current time, without being based on past photographed images. In this embodiment, the congestion degree estimation means 50 inputs the photographed image to an estimator that has been trained in advance to output the congestion degree of an arbitrary position in the photographed image when the photographed image is input, and estimates the congestion degree of an arbitrary position in the photographed image. Specifically, the congestion degree estimation means 50 inputs the photographed image to an estimator that has been trained in advance to output a congestion degree map that estimates the congestion degree of each pixel when the photographed image is input, and causes the estimator to output a congestion degree map of the photographed image, and stores the obtained congestion degree map in the congestion degree storage means 40.

Specifically, the estimator can be realized using deep learning technology. That is, the estimator can be modeled as a CNN (convolutional neural network) that receives a photographed image and outputs a congestion map of the photographed image. For learning, for example, a large number of learning images of a crowd are prepared, and a congestion map is prepared in which a probability density function having an average value of the center of gravity of each person's head in each learning image and a variance according to the size of the head is set and the value of the function for each head is added for each pixel. Then, learning is performed in advance to make the output when each learning image is input into the model closer to the congestion map corresponding to the image. The learned model obtained in this way is stored in the memory unit 4 as an estimator that is part of the program of the congestion degree estimation means 50. For example, the MCNN (multi-column convolutional neural network) described in "Single image crowd counting via multi-column convolutional neural network", Zhang, Y., Zhou et al., CVPR 2016, is an example of an estimator, and the crowd density map described in the same paper is an example of a congestion map.

The congestion degree estimation means 50 also detects ultra-high congestion areas where the congestion degree is equal to or greater than the threshold T3, and stores information about the ultra-high congestion areas in the congestion degree storage means 40. Ultra-high congestion areas occur when moving objects are crowded together in a position with a small depression angle (a position that appears as if the object is photographed from the side). For example, if the moving object is a person, only the head and shoulders will be captured in the ultra-high congestion area, and occlusion of the heads will occur, making tracking difficult. The threshold T3 is determined in advance through prior experiments.

The congestion degree storage means 40 stores the congestion degree estimated for the captured image. Specifically, it cyclically stores a predetermined number of the most recent congestion degree maps. Figure 3 is a schematic diagram showing the processing of the congestion degree estimation means 50. Figure 3(b) shows the congestion degree map output for the captured image in Figure 3(a). Figure 3(c) shows the congestion degree on line 510 in Figure 3(a). The more crowded the location, the higher the value output, and the congestion degrees at the positions of people 501, 502, and 503 are estimated to be 0.2, 1.2, and 2.8, respectively.

The object information storage means 41 stores information (object information) about moving objects being tracked. Specifically, it stores a template of the moving object, an object position of the moving object, and a hypothesis for the moving object in association with an object ID that identifies each moving object being tracked. The object information storage means 41 also stores a shape model of the moving object.

An example of a template for a moving object is a template image obtained by cutting out the area of a person from a photographed image. Instead of or in addition to the template image, features such as a color histogram or brightness gradient calculated from the template image may be used. A hypothesis for a moving object is information that associates an object ID, a hypothesis ID, a candidate position, and a likelihood, and is generated for each combination of moving objects and candidate positions. The object position and candidate positions of a moving object are represented by the x-axis and y-axis in the horizontal and vertical directions of the photographed image, respectively, and the position in the photographed image is represented by a pair of x- and y-coordinates (x, y). An example of a shape model for a moving object is graphic data of an ellipse that approximates a person in a standing position. Instead of an ellipse, a rectangle or a figure of three connected ellipses that approximate the head, torso, and legs may be used. Preferably, the shape model corresponds to the difference in size of the image of a person at each position in the photographed image, and is composed of reference graphic data and data that associates the position with the size of the graphic data.

The candidate position number allocation means 51 allocates a number of candidate positions to each moving object according to the degree of congestion. As an example, the candidate position number allocation means 51 allocates a larger number of candidate positions to a moving object as the degree of congestion increases.

Essentially, increasing the number of candidate positions increases the likelihood that candidate positions with small deviations (positional shifts) from the true object position will be included, allowing for stricter comparisons between moving objects and improved tracking accuracy. For example, template matching that is shifted from the true object position involves comparisons with background features, so it does not result in a strict comparison between the features of moving objects.

However, the more candidate positions there are, the greater the processing load becomes. Furthermore, as the degree of congestion increases, the distance between moving objects becomes closer and the number of nearby moving objects also increases, making it easier for tracking accuracy to decrease due to position shifts. On the other hand, as the degree of congestion decreases, the distance between moving objects becomes farther and the number of nearby moving objects decreases, making it less likely for tracking accuracy to decrease due to position shifts.

Therefore, by increasing the number of candidate positions for each moving object as the degree of congestion increases, it is possible to prevent a decrease in tracking accuracy due to congestion, and by keeping the number of candidate positions for each moving object low when the degree of congestion is low, it is possible to reduce the processing load.

In this way, for example, when estimating one congestion level for the entire captured image, even if a processing delay occurs during a period when congestion is high, the delay can be eliminated in a short time during the subsequent period when congestion levels decrease. Also, for example, when estimating one congestion level for each captured image captured by multiple image capture units, the bias in the distribution of moving objects among the captured images can be utilized to reduce the processing load while preventing a decrease in tracking accuracy due to congestion.

The candidate position number allocation means 51 may also be configured to allocate the number of candidate positions for a moving object according to the degree of congestion at the object position of each moving object in the captured image. In other words, the number of candidate positions is increased for moving objects whose object positions are predicted to be highly crowded, and the number of candidate positions for moving objects whose object positions are predicted to be low crowded. In this way, the bias in the distribution of moving objects in the captured image can be utilized to prevent a decrease in tracking accuracy due to congestion while reducing the processing load.

Specifically, the candidate position number allocation means 51 determines the number of candidate positions for each moving object by referring to the congestion map stored in the congestion degree storage means 40 and the object positions and number of moving objects for each moving object stored in the object information storage means 41, and outputs the determined number of candidate positions to the candidate position setting means 52.

A specific example of a method for calculating the number of candidate positions to be assigned to each moving object by the candidate position number allocation means 51 is shown below. Functions (a) to (d) shown in FIG. 4 are functions for calculating the number of candidate positions according to the congestion level, and the number of candidate positions is calculated from the congestion level based on these functions.

(d) is the simplest function for calculating the number of candidate positions, and the number of candidate positions increases in proportion to the degree of congestion. In contrast, in (c), when the degree of congestion is less than the threshold T1, the number of candidate positions is constant, and when the degree of congestion is equal to or greater than the threshold T1, the number of candidate positions increases in proportion to the degree of congestion. When the degree of congestion is less than the threshold T1, the number of candidate positions is the minimum value of the number of candidate positions to be assigned to one moving object, and is set through prior experiments. By setting the minimum value, it is possible to prevent the degree of congestion at the position where the moving object actually exists from being low due to the influence of an error in the degree of congestion estimation, and the number of candidate positions being set to an extremely small number. On the other hand, the range in which a person to be tracked can move is limited to a certain extent, and even if the number of candidate positions is increased more than necessary, an improvement in accuracy cannot be expected. For this reason, in (b), when the degree of congestion is equal to or greater than the threshold T2 at which the number of candidate positions reaches a certain number (maximum value), the number of candidate positions is fixed at the maximum value even if the degree of congestion further increases. (a) is a function in which both the minimum and maximum values are set for the number of candidate positions. Also, T3 in FIG. 4 is a threshold value for discriminating ultra-high congestion areas.

[Compression of the number of candidate positions by the total upper limit]
When the number of candidate positions for a person to be tracked is determined by the above function, as the number of people existing in a highly congested area increases, the total number of candidate positions for all people to be tracked increases, causing a delay in processing.

For example, if the monitored space is an area where traffic is waiting for the light to turn at an intersection, the degree of congestion fluctuates periodically depending on the signal cycle, so even if delays occur temporarily due to high congestion, it is possible to recover from the delay once the congestion is resolved. On the other hand, congestion may continue for long periods at places such as the entrances and exits of an event venue. In such monitored spaces, recovery is difficult once a delay occurs, so it is desirable to set an upper limit on the total number of candidate locations in advance to prevent delays from occurring.

Specifically, the number of candidate positions of each moving object once calculated for each person in the above process is compressed so that the sum of candidate positions of all moving objects to be tracked at one time falls within the total upper limit value. The number of candidate positions for a certain tracking person i can be expressed by the following formula (1), where f(d _i ) is the number of candidate positions before compression calculated from the congestion degree d _i and function f(x). The total upper limit value is the upper limit number of candidate positions to be assigned at one time. In other words, it is the upper limit value of the sum of the number of candidate positions. The total upper limit value is set in advance to a value within a range that does not cause processing delays depending on the performance of the moving object tracking device, etc.

Here, min(1.0,total upper limit value/ _Σjf ( _dj )) is the compression rate of the number of candidate locations, and by setting 1.0 as the maximum value, the number of candidate locations is maintained if the total number of candidate locations _Σjf ( _dj ) has not reached the total upper limit value. This is because it is assumed that the number of candidate locations before compression is a necessary and sufficient number. By setting a total upper limit value in this way, it becomes possible to allocate the number of candidate locations according to the degree of congestion without exceeding the total upper limit value, so that it is possible to prevent a decrease in tracking accuracy and the occurrence of delays even in a monitored space where congestion continues for long periods of time.

[Minimum number of candidate positions for each tracked person]
When compressing the number of candidate positions using the total number upper limit, the number of compressed candidate positions becomes significantly smaller, and there is a possibility that tracking performance in low-congestion areas will be extremely degraded. To prevent this, a lower limit may be set for the number of candidate positions for a person to be tracked. In formula (2), the lower limit is assigned to each moving object, and the number of assigned candidate positions is subtracted from the total number upper limit to obtain the remaining number, which is then distributed according to the ratio of the number of candidate positions of each moving object before compression minus the lower limit. The lower limit is the minimum number of candidate positions to be assigned to one moving object. The lower limit is set to the minimum number of assignments that can obtain tracking accuracy equal to or higher than the target value in a low-congestion area through prior experiments.

The candidate position number allocation means 51 determines, for each moving object, an area in which the moving object may exist from the object position of the moving object, and obtains the congestion degree of the area from the congestion degree map to determine the congestion degree of the area in which the moving object may exist. In this embodiment, the candidate position number allocation means 51 extrapolates the object position at the current time to the past object positions stored in the object information storage means 41 for each person, and obtains the congestion degree of the extrapolated object position at the current time. However, for a person who does not have two or more past object positions, the congestion degree of the object position one time ago is obtained. In that case, it is desirable to use the congestion degree map from one time ago as well. In another embodiment, the candidate position number allocation means 51 reads the extrapolated object position at the current time and the congestion degree around it for each person, and determines the representative value of these as the congestion degree of the area in which the moving object may exist. The representative value can be the average value, the mode value, or the maximum value. However, for a person who does not have two or more past object positions, the representative value of the object position one time ago and the congestion degree around it is used. Furthermore, the congestion degree may be obtained by applying the candidate position one time ago to the congestion degree map one time ago. In this case, the congestion degree of the moving objects is calculated by weighting the congestion degree from one time point before with the likelihood of the candidate position from one time point before.

The candidate position setting means 52 determines candidate positions of the moving object at the current time from the past positions of the moving object, the same number as the number of candidate positions assigned to each moving object. That is, for each moving object being tracked, the means sets candidate positions of the moving object at the current time from the moving object's past object positions, or past candidate positions, or past object positions and candidate positions, the same number as the number of candidate positions assigned to the moving object, and sets hypotheses including each candidate position of the moving object.

Specifically, the number of candidate positions input from the candidate position number allocation means 51 and the past object positions and/or past hypotheses of each moving object stored in the object information storage means 41 are referenced as necessary to set a hypothesis for each moving object at the current time, and the set hypothesis is stored in the object information storage means 41. Furthermore, a congestion degree map may be input from the congestion degree storage means 40 and referenced.

The candidate position setting means 52 sets, for each moving object, a candidate position where the moving object is predicted to exist at the current time. The i-th candidate position P _i,t at the current time t can be calculated based on the past object position or the past candidate position by any of the following three methods:

[First Method]
This method distributes candidate positions P _i,t around the object position Q _t-1 from one time before. The candidate position setting means 52 sets a diffusion amount Δ0 for each moving object based on a random number and adds the diffusion amount Δ0 to the object position Q _t-1 to set the candidate position P _i,t (P _i,t = Q _t-1 + Δ0) the same number of times as the number of candidate positions.

[Second Method]
This method distributes candidate positions P _i,t around a predicted value _qt that predicts the current object position. The candidate position setting means 52 first predicts the predicted value _qt for each moving object by applying a motion model such as a constant velocity motion model or a prediction filter such as a Kalman filter to a plurality of past object positions, and then sets a diffusion amount Δ1 based on a random number and adds the diffusion amount Δ1 to the predicted value _qt to obtain the candidate position P _i,t (P _i,t = _qt + Δ1), performing this process the same number of times as the number of candidate positions.

[Third Method]
This is a method for predicting a current candidate position from a past candidate position. First, the candidate position setting means 52 selects, for each moving object, the same number of hypotheses as the number of candidate positions from the hypotheses one time ago, allowing overlapping, based on a random number. The winning probability of each hypothesis one time ago is, for example, the ratio of the likelihood of the hypothesis to the sum of the likelihoods of the hypotheses one time ago for the moving object. Next, for each selected hypothesis, the candidate position setting means 52 applies the above-mentioned motion model or the above-mentioned prediction filter to the candidate positions of the hypothesis and multiple past hypotheses corresponding to the hypothesis to calculate a predicted value q _i,t for each moving object, and sets a diffusion amount Δ2 based on a random number, and adds the diffusion amount Δ2 to the predicted value q _i, t to obtain a candidate position P _i,t (P _i,t = q _i,t + Δ2).

In this embodiment, until the speed can be calculated, the candidate positions are calculated using the first method (times 2 to 5, including the time when the object position determination means 53 registers the object as a new moving object), and once the speed can be calculated, the candidate positions are calculated using the third method.

The range in which the candidate positions of the moving object are set may be wider as the degree of congestion becomes smaller. That is, the candidate position setting means 52 sets the diffusion amount Δ0 to Δ2 described in the first to third methods to a larger amount as the degree of congestion becomes smaller. For example, the diffusion amount Δ0 to Δ2 is set to a range of 0 cm to 60 cm for a person in an area with a high degree of congestion, and is set to a range of 30 cm to 100 cm for a person in an area with a low degree of congestion. The reason for this is as follows. That is, when the degree of congestion is small, the range in which the moving object can move becomes wide, so if the candidate positions are not set to a wide range, matching failures are likely to occur. Therefore, by controlling the range in which the candidate positions are set based on the degree of congestion, it is possible to prevent matching failures when the degree of congestion is small. In particular, since the tracking parameters can be controlled by the degree of congestion estimated independently of tracking, highly reliable control is possible.

The object position determination means 53 determines the object position of each moving object at the current time based on multiple candidate positions for each moving object. That is, for each moving object being tracked, the degree to which the feature of the moving object appears at a position on the captured image at the current time corresponding to the candidate position indicated by each of multiple hypotheses set for the moving object at the current time is calculated as the likelihood of the hypothesis, and the candidate positions indicated by the multiple hypotheses are integrated according to the likelihood of the hypothesis to calculate the object position of the moving object at the current time. However, hypotheses that are located outside the captured image and hypotheses that are located in an extremely crowded area are excluded from the calculation in advance because the reliability of the likelihood is low.

Specifically, the object position determination means 53 reads out the current hypothesis, shape model, and template for each moving object stored in the object information storage means 41, and for each moving object, places the shape model at the candidate position indicated by the hypothesis for that moving object, extracts a partial image of the area overlapping with the shape model from the captured image, calculates the similarity between the image features of the partial image and the image features of the template for that moving object, and calculates a likelihood according to the similarity. The object position determination means 53 also causes the object information storage means 41 to additionally store the likelihood in association with the hypothesis.

At this time, the object position determination means 53 may further perform a difference process between the background image and the captured image to detect a background difference region, and correct the likelihood for each moving object with a correction value according to the proportion of the background difference region in the partial image of the moving object. The background image is stored in the storage unit 4 for the new moving object detection process described later.

The object position determination means 53 calculates the object position of each moving object by integrating the candidate positions indicated by the hypotheses of the moving object based on the likelihood of the hypotheses. Preferably, the weighted average value obtained by weighting the candidate positions indicated by the hypotheses of the moving object by the likelihood of the hypotheses is determined as the object position of the moving object. In addition, it is also possible to select a plurality of likelihoods equal to or greater than a preset threshold TL, and determine the weighted average value obtained by weighting the candidate positions paired with the selected likelihoods by the likelihoods and averaging them as the object position. It is also possible to select a predetermined number of likelihoods in descending order of likelihood, and determine the weighted average value obtained by weighting the candidate positions paired with the selected likelihoods by the likelihoods and averaging them as the object position. In addition, it is also possible to select the maximum likelihood for each moving object, compare the likelihood with the threshold TL, and determine the candidate position paired with the likelihood as the object position if the maximum likelihood is equal to or greater than the threshold TL.

The calculated likelihood is associated with the hypothesis and stored in the object information storage means 41, and the calculated object position is associated with the moving object and stored in the object information storage means 41.

The object position determination means 53 also performs processing to update the object position and template for the moving object being tracked, determines the presence of a new moving object, registers object information for the new moving object, and processes a disappeared moving object. Below, the processing for the moving object being tracked, the processing for the new moving object, and the processing for the disappeared moving object will be explained in turn.

[Tracking moving object]
The object position determination means 53 additionally stores the determined object position of the moving object in the object information storage means 41, and updates the template of the moving object with the image features at the current time. The updating may be performed by replacing the extracted image features with the stored image features, or by taking a weighted average of the extracted image features and the stored image features.

[New moving object]
The object position determination means 53 performs a difference process between a background image taken when there is no moving object in the monitoring space and the taken image to detect a background difference region, and also arranges a shape model at each object position at the current time to extract a background difference region that does not overlap with any shape model. Then, if the non-overlapping background difference region has an area that is effective as a moving object (person), the object position determination means 53 determines that a new moving object exists in the non-overlapping background difference region. The object position determination means 53 applies a shape model to the non-overlapping background difference region to determine the object position and region of the new moving object, and stores the template of the moving object and the object position of the moving object in the object information storage means 41 in association with the object ID. In addition, the object position determination means 53 stores the image taken when there is no moving object in the storage unit 4 as a background image, and updates the background image with the image taken in the region where the background difference region was not detected. Note that when detecting a background difference region in the calculation of the likelihood, it is sufficient to use it without performing a new detection.

[Disappearing Moving Object]
The object position determination means 53 determines a moving object whose likelihood of all candidate positions is equal to or less than the threshold TL as a lost moving object, and deletes the object information of the moving object. A moving object that is entirely hidden by an obstruction, a moving object that has moved outside the captured image, or a moving object that has moved into an ultra-highly congested area is determined to be a lost moving object. Then, the moving object is stored in an ultra-highly congested area storage means (not shown) as a group of moving objects not to be tracked. Note that when the moving object leaves the ultra-highly congested area, it is tracked again as a new moving object.

The tracking result output means 31, for example, generates a trajectory image in which a series of object positions for each moving object being tracked is plotted, and generates a congestion image in which a color corresponding to the congestion level is determined in advance and a pixel value of a color corresponding to the congestion level of each pixel in the congestion level map is set for each pixel corresponding to that pixel, and outputs an image in which the trajectory image and the congestion level image are transparently synthesized to the display unit 6. In addition, the captured image at the current time may be superimposed.

[Example]
The operation of the moving object tracking device 1 will be described below. Fig. 5 is an overall flow diagram of the operation of the moving object tracking device 1. When the operation of the moving object tracking device 1 is started, the image capture unit 2 acquires captured images at a predefined frame rate and outputs the captured images to the image processing unit 5. Every time a captured image is input (step S1), the image processing unit 5 repeats a series of processes from steps S2 to S11.

The image processing unit 5 outputs a congestion map for the captured image acquired by the image capturing unit 2 using the congestion estimation means 50. Furthermore, the congestion degree of each moving object to be tracked recorded in the object information storage means 41 is calculated by referring to the value of the congestion map corresponding to the predicted object position of the moving object to be tracked at the current time. In addition, an ultra-high congestion area where tracking becomes difficult is extracted (step S2).

The image processing unit 5 assigns the number of candidate positions to each person based on the degree of congestion of each tracked person using the candidate position number assignment means 51 (step S3).

The image processing unit 5 performs a tracking process on the person to be tracked and estimates the current object position (steps S4 to S9). The image processing unit 5 sequentially sets the people recorded in the object information storage means 41 as people of interest and performs tracking process on the people of interest. When tracking process is complete for all people, the image processing unit 5 proceeds to step S10, whereas if there are unprocessed people, the tracking process continues (step S9).

The tracking process of steps S4 to S9 will be explained in more detail below. The image processing unit 5 functions as a candidate position setting means 52, and sets hypotheses based on the number of candidate positions assigned to the person of interest in step S3 (step S4). That is, the candidate position setting means 52 extrapolates the current object position to the past object position of the person of interest, sets the same number of candidate positions in the vicinity as the assigned number, and sets hypotheses including each candidate position. Thereafter, hypotheses in which the candidate positions are outside the captured image or within the ultra-highly congested area extracted in step S2 are deleted.

The image processing unit 5 functions as an object position determination means 53, which calculates the likelihood for each hypothesis set in step S4 (step S5).

Then, the object position determination means 53 determines whether the target object is a vanishing moving object (step S6), and performs tracking termination processing if it is determined to be a vanishing moving object (step S7). That is, if the likelihood of all hypotheses remaining in step S4 that were not deleted is less than the threshold value TL, the object position determination means 53 determines that the target person is a vanishing moving object and deletes information about the target person from the object information storage means 41.

If it is determined in step S6 that tracking can be continued, the object position determination means 53 estimates the object position of the tracked person based on the candidate positions of the hypothesis group set in step S4 and the likelihood calculated in step S5 (step S8).

When the above-mentioned tracking processes S4 to S9 have been performed for all people registered in the object information storage means 41, as already mentioned, the image processing unit 5 advances the process to step S10, and the object position determination means 53 detects people in the captured image who have not yet been set for tracking, and if detected, adds them as new people to be tracked (step S10).

When tracking of people is completed for the captured image input in step S1 by the above-mentioned processes S2 to S10, the image processing unit 5 outputs the tracking results to the display unit 6 (step S11). For example, the image processing unit 5 causes the display device of the display unit 6 to display the estimated positions of all people as the tracking results.

[Modification]
(1) In the above embodiment, an example was shown in which the congestion level estimation means 50 used an estimator that outputs a continuous value. However, an estimator that outputs a discrete congestion level may also be used.

For example, the estimator is modeled using a multi-class SVM (Support Vector Machine), and the model is trained using learning images that are classified and labeled into five classes according to the degree of congestion: "background (unmanned)", "low congestion", "medium congestion", "high congestion", and "very high congestion". The congestion degree estimation means 50 then sets a window centered on each pixel of the captured image, inputs the image features within the window to the estimator, and identifies the class of each pixel. In this way, when the congestion degree estimation means 50 estimates a discrete congestion degree, the candidate position number allocation means 51 calculates the number of candidate positions using, for example, the function shown in FIG. 6. FIG. 6 assumes a case in which the congestion degree is estimated discretely in five stages, and the number of candidate positions also changes discretely according to each congestion degree stage.

In addition to the multi-class SVM, the estimator can also be realized by various multi-class classifiers trained by the decision tree type random forest method, the multi-class AdaBoost method, or the multi-class logistic regression method. Alternatively, the estimator can be realized by a discriminative CNN (window scanning is not required for CNN). Even when using classified learning images, an estimator that outputs a continuous value of crowding degree can be realized by using a regression type model that regresses the crowding degree from the feature amount. In that case, the parameters of the regression function for calculating the crowding degree from the feature amount are learned by the ridge regression method, the support vector regression method, the regression tree type random forest method, or the Gaussian process regression. Alternatively, the estimator can be an estimator using a regression type CNN (window scanning is not required for CNN).

(2) In each of the above embodiments and each of the modified examples, an example has been shown in which the object position determination means 53 detects a new moving object based on background subtraction processing. Instead, however, a new moving object may be detected using a trained model that has been machine-learned from an unspecified number of images of the moving object to be tracked (e.g., deep learning from an unspecified number of images of people). In this case, the object position determination means 53 inputs the captured image into the trained model to detect the area of the moving object, and determines that a new moving object exists in the area of the moving object where the area that does not overlap with any shape model is equal to or larger than the threshold value TD.

(3) In the above embodiment and its modified examples, tracking using one image capture unit is exemplified, but tracking can also be performed using multiple image capture units that share a field of view.

First, a three-dimensional coordinate system common to multiple image capture units is defined, the relationship between the position and orientation of each image capture unit in the common three-dimensional coordinate system is found, and the parameters required for coordinate conversion from the three-dimensional coordinate system to the image coordinate system of each image capture unit are calculated. The object position and candidate positions are represented by a set of X, Y, and Z coordinates (X, Y, Z) in a three-dimensional coordinate system represented by orthogonal X and Y axes and a vertical Z axis on a horizontal plane such as the ground in the monitored space.

The congestion degree estimation means 50 first estimates a congestion degree map for the images captured by all the imaging units. Then, the congestion degree for the moving object in the common three-dimensional coordinate system is calculated as an integrated congestion degree based on the congestion degree map of each imaging unit. Specifically, for each moving object, the object position in the common three-dimensional coordinate system is converted to the image coordinate system of each imaging unit, the position on the congestion degree map of each imaging unit is obtained, and the congestion degree on the image captured by each imaging unit is calculated. Then, for each moving object, the highest value of the congestion degrees for each imaging unit is calculated as the overall congestion degree of the moving object. The reason why the overall congestion degree is set to the highest value of the congestion degree for each imaging unit is that when allocating the number of candidate positions based on the overall congestion degree, the number of candidate positions required to maintain tracking accuracy can be met for each imaging unit by matching it to the imaging unit with the highest congestion degree, which requires the largest number of candidate positions.

Although the object position at the current time is unknown at the time of congestion degree estimation, the congestion degree estimation means 50 predicts the object position at the current time for each moving object from the object's past positions, speed, etc., and calculates the overall congestion degree of each moving object using the predicted object position at the current time.

The candidate position number allocation means 51 uses the overall congestion degree to allocate the number of candidate positions for moving objects in the same manner as in the above embodiment, and the candidate position setting means 52 sets candidate positions in a common three-dimensional coordinate system based on the allocated number. As with the calculation of the overall congestion degree, the likelihood of the candidate positions is calculated by converting the coordinates of each candidate position to a position on the captured image for each capture unit, and comparing the features with a template at the calculated position on the captured image to calculate the likelihood for each capture unit, and averaging them to determine the likelihood of the final hypothesis (candidate position). The object position at the current time is calculated from the likelihood of each hypothesis (candidate position) in the same manner as in the embodiment.

(4) The present invention can also be applied to moving objects other than people that may cause congestion, such as vehicles and animals.

1...moving object tracking device, 2...photographing unit, 3...communication unit, 4...storage unit, 5...image processing unit, 6...display unit, 30...image acquisition unit, 31...tracking result output means, 40...congestion level storage means, 41...object information storage means, 50...congestion level estimation means, 51...candidate position number allocation means, 52...candidate position setting means, 53...object position determination means

Claims (7)

A moving object tracking device that tracks one or more moving objects based on sequentially captured images,
A congestion degree estimation means for estimating a congestion degree of the moving objects captured in the captured image;
a candidate position number allocation means for allocating a candidate position number for each moving object according to the congestion degree;
a candidate position setting means for determining candidate positions of the moving object at the current time from past positions of the moving object, the number of which is the same as the number of candidate positions assigned to each of the moving objects;
an object position determining means for determining an object position of the moving object at a current time based on the plurality of candidate positions for each of the moving objects;
Equipped with
The moving object tracking device according to claim 1, wherein the candidate position number allocating means allocates a larger number of candidate positions of the moving object as the degree of congestion increases . the congestion degree estimation means inputs the photographed image to an estimator that has been trained in advance to output the congestion degree at an arbitrary position within the photographed image when the photographed image is input, and estimates the congestion degree at an arbitrary position within the photographed image;
2. The moving object tracking device according to claim 1, wherein said candidate position number allocating means allocates the number of candidate positions of each moving object in accordance with the degree of congestion of the object position of the moving object or the candidate positions on the captured image. 3. The moving object tracking device according to claim 1 , wherein the candidate position number allocation means compresses the number of candidate positions of each moving object so that the sum of the candidate positions of all the moving objects to be tracked at one time falls within a total number upper limit value. 4. The moving object tracking device according to claim 1, wherein the candidate position setting means sets the candidate positions of the moving object in a wider range as the degree of congestion becomes smaller. A moving object tracking device that tracks one or more moving objects based on sequentially captured images,
A congestion degree estimation means for estimating a congestion degree of the moving objects captured in the captured image;
a candidate position number allocation means for allocating a candidate position number for each moving object according to the congestion degree;
a candidate position setting means for determining candidate positions of the moving object at the current time from past positions of the moving object, the number of which is the same as the number of candidate positions assigned to each of the moving objects;
an object position determining means for determining an object position of the moving object at a current time based on the plurality of candidate positions for each of the moving objects;
Equipped with
In an area where the degree of congestion is equal to or higher than a certain level, the candidate position number allocation means, the candidate position setting means, and the object position determination means do not perform processing,
A moving object tracking device characterized in that the positions of a group of moving objects on the sequentially captured images are stored. A moving object tracking method using a moving object tracking device that tracks one or more moving objects based on sequentially captured images, comprising:
Estimating a congestion degree of the moving objects captured in the captured image;
the higher the congestion level, the greater the number of candidate positions of each of the moving objects is allocated;
determining candidate positions of the moving object at the current time from past positions of the moving object, the number of which is equal to the number of candidate positions assigned to each of the moving objects;
determining an object position of the moving object at a current time based on the plurality of candidate positions for each of the moving objects. A moving object tracking program executed in a moving object tracking device that tracks one or more moving objects based on sequentially captured images, comprising:
A process of estimating a congestion degree of the moving objects captured in the captured image;
A process of allocating a number of candidate positions for each moving object according to the congestion degree;
A process of determining candidate positions of the moving object at the current time from past positions of the moving object, the number of which is equal to the number of candidate positions assigned to each of the moving objects;
A process of determining an object position of the moving object at a current time based on the plurality of candidate positions for each of the moving objects;
Run the command ,
A moving object tracking program, wherein the process of allocating the number of candidate positions allocates a larger number of candidate positions of the moving object as the degree of congestion increases.

Images