JP7548079B2 - OBJECT TRACKING DEVICE AND OBJECT TRACKING METHOD - Google Patents

Description

The present invention relates to an object tracking device and an object tracking method.

Object tracking involves a detection process to find the object to be detected (detection object) from the input image, and a tracking process to find an object identical to the detected object from each frame. In object tracking, a template that extracts the characteristics of the object to be tracked (tracking object) is used to find an area in the input image that has the same characteristics as the template. Here, the characteristics of the tracked object may change as the object moves or the background changes. Therefore, tracking can be performed with high accuracy by tracking while sequentially updating the template.

One of the conventional methods used for updating templates is a sequential learning method in which the results obtained by tracking are partially reflected in the template. With this method, the past state of the tracked object is taken into account when updating the template, making it possible to track robustly against changes in the subject. However, if learning fails once, the object may get stuck in the background. There is also a risk that tracking errors will accumulate and cause tracking to fail.

Another conventional method is the reset type method, which replaces the template every time it is updated. With this method, the object is less likely to stick to the background, but since past information about the object being tracked is not learned, tracking is likely to fail if the state of the subject changes suddenly from the previous frame.

As such, each method has its strengths and weaknesses. However, since all possible situations are possible for the target object, it is difficult to determine whether to use the sequential learning type or the reset type method for template update. In response to this, Patent Document 1 proposes a method of calculating the reliability of the tracking result, updating the template based on the reliability, and determining whether tracking should be terminated. The method of Patent Document 1 performs detection only in the first frame, and updates the template based on the reliability of the tracking result from the next frame onwards, and performs tracking using the updated template. However, in the case of a target object whose shape is constantly changing, even if tracking is performed correctly, the reliability is calculated low every frame, so there is a risk of erroneously ending tracking and making tracking prone to failure. Therefore, with a method such as Patent Document 1, the template cannot be updated accurately, and tracking may fail.

JP 2010-147950 A

The present invention has been made in consideration of the above circumstances, and aims to perform object tracking with greater accuracy than conventional methods.

To achieve the above objective, the present invention adopts the following configuration.

A first aspect of the present invention is an object tracking device that tracks an object included in an image consisting of a plurality of frames, comprising an image acquisition unit that acquires the image, a memory unit that stores features of a detected object and features of a tracked object, an update unit that updates the features of the tracked object in the memory unit, a detection unit that detects the detected object from the acquired image based on the features of the detected object stored in the memory unit, and a track unit that tracks the tracked object from the acquired image based on the features of the tracked object stored in the memory unit, wherein the object tracking device performs both detection of the detected object by the detection unit and tracking of the tracked object by the track unit for a current frame, the update unit updates the features of the tracked object based on the tracking result and the detection result, and the track unit performs the tracking using the updated features of the tracked object for the next frame.

The characteristics of the detected and tracked objects may be, for example, features related to color or shape. The detection and tracking results may include information about the detected or tracked area.

The object tracking device according to this embodiment performs both detection and tracking for each frame, and updates the template (characteristics of the tracked object) used for tracking using the tracking result or detection result. For the next frame, tracking is performed using the updated template. In this way, by performing tracking using a template updated based on the tracking result or detection result, a template that corresponds to changes in the tracked object or background can be used for tracking, enabling tracking with high accuracy.

The update unit in this embodiment may switch the template update method depending on the degree of match between the tracking result and the detection result. By switching the template update method depending on the degree of match, it is possible to prevent tracking errors from occurring in the next frame.

The template update method may be switched, for example, depending on whether the degree of match is higher than the first threshold value as follows. Here, the first threshold value is set to a value that can be considered to be correct if the degree of match is equal to or higher than that value. If the degree of match is higher than the first threshold value, the template may be updated using features of either or both of the detection result and the tracking result (first update process). On the other hand, if the degree of match is equal to or lower than the first threshold value, the template may be updated so that the detection result is more significantly affected than when the degree of match is higher than the first threshold (second update process). If the detection result and the tracking result do not match, it is assumed that there is an error in one of them. Although it is rare, the tracking process may capture an object that is significantly different from the target object, and the detection process rarely captures an object that is significantly different from the target object. Therefore, by prioritizing the detection result when the detection result and the tracking result do not match as described above, the possibility of erroneous tracking in the next frame can be reduced.

In this embodiment, the degree of match is an index that indicates the degree of match between an object obtained by detection and an object obtained by tracking. The degree of match can be calculated, for example, based on the difference in center position or the difference in size between the tracked object included in the tracking result and the detected object included in the detection result.

In this aspect, the update unit may update the template using the tracking result by the tracking unit for the current frame in the first update process, and may update the template using the detection result by the detection unit for the current frame in the second update process. Alternatively, the update unit may replace the detection result by the detection unit for the current frame as the template in the second update process. The update unit may also set a larger learning rate for template update in the second update process than in the first update process. In this way, when the degree of match is equal to or less than the first threshold, the template can be updated so that the detection result is more significantly affected than when the degree of match is higher than the first threshold.

The update unit in this aspect may switch the template update method according to the reliability of the tracking result or the detection result. The reliability of the tracking result or the detection result may be calculated, for example, using the difference between the tracking result or the detection result of the current frame and the background image. Note that the background image is an image when the detection object does not exist in the detection area. The reliability of the tracking result or the detection result may be calculated using the difference between the tracking position or the detection position between frames. The reliability of the tracking result may be calculated using a tracking score indicating the reliability of the tracking result, and the reliability of the detection result may be calculated using a detection score indicating the reliability of the detection result. In this way, by switching using the reliability and the detection score, the template can be updated more appropriately.

The template update method may be switched as follows depending on the reliability of the tracking result or the detection result. For example, the update unit may not update the template in the current frame when the reliability of both the tracking result and the detection result is equal to or lower than the second threshold. Alternatively, the update unit may perform the first update process regardless of the degree of match between the tracking result and the detection result when the reliability of the tracking result is higher than the second threshold and the reliability of the detection result is equal to or lower than the second threshold. Also, the update unit may perform the second update process regardless of the degree of match between the tracking result and the detection result when the reliability of the tracking result is equal to or lower than the second threshold and the reliability of the detection result is higher than the second threshold. Furthermore, when the reliability of both the tracking result and the detection result is higher than the second threshold, the update unit may perform the first update process or the second update process depending on the degree of match. Furthermore, the update unit may switch the template update method considering only one of the tracking result and the detection result. For example, if the reliability of the detection result is higher than a third threshold, the first or second update process may be performed depending on the degree of match between the tracking result and the detection result, and if the reliability is equal to or lower than the third threshold, the template may not be updated in the current frame. The same can be done when switching is performed taking into account only the reliability of the tracking result.

When multiple tracking results and detection results are obtained from the same frame, the object tracking device according to this embodiment may match (pair) the tracking results with the detection results. By pairing the tracking results with the detection results in this way, even when multiple tracking results and detection results are obtained, the template can be updated for each pair, allowing for accurate tracking.

A second aspect of the present invention is an object tracking method performed by a computer for tracking an object included in an image consisting of a plurality of frames, the object tracking method including an image acquisition step for acquiring the image, a detection step for detecting the detected object from the acquired image based on the characteristics of the detected object stored in a storage unit, and a tracking step for tracking the tracked object from the acquired image based on the characteristics of the tracked object stored in the storage unit, the object tracking method being characterized in that the detection step, the tracking step, and an update step for updating the characteristics of the tracked object are performed for a current frame, and the tracking step is performed for a next frame using the updated characteristics of the tracked object.

The present invention may be understood as an object tracking device having at least some of the above means. The present invention may also be understood as an object tracking method including at least some of the above processing. The present invention may also be understood as a program for implementing such a method, or a recording medium on which such a program is non-temporarily recorded. Note that the above means and processing can be combined with each other as much as possible to constitute the present invention.

The present invention makes it possible to perform object tracking with greater accuracy than ever before.

FIG. 1 is a diagram showing the functional configuration of an object tracking device. FIG. 2 is a diagram for explaining a template update (learning) process in the object tracking device. FIG. 3 is a diagram showing a hardware configuration of the object tracking device. FIG. 4 is a flowchart of the overall process performed by the object tracking device. FIG. 5 is a flowchart showing a case where the reliability of the detection result and the tracking result is used. FIG. 6 is a flow chart for the case where the detection score is used.

<Application Examples>
With reference to FIG. 1, an application example of the object tracking device according to the present invention will be described. The object tracking device 1 is a device that analyzes an acquired image to detect and track an object. The image may be acquired from a camera, may be received from a communication device, or may be read from a recording medium. The object may be any object, for example, a human body, a face, an animal, a vehicle, or a part. As an example, a case where a human body is tracked is assumed here. The object tracking device 1 detects, recognizes, and tracks a target human body in, for example, an office or a factory. The tracking result of the object tracking device 1 is output to an external device and is used, for example, for counting the number of people, controlling various devices such as lighting and air conditioning, and the like.

In this application example, both detection and tracking are performed for every frame. The template (features of the tracked object) used for tracking is basically learned using a sequential learning algorithm. More specifically, the template is updated sequentially using the results obtained by detection or tracking, and tracking is performed using the updated template in the next frame. If inaccurate tracking results are obtained, or if features other than the tracked object, such as the background, gradually intrude as a result of sequential learning, tracking errors may accumulate and ultimately cause tracking to fail. Therefore, in this application example, we propose a method to improve tracking performance by performing both detection and tracking for the current frame and switching the method of updating the template.

The image acquisition unit 2 has a function of acquiring image data to be tracked. The detection unit 3 searches for a human body, which is the object to be detected, from the acquired image based on the characteristics of the human body. The tracking unit 4 searches for a human body that is the same as the human body detected by the detection unit 3 up to the previous frame from the acquired image. The template used for tracking is updated by the update unit 5.

The update unit 5 updates the template using the tracking or detection results. The tracking unit 4 performs tracking using the updated template from the next frame onwards. In this way, tracking can be performed with high accuracy by performing tracking using a template that is updated based on the tracking or detection results of the current frame.

The template update method may be switched depending on, for example, the degree of match between the detection result and the tracking result. The degree of match indicates the degree of match between the human body shown by the detection unit 3 and the human body shown by the tracking unit 4. For example, it may be calculated using the degree of match between the position coordinates of the detection rectangle and the tracking rectangle, or the degree of match between the images of the detection rectangle and the tracking rectangle.

The template may be updated so that when the degree of match is higher than a first threshold, the detection result has a greater influence than when the degree of match is not higher. For example, when the degree of match is higher than the first threshold, the template may be updated using the tracking result, and when the degree of match is equal to or lower than the first threshold, the template may be updated using the detection result. Note that the switching of the update method is not limited to this method.

In this way, by switching the template update method depending on the degree of match, template updates can be performed more accurately, improving tracking accuracy.

The features used for updating the template may be switched depending on the reliability of the detection or tracking results. The reliability may be calculated for both the tracking and detection results, or for just one of them. The reliability of the tracking and detection results may be calculated, for example, based on the background image, or based on the difference in position between frames of the tracked or detected object. The reliability of the detection result may be calculated based on a detection score that indicates the likelihood of the detection result. In this way, by switching the features used for updating the template while taking into account the reliability and detection score, the template can be updated more accurately, and tracking accuracy can be improved.

<Embodiment 1>
The embodiment described below relates to an object tracking device 1. As shown in FIG. 2, a device that tracks a human body 10 from an image captured by a camera is assumed as an example here, but this is merely an example, and the object detected and tracked from the image does not have to be a human body, and may be any object. For example, animals, vehicles, and parts are examples. In addition, the source of the image is not limited to a camera, and the object tracking device may be applied to images recorded on a hard disk, for example.

The object tracking device 1 according to this embodiment can be used, for example, for human body detection using surveillance cameras or vehicle-mounted cameras, autofocus (AF), and people counting.

<Hardware Configuration>
FIG. 3 is a diagram showing a hardware configuration of the object tracking device 1 according to the present embodiment. The object tracking device 1 includes a communication IF (interface) 20, a communication device 21, a storage device 22, a calculation device 23, an input device 24, and an output device 25. In the present embodiment, the communication IF 20 receives image data directly from the camera, but may receive image data via the communication device 21 or a recording medium. The communication device 21 is a device for the object tracking device 1 to communicate with an external computer. The form of communication may be wired or wireless, and any communication standard may be used. The storage device 22 includes a main storage device and an auxiliary storage device, and stores programs executed by the calculation device 23, image data, and temporary data during program execution. The input device 24 is a device for a user to input instructions to the object tracking device 1, and is composed of, for example, a button, a keyboard, etc. The output device 25 is a device for the object tracking device to output to the user, and is composed of, for example, a display device, a control device, a speaker, etc.

Figure 1 is a functional block diagram of an object tracking device 1 according to this embodiment. The object tracking device 1 according to this embodiment has the following functional units: an image acquisition unit 2, a detection unit 3, a tracking unit 4, an update unit 5, a storage unit 6, and an output unit 7. These functional units are realized by a calculation device 23 executing a program stored in a storage device 22. However, some or all of these functional units may be realized by a dedicated device such as an ASIC.

(Image Acquisition Unit 2)
The image acquisition unit 2 has a function of acquiring image data to be tracked. The acquired image data is an image consisting of multiple frames, and may be a still image or a moving image. The acquired image may be acquired from the camera 26, the communication device 21, or the storage device 22. The captured image data is passed to the detection unit 3 and the tracking unit 4.

(Detection Unit 3)
The detection unit 3 detects the detection object from the image acquired by the image acquisition unit 2 based on the characteristics of the detection object stored in the storage unit 6. That is, the detection unit 3 searches for an area including a human body based on characteristics common to humans in general, which is the detection object. Any algorithm for detecting a human body can be used as the algorithm used for detection. The detection result is passed to the update unit 5.

(Tracking Unit 4)
The tracking unit 4 tracks the target object from the images acquired by the image acquisition unit 2, based on the features of the target object stored in the memory unit 6. That is, the tracking unit 4 searches for an area including a human body identical to the human body detected by the detection unit 3 up to the previous frame, based on features that can identify the specific human body detected by the detection unit 3, which is the target object. The template used by the tracking unit 4 for tracking is updated by the update unit 5, and tracking is performed using the updated template for the next frame. The tracking result is handed over to the update unit 5.

In this embodiment, a color histogram, which is a color feature, is used as an example of a feature used by the detection unit 3 and the tracking unit 4 to search for an object. A color histogram is suitable for tracking a person because the color distribution is constant even if the shape changes, such as a walking person. Note that the feature used is not limited to color, and features related to brightness or shape, for example, may be used, or a combination of multiple features may be used.

(Update section 5)
The update unit 5 has a function of switching an update (learning) method according to the degree of coincidence between the detection result by the detection unit 3 and the tracking result by the tracking unit 4. In this embodiment, the update unit 5 switches the features used for template update between the tracking result and the detection result according to the degree of coincidence, and changes the learning rate. The degree of coincidence is calculated using the difference in the center position between the tracking rectangle 12 indicated by the tracking unit 4 and the detection rectangle 11 indicated by the detection unit 3. Note that the degree of coincidence may be anything that can compare the similarity between the tracking result and the detection result. For example, the similarity of the results may be calculated using the center distance, the difference in the size of the area, or IoU (Intersection over Union), or may be calculated from image similarity. The image similarity may be calculated using SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), cross-correlation, or the like, or may be calculated using the similarity of the subject in the image, such as the face direction and posture.

Here, if both tracking and detection are performed correctly, the tracking result and the detection result output similar areas. On the other hand, if the tracking result and the detection result output different areas, tracking may not have been performed correctly. Therefore, the first threshold is set to a value that can be determined to be correct if the degree of match is equal to or greater than this value. By setting such a value and switching the method used for template update at the first threshold, template update can be performed with greater accuracy. Note that the first threshold may be any value, and for example, in the case of object tracking that requires strict accuracy, the first threshold may be set to a predetermined percentage higher than in other cases. The switching of update methods by the update unit 5 will be described in detail below with reference to a flowchart.

(Memory unit 6)
The storage unit 6 has a function of storing features of a detection target object (a detection template) used by the detection unit 3 and features of a tracked target object (a tracking template) used by the tracker 4. The storage unit 6 also stores various parameters, such as coefficients used for updating, used in the tracking process and the update process.

(Output unit 7)
The output unit 7 has a function of outputting information such as images and detection results/tracking results to an external device. For example, the output unit 7 may display information on a display as an external device, transmit information or control signals to a lighting device or an air conditioning device as an external device, or store the information in a recording device.

<Overall processing>
4 is a flowchart showing the entire object tracking process. The entire flow of the object tracking process will be described with reference to FIGS.

In step S101, the detection unit 3 detects a human body from the current frame image based on the characteristics of the human body, which is the object to be detected.

In step S102, the tracking unit 4 tracks the tracking target from the current frame image based on the template. The tracking target refers to the human body tracked in the previous frame image if tracking has been performed up to the previous frame image. If tracking has not been performed up to the previous frame and there is only a detection result by the detection unit 3, the tracking target refers to the human body detected by the detection unit 3.

In step S103, the update unit 5 performs matching (pairing) based on the center-to-center distance of the rectangles in the tracking results and the detection results. Note that pairing may be performed based on the center-to-center distance of the rectangles, or based on the shortest distance between the rectangles or the similarity in size of the rectangles. The processing from loop processing L1 in steps S104 to S106 onwards processes the paired tracking results and detection results, respectively.

In step S104, the update unit 5 determines the degree of match from the center coordinates of the tracking rectangle 12 indicated by the tracking unit 4 and the center coordinates of the detection rectangle 11 indicated by the detection unit 3. If the degree of match is higher than the first threshold, the process proceeds to step S105, and if it is equal to or lower than the first threshold, the process proceeds to step S106. In this embodiment, the process of step S105 corresponds to the first update process, and the process of step S106 corresponds to the second update process.

In this embodiment, the template is updated using the tracking or detection results of the current frame. An example of the formula used to update the template is given below.

F _n+1 = α×F _n + (1-α)×F _n-1
Here, F _n+1 : feature amount used for tracking in the next frame, F _n : feature amount of the object obtained from the current frame, F _n-1 : feature amount stored up to the previous frame, and α: learning rate.

In step S105, the update unit 5 uses the "feature value indicated by the tracking result of the current frame" as the "feature value (Fn) of the object obtained from the current frame" and performs template update (sequential learning) with a learning rate (α) of "0.1". The learning rate is a parameter that indicates the extent to which the template stored in the storage unit 6 is changed. That is, in S105, as shown in the above formula, the template is updated so that the feature value indicated by the tracking result of the current frame and the feature value stored up to the previous frame are combined at a ratio of 1:9. This allows tracking to be performed in the next frame using an appropriately updated template. In step S105 of this embodiment, the tracking result is used as the feature used for updating, but the detection result may be used, or both the tracking result and the detection result may be used in combination. The value of the learning rate is not limited to "0.1" and may be any value.

In step S106, the update unit 5 uses the "feature value indicated by the detection result of the current frame" as the "feature value (Fn) of the object obtained from the current frame" and sets the learning rate (α) to "1", that is, replaces (resets) the template with the feature value indicated by the detection result of the current frame. Here, if the degree of match is low, the tracking result may indicate an inaccurate result. Therefore, by immediately replacing the template with the detection result of the current frame, the tracking accuracy from the next frame can be quickly restored. The learning rate may be higher than S105, and for example, a "learning rate up" process with a learning rate of "0.5" may be adopted. In this case, the template can be updated to approach the detection result of the current frame while maintaining the information of the past template. For example, the "reset" and "learning rate up" may be switched at a predetermined threshold value smaller than the first threshold value.

In this embodiment, when the degree of agreement is higher than the first threshold (step S105), the template is updated using the tracking result by the tracking unit 4 for the current frame, and when the degree of agreement is equal to or lower than the first threshold (step S106), the template is updated using the detection result by the detection unit 3 for the current frame, and different learning rates are used in steps S105 and S106. However, as long as the template can be updated so that the detection result has a greater influence when the degree of agreement is higher than the first threshold compared to when the degree of agreement is not higher than the first threshold, the switching of the update method is not limited to this method. For example, the update unit 5 switches the features used for template update between the tracking result and the detection result depending on the degree of agreement, but the same learning rate may be used regardless of the degree of agreement. In this case, the update unit 5 may use the tracking result when the degree of agreement is higher than the first threshold, and use the detection result when the degree of agreement is equal to or lower than the first threshold, and update the template with an arbitrary learning rate (for example, "0.1"). The update unit 5 may also use the detection result for template update regardless of the degree of agreement, and change the learning rate depending on the degree of agreement. In this case, the learning rate when the degree of agreement is higher than the first threshold may be larger than the learning rate when the degree of agreement is equal to or lower than the first threshold. The update unit 5 may use the result of combining the tracking result and the detection result regardless of the degree of agreement for template update, and may change the ratio of combination according to the degree of agreement. In this case, when the degree of agreement is higher than the first threshold, the update unit 5 may combine the tracking result so that the weight is larger than when the degree of agreement is equal to or lower than the first threshold. Furthermore, in this case, the learning rate may be changed according to the degree of agreement, or the same learning rate may be used. The template may be updated by combining the above-mentioned multiple update methods. In this way, when the degree of agreement is equal to or lower than the first threshold, it is preferable to update the template so that the detection result has a larger influence than when the degree of agreement is not equal to the first threshold. This makes it possible to avoid a decrease in tracking accuracy due to inaccurate template update even when the degree of agreement is low and there is a possibility that either the tracking result or the detection result is inaccurate.

The tracking unit 4 performs tracking using the updated template in the next frame, and the process from step S101 to step S105 or step S106 is repeated for each frame.

In this embodiment, both detection by the detection unit and tracking by the tracking unit are performed on the current frame image, and the template update method is switched depending on the degree of match between the detection result and the tracking result. If the detection result and the tracking result match highly, the template can be updated correctly using either result. On the other hand, if the detection result and the tracking result differ, one of the results may be inaccurate. Although it is rare, the tracking process may capture an object that is significantly different from the target, and the detection process rarely captures an object that is significantly different from the target. Therefore, as described above, the detection result is prioritized when the detection result and the tracking result do not match. This reduces the occurrence of the template being stuck to the background when updating the template using an inaccurate result, and improves tracking accuracy.

Second Embodiment
In the first embodiment, the update method is switched based on the degree of coincidence, but in the second embodiment, the update method is switched by further considering the reliability of the tracking result and the detection result. Since the configuration of the object tracking device according to this embodiment is basically the same as that of the first embodiment, a detailed description will be omitted and differences will be mainly described.

Here, reliability is an index that indicates the likelihood of a tracking result or a detection result. In this embodiment, reliability is calculated for both the tracking result and the detection result, but it may be calculated for either the tracking result or the detection result. In this way, by switching the template update method taking into account reliability, it is possible to more appropriately update the template and improve tracking accuracy.

The reliability may be calculated using, for example, the difference from the background image. That is, the reliability of the tracking result by the tracking unit 4 may be determined according to the difference between the tracking result for the current frame and the background image, and the reliability of the detection result by the detection unit 3 may be determined according to the difference between the detection result for the current frame and the background image. Note that the background image is an image when there is no detection target in the detection area. In this embodiment, an image at a specific time when there is no detection target is used as the background image for the background difference. Note that the background image may be an image at a specific time, or different images may be used as the background image successively. For example, the background image may be the frame image immediately before there is no detection target, or an image synthesized using the average of the most recent specified frames may be set as the background image. Also, the background difference of the tracking result and the background difference of the detection result may use the same background image, or different background images.

The reliability based on the difference with the background image is evaluated based on the image dissimilarity between the area output by the tracking or detection result for the current frame and the corresponding area of the background image. Here, if the tracking or detection result is similar to the background image, there is a possibility that the background has been mistakenly recognized as the target object. Therefore, the dissimilarity between the tracking or detection result and the background image can be used as the reliability, and the higher the dissimilarity (the less similar), the higher the reliability is evaluated to be. Note that the amount of image difference can be used as the dissimilarity, but other indices may also be used.

With reference to FIG. 5, a method of switching the template update method using the difference from the background image as an example of reliability will be described. Note that the same processes as in FIG. 4 are given the same reference numerals, and detailed explanations will be omitted. The flow of the process from step S201 onwards will be described with reference to FIG. 1 and FIG. 5.

In step S201, the reliability of the detection result by the detection unit 3 and the reliability of the tracking result by the tracking unit 4 are compared with a second threshold value. Here, the second threshold value is set to a value that, if equal to or greater than this value, the detection result or tracking result can be determined to be reliable. In this embodiment, the same value is used as the second threshold value for the detection result and the tracking result, but different values may also be used.

If it is determined in step S201 that the reliability of both the detection result and the tracking result is higher than the second threshold, the process proceeds to step S104. In other words, if both the detection result and the tracking result are reliable, the template update method is switched taking into account the degree of match, as in the first embodiment.

In the judgment of step S201, if the reliability of the detection result is equal to or less than the second threshold value and the reliability of the tracking result is higher than the second threshold value, the detection result may be inaccurate, but the tracking result is considered to be correct. Therefore, the process proceeds to step S105 without judging the degree of match, and the update unit 5 uses the tracking result to perform template update (sequential learning). This makes it possible to update the template using a more reliable result.

In the judgment of step S201, if the reliability of the detection result is higher than the second threshold and the reliability of the tracking result is equal to or lower than the second threshold, it is considered that the detection result is correct, but the tracking result may be inaccurate. Therefore, the process proceeds to step S106 without judging the degree of match, and the update unit 5 replaces (resets) the template with the detection result of the current frame. As a result, if the tracking result is inaccurate, the template is corrected to approach the detection result, and the tracking accuracy can be improved.

If it is determined in step S201 that the reliability of both the detection result and the tracking result is equal to or less than the second threshold, the process proceeds to step S107, in which the update unit 5 does not update the template ("no learning"). Note that "not updating the template" can be considered to be a learning process with a learning rate of 0%. This makes it possible to avoid updating the template using features that may be inaccurate.

The tracking unit 4 uses the updated template in the next frame to perform tracking, and the processes from step S101 onwards are repeated for each frame. In this way, by switching the template update method depending on the reliability, the template can be updated to more appropriately reflect the results of the current frame, and tracking accuracy can be improved.

When the result of the reliability determination is used, all of the processes from S104 to S107 may be adopted after the determination in step S201, or only one of the processes may be adopted. For example, if step S107 ("no learning") is not adopted and both the reliability of the detection result and the reliability of the tracking result are equal to or lower than the second threshold, the process may proceed to the matching process in step S104 and the template may be updated according to the matching. For example, if the reliability of the tracking result is high without adopting the matching determination in step S104, the template may be updated by the "sequential learning" process. In addition, a switching method different from the above may be adopted as the template update method. For example, if either the reliability of the detection result or the reliability of the tracking result is lower than the second threshold, the process of not updating the template (step S107) may be adopted. As in the first embodiment, the learning rate used in the second embodiment is merely an example, and any value may be used.

The reliability is not limited to the dissimilarity with the background image, and may be calculated based on other indices. For example, the difference in position between frames may be used as an index of reliability. In this case, if the difference in position between frames is within the range in which the target object can move, it may be assumed that the result is reliable. The reliability of the tracking result by the tracking unit 4 may be determined according to the difference in tracking position between frames, and the reliability of the detection result by the detection unit 3 may be determined according to the difference in detection position between frames. The tracking position and detection position are, for example, the coordinates of the center position of the tracked and detected area. In this way, by calculating the reliability according to the difference in position between frames, it is possible to reduce the possibility of updating the template using a result that may have been erroneously tracked or detected.

The criterion used for the reliability is not limited to one condition. For example, the reliability may be calculated by combining the difference from the background image, the position difference between frames, and other criteria.

In the second embodiment, the template update method is switched taking into account the reliability of the detection and tracking results. According to this embodiment, the template can be updated using reliable results, which allows for more appropriate updates and further improves tracking accuracy.

<Modification>
In the second embodiment, the template update method is switched based on the reliability of the detection result and the tracking result, but the template update method may be switched based only on the reliability of the detection result. The configuration of the object tracking device according to this modification is basically the same as that of the second embodiment, so a detailed description will be omitted and differences will be mainly described.

Figure 6 is a flowchart when the reliability is calculated only from the detection result. Here, a detection score indicating the likelihood of the detection result may be used as the reliability of the detection result. Note that, when switching the template update method based only on the reliability of the detection result, it is not necessarily necessary to use the detection score. For example, as described in the second embodiment, the difference from the background image or the position difference between frames may be used as the reliability of the detection result, or a reliability calculated by another method may be used as the reliability of the detection result.

The detection result score is an index that indicates the likelihood that the detection area contains the object to be detected, and is output by the detection unit along with the detection area. The higher the likelihood that the detection area contains a human body, the higher the detection score that is calculated. In this way, by calculating only the reliability of the detection result as the reliability and switching the method of updating the template depending on the reliability of the detection result, it is possible to improve tracking accuracy with simpler processing.

With reference to FIG. 6, a method for switching the template update method using the detection score will be described. Note that the process up to step S103 is the same as that in FIG. 4. The process flow from step S301 onwards will be described with reference to FIG. 1 and FIG. 6.

In step S301, the detection score of the detection result is compared with a third threshold value. Here, the third threshold value is set to a value at which, if the detection score is equal to or greater than this value, it can be determined that the detection result contains the object to be detected and that the detection result is reliable.

If it is determined in step S301 that the detection score is higher than the third threshold, the process proceeds to step S104. In other words, if the detection result is reliable, the template update method is switched taking into account the degree of match, as in the first embodiment.

If it is determined in step S301 that the detection score is equal to or less than the third threshold, the process proceeds to step S107, where the update unit 5 does not update the template ("no learning"). Note that "not updating the template" can be considered as a learning process with a learning rate of 0%. . If the detection score is equal to or less than the third threshold and the detection result is likely to be inaccurate, it is unclear whether the tracking result shows a correct result. Therefore, the template is not updated, and the current template is maintained. This makes it possible to avoid updating the template using features that may be inaccurate.

The tracking unit 4 performs tracking using the updated template in the next frame, and the processes from step S101 onwards are repeated for each frame. In this way, by switching the template update method depending on the detection score, the template can be updated to appropriately reflect the results of the current frame, and tracking accuracy can be improved.

In the modified example, the reliability of the detection result is used as an example of switching the template update method using the reliability of either the tracking result or the detection result, but the template update method may be switched using only the reliability of the tracking result.

＜Other＞
The above-described embodiment merely describes an exemplary configuration of the present invention. The present invention is not limited to the above-described specific embodiment, and various modifications are possible within the scope of the technical concept thereof.

In the above description, a color histogram is used as a feature amount representing a color feature as an example of a feature used by the detection unit 3 and the tracking unit 4 to search for an object, but this is not limiting. For example, a luminance histogram may be used as a feature amount representing a luminance feature, and a Histogram of Gradient (HOG) feature amount, Scale-Invariant Feature Transform (SIFT) feature amount, Speeded Up Robust Features (SURF) feature amount, etc. may be adopted as a feature amount representing the shape and contour of an object.

Any algorithm may be used for the detection method. For example, a classifier that combines image features such as HOG or Haar-like with boosting may be used, or an SVM may be used.
A pattern recognition model such as a Support Vector Machine (SVM) may be used, or image recognition using deep learning (e.g., R-CNN, Fast R-CNN, YOLO, etc.) may be used.

In the above embodiment, the "sequential learning" process (S105, S103, S303) shows an example of updating the template using the tracking result, but the detection result may be used, or the template may be updated using a result of combining the tracking result and the detection result. In addition, in FIG. 5 and FIG. 6, which show how to switch the template update method using the reliability, an example of adopting the "reset" process is shown in the process when the degree of match is low (S204, S304), but a "learning rate increase" process with a higher learning rate than the "sequential learning" process may be adopted. For example, the "reset" and "learning rate increase" processes may be switched depending on the value of the degree of match or the number of consecutive frames with a degree of match smaller than the first threshold. The value of the learning rate is not limited to 10%, 50%, or 100%, and any value may be used.

In the modified example, when the detection score is low, neither the detection result nor the tracking result is used for updating, and the update unit does not update the template (S305), but the template may be updated using the tracking result. For example, when the detection score is low but the tracking result is stable, the tracking accuracy can be improved by updating the template using the tracking result.

In the above embodiment, a rectangular detection frame is exemplified (see FIG. 2, etc.), but a detection frame of a shape other than a rectangle (polygon, ellipse, free-form shape, etc.) may also be used.

<Additional Notes>
(1) An object tracking device (1) for tracking an object included in an image consisting of a plurality of frames, comprising:
An image acquisition unit (2) for acquiring the image;
A memory unit (6) that stores features of a detection target object and features of a tracking target object;
an update unit (5) for updating the characteristics of the tracked object in the memory unit (6);
a detection unit (3) that detects the detection object from an acquired image based on the characteristics of the detection object stored in the memory unit (6);
a tracking unit (4) that tracks the tracking object from an acquired image based on the features of the tracking object stored in the memory unit (6);
Equipped with
The object tracking device (1) performs both detection of the detection object by the detection unit (3) and tracking of the tracking object by the tracking unit (4) for the current frame,
The update unit (5) updates the features of the tracked object based on the tracking result and the detection result,
The tracking unit (4) performs the tracking using the updated features of the tracked object for the next frame.
An object tracking device (2) characterized in that an object tracking method for tracking an object included in an image consisting of a plurality of frames, which is performed by a computer, includes an image acquisition step for acquiring the image, and a detection step for detecting the detection object from the acquired image based on the characteristics of the detection object stored in a storage unit (
the object tracking method includes a detection step (S101), a tracking step (S102) of tracking the tracked object based on features of the tracked object stored in a storage unit, wherein the object tracking method performs the detection step (S101), the tracking step (S102), and an updating step (S105, S106) of updating the features of the tracked object for a current frame, and performs the tracking step (S102) for a next frame using the updated features of the tracked object.

1: Object tracking device 2: Image acquisition unit 3: Detection unit 4: Tracking unit 5: Update unit 6: Tracking unit 7: Output unit

Claims (17)

An object tracking device that tracks an object included in an image consisting of a plurality of frames,
An image acquisition unit that acquires the image;
A storage unit that stores features of a detection target and features of a tracking target;
an update unit that updates the characteristics of the tracked object in the storage unit;
a detection unit that detects the detection object from an acquired image based on the characteristics of the detection object stored in the storage unit;
a tracking unit that tracks the tracking object from an acquired image based on the features of the tracking object stored in the storage unit;
Equipped with
the object tracking device performs both detection of the detection object by the detection unit and tracking of the tracking object by the tracking unit for the current frame;
the update unit switches an update method according to a degree of match between the detection result by the detection unit and the tracking result by the tracking unit, and updates the feature of the tracked object based on the tracking result and the detection result by the switched update method ;
The tracking unit performs the tracking using the updated features of the tracked object for the next frame.
An object tracking device comprising: When the degree of agreement is higher than a first threshold, the update unit performs a first update process to update features of the tracked object using at least one of the tracking result by the tracker and the detection result by the detector for the current frame, and when the degree of agreement is equal to or lower than the first threshold, performs a second update process to update features of the tracked object using the detection result by the detector for the current frame so that the detection result by the detector has a greater influence than when the degree of agreement is not higher than the first threshold.
The object tracking device according to claim 1 . the tracking result by the tracking unit and the detection result by the detection unit include information regarding a central position or a size of the tracked object and the detected object, and the degree of match is calculated based on a difference in central position or a difference in size between the tracking result and the detection result.
The object tracking device according to claim 2 . When the degree of agreement is higher than the first threshold, the update unit updates the feature of the tracked object using a tracking result by the tracker for the current frame, and when the degree of agreement is equal to or lower than the first threshold, the update unit updates the feature of the tracked object using a detection result by the detector for the current frame.
The object tracking device according to any one of claims 2 to 3 . When the degree of match is equal to or less than the first threshold, the update unit replaces the detection result by the detection unit for the current frame as a feature of the tracked object.
The object tracking device according to any one of claims 2 to 4 . the update unit determines a reliability of at least one of the tracking result by the tracking unit and the detection result by the detection unit, and switches a method of updating the feature of the tracked object in accordance with the reliability.
The object tracking device according to any one of claims 2 to 5 . the update unit does not update the feature of the tracked object in the current frame when the reliability of both the tracking result by the tracker and the detection result by the detector is equal to or less than a second threshold.
The object tracking device according to claim 6 . the update unit performs the first update process when the reliability of the tracking result by the tracking unit is higher than a second threshold and the reliability of the detection result by the detection unit is equal to or lower than the second threshold.
The object tracking device according to claim 6 or 7 . the update unit performs the second update process when the reliability of the tracking result by the tracking unit is equal to or less than a second threshold and the reliability of the detection result by the detection unit is higher than the second threshold.
The object tracking device according to any one of claims 6 to 8 . The object tracking device according to any one of claims 6 to 9, wherein when the reliability of both the tracking result by the tracking unit and the detection result by the detection unit is higher than a second threshold, the update unit performs the first update process or the second update process depending on the degree of match. 11. The object tracking device according to claim 6, wherein the update unit performs the first update process or the second update process depending on the degree of match when the reliability of the detection result by the detection unit is higher than a third threshold, and does not update the features of the tracked object in the current frame when the reliability is equal to or lower than the third threshold . a reliability of the tracking result by the tracking unit is determined according to a difference between the tracking result for the current frame and a background image;
The reliability of the detection result by the detection unit is determined according to a difference between the detection result for the current frame and a background image.
An object tracking device according to any one of claims 6 to 11 . the reliability of the tracking result by the tracking unit is determined according to a difference in tracking position between frames;
The reliability of the detection result by the detection unit is determined according to a difference in detection position between frames.
An object tracking device according to any one of claims 6 to 11 . The reliability of the detection result by the detection unit is determined according to a score indicating the likelihood of the detection result output by the detection unit.
An object tracking device according to any one of claims 6 to 11 . 15. The object tracking device according to claim 1, wherein, when a plurality of tracking results and detection results are obtained from the same frame, the tracking results and the detection results are matched with each other, and the characteristics of the tracked object are updated using the corresponding tracking results and detection results. 1. An object tracking method for tracking an object included in an image consisting of a plurality of frames, the method comprising:
an image acquisition step of acquiring the image;
A detection step of detecting a detection target from an acquired image based on features of the detection target stored in a storage unit;
a tracking step of tracking the tracking target from the acquired image based on the characteristics of the tracking target stored in the storage unit;
an updating step of switching an update method according to a degree of coincidence between a detection result of the detection object by the detection step and a tracking result of the tracked object by the tracking step for a current frame, and updating the feature of the tracked object stored in the storage unit based on the tracking result and the detection result by the switched update method ,
In the tracking step for the next frame, the tracked object is tracked using the updated features of the tracked object.
13. An object tracking method comprising: A program for causing a computer to execute each step of the method according to claim 16 .

Images