JP6789876B2 - Devices, programs and methods for tracking objects using pixel change processed images - Google Patents

Description

The present invention relates to a target tracking technique for tracking a time-series image group that may include a tracking target.

Currently, tracking technology that analyzes time-series image data captured and generated by a camera and sequentially determines the position of a moving tracking target in real space for the purpose of monitoring, marketing, or the like has been put into practical use. As a tracking target, various objects such as a person and a vehicle can be set as long as they can be photographed. In addition, a technology for tracking a plurality of objects in parallel has been developed and is being actively improved. In this technique, for example, it is intended to more accurately grasp the flow lines of a large number of clerk, customers, etc. who stay or move in the store.

The object tracking technique using such a time-series image group is generally based on (a) a classifier that outputs information related to the position of the object while learning the appearance of the object to be tracked online from the momentary image. And (b) by a classifier that does not perform apparent online learning.

For example, Non-Patent Document 1 discloses the online learning type tracking technique of (a). In this technique, every time a new time frame is acquired, detection is performed and tracking processing is performed for each ID (identifier) assigned to the object to be tracked. Specifically, the appearance of the object related to each ID is learned online, and the appearance identification model is constructed for each ID. Next, using the apparent discriminative model for each ID learned online in this way, the new detection result is linked with the trajectory of the object related to the ID tracked so far.

On the other hand, Non-Patent Document 2 discloses a technique for tracking a single object, which is a type (b) in which online learning is not performed. In this technique, deep learning is used, and an image area pair of an object image area at the previous time and a candidate image area cut out as a candidate from the periphery of the area corresponding to the area at the previous time in the image at the current time. Therefore, tracking is performed by estimating the image area where the object exists at the current time.

S.-H. Bae and K.-J. Yoon., "Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning", Published in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference, 2014 , 1218-1225 D. Held, S. Thrun, and S. Savarese, "Learning to track at 100 fps with deep regression networks", Cornell University Library, Subjects: Computer Vision and Pattern Recognition (cs.CV), Cite as: arXiv: 1604.01802, 2016

However, in the conventional technique as described above, there still arises a problem that cannot be solved, especially when tracking a plurality of objects in parallel.

For example, the technique described in Non-Patent Document 1 learns the appearance of an object online, so that when a plurality of objects exist in the image data, a large amount of memory is consumed and the consumption amount is controlled. Has the problem that it is difficult. For example, if this technology is applied in a situation where a large number of people exist and the people included in the video change from moment to moment, such as in an exhibition hall or a public road, the memory consumption becomes enormous, and finally the tracking process is performed. There is a possibility that it will not be possible. Specifically, as the ID of the person to be tracked increases while the analysis for tracking is performed for a long time, the person model to be learned continues to increase, resulting in a situation where memory resources are insufficient. It will end up.

On the other hand, in the technique described in Non-Patent Document 2, since it is not necessary to learn the appearance of the object online, the problem of memory consumption as described above does not occur. However, Non-Patent Document 2 sets a single object as a tracking target to the last, and does not assume tracking of a plurality of objects. Here, if the technique described in Non-Patent Document 2 is applied to the tracking of a plurality of objects as it is, the problem of memory consumption does not occur, but an object having a similar appearance exists near the object to be tracked. When doing so, the possibility of a phenomenon called drift increases.

Here, drift is a phenomenon in which an image region related to an object having a similar appearance in the vicinity is regarded as a correct region, and an erroneous object is started to be tracked. For example, it is a phenomenon that may occur frequently when a plurality of people overlapping each other in a video are targeted for tracking. However, Non-Patent Document 2 does not propose any method for dealing with this drift because a single object is targeted for tracking in the first place.

By the way, when tracking a plurality of objects, this drift is not completely eliminated even in the object tracking using online learning, and the current situation is that countermeasures are still desired.

Therefore, an object of the present invention is to provide a device, a program and a method capable of more reliably tracking a plurality of objects.

According to the present invention, based on the output from the image or the identification device receives the image area in the image included in the image group of time series may include a plurality of tracked, each of the plurality of tracking target in the image A device that tracks the tracking target using a target positioning means that determines the position of the
An image included in the image group or a processing target area in an image area within this image, and for other tracking targets other than one tracking target, in an image related to a time point earlier than this image or at this past time point. A mask that performs pixel change processing that changes the processing target area determined based on the position determined in the image area in the image to a pixel pattern that eliminates or reduces the characteristics of the other tracking target. Processing means and
The image or the image area that has undergone the pixel change processing at one time point, and the image or the image area that has undergone the pixel change process at one or more time points prior to the one time point. and possess an image output means for outputting to the learned classifier,
The target position determining means is from a trained classifier that inputs the image or the image area that has undergone the pixel change processing at the one time point and a plurality of time points that precede it. Determines the position of the one tracked object at that one point in time based on the output of
Object-tracking device is provided, characterized in that.
Here, in this target tracking device, the image output means also inputs information related to a plurality of time points to the trained classifier, and the target position determining means is subjected to the pixel change processing at the plurality of time points. To determine the position of the one tracking target at the one time point based on the output from the trained classifier which is input by combining the image or the image area and the information related to a plurality of time points. Is also preferable.

As an embodiment of the pixel change processing in the target tracking device of the present invention, the mask processing means has a processing target area related to at least one other tracking target other than the image or the tracking target in the image area. It is also preferable to change the pixel pattern determined based on the image region or a predetermined pixel pattern.

Further, as another embodiment of the pixel change processing, it is also preferable that the mask processing means performs a process of blurring the appearance of the other tracking target on the processing target region related to at least one other tracking target.

Further, in the mask processing means, as a further embodiment of the pixel change processing, the processing target area related to the other tracking target is superimposed on the tracking target area including the position determined for the one tracking target. When the superimposed region is provided, it is also preferable not to perform the pixel change processing on the superimposed region.

Further, in the target tracking device according to the present invention, when the processing target area related to the other tracking target has a superimposing area superimposed on the tracking target area including the position determined for the one tracking target, the one It is also preferable to further have a tracking target area determination unit that changes the tracking target area related to the tracking target to a region excluding the superposed area.

Further, the identifier of the present invention, a plurality of first neural networks, respectively and inputs the image or the image region has undergone the pixel changing process in the plurality of time points, respectively the image or the image a plurality of first neural networks generating feature information according to the characteristics of the regions, as an input a plurality of the feature information output from the first neural network of the plurality of, and information related to a plurality of time points, the 1 It is also preferable to include a second neural network that outputs information related to the position of the one tracking target at one time point .
Furthermore, the trained classifier according to the present invention includes the image or the image area that has undergone the pixel change processing at a certain time point, and the pixel change at one or more time points prior to this time point. It is also preferable that the training is performed by the training data set including the processed image or the image area and the information related to the position of the tracking target as the correct answer data.

Further, as a further embodiment of the pixel change processing in the target tracking device of the present invention, the mask processing means is displaced from the determined processing target area when the classifier according to the present invention is trained. It is also preferable to perform pixel change processing on the processing target area.

Further, as another embodiment of the classifier, the image output means outputs the image or the image area at at least one time point after the pixel change processing is performed to the classifier for learning online. object position determination means, based on the output from the discriminator which performed learning online, the a is also preferable one Turkey to determine the position of the tracking target in the single point in time.

According to the present invention, also based on the output from the image or the identification device receives the image area in the image included in the image group of time series may include a plurality of tracked, multiple tracked in the image It is a program for operating a computer mounted on a device for tracking the tracking target by using a target positioning means for determining each position of the tracking target.
An image included in the image group or a processing target area in an image area within this image, and for other tracking targets other than one tracking target, in an image related to a time point earlier than this image or at this past time point. A mask that performs pixel change processing that changes the processing target area determined based on the position determined in the image area in the image to a pixel pattern that eliminates or reduces the characteristics of the other tracking target. Processing means and
The image or the image area that has undergone the pixel change processing at one time point, and the image or the image area that has undergone the pixel change process at one or more time points prior to the one time point. To function as an image output means to output to a trained classifier ,
The target position determining means is from a trained classifier that inputs the image or the image area that has undergone the pixel change processing at a plurality of time points that are one time point and one or more time points preceding the time point. Determines the position of the one tracked object at that one point in time based on the output of
A target tracking program characterized by this is provided.

According to the present invention, furthermore, based on the output from the image or the identification device receives the image area in the image included in the image group of time series may include a plurality of tracked, multiple tracked in the image It is a target tracking method in a computer mounted on a device for tracking the tracking target by using a target positioning means for determining each position of the tracking target.
An image included in the image group or a processing target area in an image area within this image, and for other tracking targets other than one tracking target, in an image related to a time point earlier than this image or at this past time point. A step of performing pixel change processing for changing a processing target area determined based on a position determined in an image area in the image to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced. When,
The image or the image area that has undergone the pixel change processing at one time point, and the image or the image area that has undergone the pixel change process at one or more time points prior to the one time point. and a step of outputting to the learned classifier,
A step carried out by the target position determining means, in which the image or the image area subjected to the pixel change processing at a plurality of time points, which is one time point and one or more time points preceding the time point, is input. A target tracking method is provided that has a step of determining the position of the one tracked object at that one time point based on the output from the trained classifier .

According to the object tracking device, program and method of the present invention, a plurality of objects can be tracked more reliably.

It is a schematic diagram which shows one Embodiment of the object tracking system which includes the object tracking apparatus by this invention. It is a functional block diagram which shows the functional structure in one Embodiment of the object tracking apparatus by this invention. It is a schematic diagram which shows one Embodiment of the pixel change processing (mask processing) which concerns on this invention. It is a schematic diagram which shows the other embodiment of the pixel change processing (mask processing) which concerns on this invention. It is a schematic diagram which shows the further other embodiment of the pixel change processing (mask processing) which concerns on this invention. It is a schematic diagram which shows one Embodiment of the classifier which concerns on this invention. It is a schematic diagram which shows the other embodiment of the classifier which concerns on this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Object tracking system]
FIG. 1 is a schematic diagram showing an embodiment of a target tracking system including a target tracking device according to the present invention.

The target tracking system of the present embodiment shown in FIG. 1 is
(A) One or a plurality of cameras 2 capable of photographing an object to be tracked and transmitting information of the photographed image in chronological order via a communication network.
(B) A target tracking device 1 capable of predicting the position information of the object and tracking the object by using a time-series image group acquired from the camera 2 via a communication network is provided.

Here, the object to be tracked corresponds to various objects such as a person, an animal, a vehicle, and other movable physical objects, if photographable. Moreover, the place where the photograph is taken is not particularly limited. For example, it may be outdoors where spectators, commuters, shoppers, workers, pedestrians, runners, etc. can be tracked, or even indoors such as in a company, school, home, or inside a store. it can.

By the way, in the present embodiment, it is assumed that there are a plurality or a large number of objects (persons) to be tracked, and an environment in which the prior art is not good at accompanying, passing each other, or superimposing in an image is assumed. ..

Further, the communication network which is a transmission path of image information can be, for example, a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark). Alternatively, the camera 2 and the target tracking device 1 are communicated and connected via the Internet via a wireless access network such as LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access) or 3G (3rd Generation). It may be.

Further, the camera 2 and the target tracking device 1 may be communicated and connected via an optical fiber network or a fixed access network such as ADSL (Asymmetric Digital Subscriber Line) via the Internet or via a private network. Further, as a modification, the camera 2 and the target tracking device 1 may be directly connected by wire. Further, a camera control device (not shown) capable of collecting image information output from the plurality of cameras 2 and transmitting the image information to the target tracking device 1 may be provided.

Similarly, as shown in FIG. 1, the target tracking device 1 in each image is based on the output from the classifier 11 that has input the acquired image of the time series image group or the image area (image patch) in this image. A device that determines the position of a tracking target and tracks the tracking target.
(A) To a pixel pattern in which the features of other tracking targets other than one tracking target are eliminated or reduced with respect to the image included in the time series image group or the "processing target area" in the image area in this image. A mask processing unit 111 that performs "pixel change processing" for changing, and
(B) It is characterized by having an image output unit 113 that outputs the image or image area subjected to this "pixel change processing" to the classifier 11 for learning and / or identification of this one tracking target. There is.

Here, the "processed area" of the above (A) is determined based on a position determined at a past time or as a correct answer for other tracking targets other than this one tracking target. Further, as a specific example of the "pixel change processing" of the above (A), as a specific example,
(A1) The "processing target area" may be changed to an area having a predetermined pixel pattern. For example, it can be filled with a single color such as black. Also,
(A2) The “processing target area” may be changed to a region having a pixel pattern determined based on an image area other than the tracking target. For example, a background color to be tracked can be determined and filled with this background color. further,
(A3) The "processed area" may be subjected to a process of blurring the appearance of another tracked object. For example, the color calculated from the surrounding colors can be used for blurring.

As a matter of course, the "pixel change process" is not limited to the above-mentioned forms (A1) to (A3), but changes to a pixel pattern in which the features of other tracking targets are eliminated or reduced. If so, various processes can be adopted.

In this way, when the target tracking device 1 tracks a plurality of objects as tracking targets, the target tracking device 1 performs "pixel change processing" on an object other than one object to be tracked, thereby performing the classifier 11 The features of the image or other object in the image area (image patch) to be input to are eliminated or reduced. As a result, by using the image or image area subjected to such processing to train the classifier 11 and / or to cause the classifier 11 to perform the discrimination process, this one object can be changed to another object. It is possible to continue to identify more reliably without being confused with.

In other words, by using the "image object masking method" of the present invention as described above, even if an object having a similar appearance exists near the object to be tracked, the drift that has been a problem in the past has been a problem. It is possible to suppress the occurrence. Here, the drift is a phenomenon in which an image region related to this similar object existing nearby is regarded as a correct region and a wrong object is started to be tracked.

By the way, the "image object mask method" of the present invention provides online learning, which has been the mainstream in the past, even for tracking processing in which online learning is not performed in order to solve the problem of increased memory consumption (calculation cost). It can also be applied to the tracking process to be performed. Of these, when applied to tracking processing of a plurality of objects that do not perform online learning, it is possible to suppress an increase in memory consumption (calculation cost) and suppress the occurrence of drift at the same time. On the other hand, when applied to the tracking process of online learning, the drift can be further eliminated.

In any case, the "image object mask method" of the present invention is a technique for suppressing false tracking and making tracking of a plurality of objects more reliable, but especially under the non-online learning which is the former case described above. It is a very important technology to realize the multi-tracking of.

The time-series image group handled by the device 1 is not limited to the image data generated by the camera shooting as in the present embodiment. Various data are applicable as long as the data is related to the actual position and appearance of the tracked object. For example, it is also possible to use the depth value information (of each target pixel) generated by the depth camera as image data.

Further, in the present embodiment, the target tracking device 1 sets the position coordinates (u, v) in the image coordinate system uv stretched in the image acquired every moment to the world coordinate system Gx- stretched in the real space. Using the coordinate conversion operation that converts to the position coordinates (gx, gy, gz) in Gy-Gz, the information related to the position in the real space is calculated from the image information of the tracked object. For example, when the position (u, v) at the previous time t-1 in the image of the object to be tracked changes to the position (u', v') at the current time t, this object is in real space (observation). In the target space), it is estimated that the position moved from the position (gx, gy, gz) at the previous time t−1 to the position (gx', gy', gz') at the current time t, and the position in the real space The change from the previous time t-1 can be acquired.

Here, the time to be used is a time set for each unit time elapsed with the unit time as 1, and the time immediately before the time t is t-1. Further, the coordinate conversion from the image coordinate system to the world coordinate system as described above can be determined by setting external parameters related to the installation position and shooting orientation of each camera 2 in advance by calibration. is there. Even when images are acquired from each of the plurality of cameras 2, it is possible to integrate these images to construct one image space and apply an image coordinate system to this image space.

As described above, in the present embodiment, the target tracking device 1 is based on the image information (position information in the image coordinate system uv) acquired every moment, and the position information (world coordinate system Gx) of the tracked object in the real space. -Position information in Gy-Gz) can be estimated.

[Device configuration, target tracking method]
FIG. 2 is a functional block diagram showing a functional configuration according to an embodiment of the target tracking device according to the present invention.

According to FIG. 2, the target tracking device 1 includes a communication interface 101 capable of communicating with one or more cameras 2, an image storage unit 102, a mask processing image storage unit 103, and an identification model storage unit 104. It has a target information storage unit 105 and a processor memory. Here, the processor memory realizes the target tracking function by executing a program that causes the computer of the target tracking device 1 to function.

Further, the processor memory has a mask processing unit 111, a tracking area determination unit 112, a classifier 11, an image output unit 113 including a learning unit 113a and an identification unit 113b, and a target position determination unit 114 as functional components. And a tracking target management unit 115, and a communication control unit 121. The processing flow shown by connecting the functional components of the target tracking device 1 in FIG. 2 with arrows is also understood as an embodiment of the target tracking method according to the present invention.

Similarly, in FIG. 2, the camera 2 is a photographing device compatible with visible light, near infrared rays, or infrared rays, which is provided with a solid-state image sensor such as a CCD image sensor or a CMOS image sensor. As described above, it is also possible to use a depth camera as the camera 2. Further, the camera 2 or a camera control device (not shown) generates captured image data including an image of an object captured by the camera 2, and transmits the data to the target tracking device 1 in chronological order or in batch. Has a function. Further, it is also preferable that the camera 2 is movable and can change the installation position, the shooting direction and the height, and has a function of receiving and processing a control signal for this change.

The communication interface 101 receives captured image data, which is a time-series image group, from the camera 2 or the camera control device via the communication network. Transmission / reception and communication data processing are controlled by the communication control unit 121 using the communication interface 101, and the acquired captured image data (image file) is stored in the image storage unit 102. Here, the captured image data may be one that is called from the camera 2 or the camera control device in chronological order and acquired, and is one that sequentially acquires images captured at regular time intervals in real time. May be good.

The mask processing unit 111 refers to an image included in the time-series image group read from the image storage unit 102 or an object other than one object to be tracked with respect to the processing target area in the image area in the image. Performs mask processing (pixel change processing) for changing to a pixel pattern in which the characteristics of the above are eliminated or reduced.

Here, the processing target area to be masked is determined based on the position determined for these other objects (a) at a past time point or (b) as a correct answer. For example, the processing target area at time t may be the correct image area of the other object at time t-1. Incidentally, the mask processing unit 111 can acquire information on the tracking target area determined by the image (or image area) at the past time from the target position determination unit 114, which will be described later.

In any case, the mask processing of the present embodiment is a process for preventing the appearance of another object from being reflected at all or clearly in the image or image area (image patch) input to the classifier 11. ing. It is also preferable that the mask-processed image or image area processed by the mask processing unit 111 is stored in the mask-processed image storage unit 103 and appropriately read out and used.

Here, as the mask processing performed by the mask processing unit 111, the following three, which have already been briefly described, that is, (A1) a process of changing the processing target area to an area having a predetermined pixel pattern,
(A2) A process of changing the processing target area to an area having a pixel pattern determined based on the background image area of the tracking target, and (A3) a process of blurring the appearance of another tracking target with respect to the processing target area. Can be mentioned.

Of these, as a specific example of the mask processing of (A1) above, the processing target area may be painted with a single color such as black. By the way, when the classifier 11 is trained using the masked image or image area subjected to such masking (as teacher data), the portion of the predetermined pixel pattern (such as a monochromatic pattern such as black) is tracked. It can also be regarded as actively performing negative learning that it is not in the area of an object. In other words, it can be said that the learning is performed that the region of the tracking target object exists in the image region excluding the portion of the predetermined pixel pattern.

Further, as a specific example of the mask processing of the above (A2), a background model in an image may be learned by using a known background modeling method, and the processing target area may be filled with the learned background color. Here, as the background model, for example, the color of each pixel may be the average color at a plurality of times, or the distribution of the background color in each pixel is modeled by a mixed Gaussian distribution. It can also be used.

The background modeling method using this mixture Gaussian distribution is, for example, non-patent literature: P. KadewTraKuPong and R. Bowden, "An improved adaptive background mixture model for real-time tracking with shadow detection", Video-Based Surveillance Systems, Computer Vision. and Distributed Processing, 2002, pp. 134-144, and Non-Patent Documents: T. Bouwmans, F. El Baf, B. Vachon, "Background Modeling using Mixture of Gaussians for Foreground Detection --A Survey", Recent Patents on Computer Science . 1, 2008, pp. 219-237

Further, in the mask processing of the above (A3), the color calculated from the surrounding colors of other tracking targets may be blurred, and the mask processing target area can be blurred by the same method as the image smoothing process. Specifically, a predetermined kernel may be set, and the smoothing process may be performed by performing a convolution operation with the kernel on the mask processing target area.

FIG. 3 is a schematic view showing an embodiment of a pixel changing process (mask process) according to the present invention.

As shown in FIG. 3A, from the acquired time-series image group, two images of the current time t, which is the current time in the tracking process, and the previous time t'(= t−1) are obtained. The mask processing when performing tracking by using will be described. A specific method for inputting such two images into the classifier 11 for tracking will be described in detail later with reference to FIG. Also, as will be described in detail later, tracking with mask processing is performed using three or four or more images including images at a time t'' (t'' <t') earlier than the time t'. It is also possible to do.

In the present embodiment, from the two images at time t and time t', the tracking target area (c _u ^t , c _v ^t ,) which is a rectangular image area related to the object i which is one tracking target at time ^t . The tracking process is performed by estimating w ^t , h ^t ). Here, c _u ^t and c _v ^t are the u coordinate and v coordinate of the center of the tracking target area (center of the object i) in the image coordinate system, respectively, and w ^t and h ^t are the widths of the tracking target area, respectively. And the height.

First, as shown in FIG. 3B, in each image at time t and time t'(= t−1), the tracking target area (c _u ^t ', c) of an object other than the object i at time t' _{Based on v} ^t ', w ^t ', h ^t '), the mask processing target area, which is the image area to be masked, is determined. For example, the mask processing target area can be an area (c _u ^t ', c _v ^t ', A · w ^t ', B · h ^t ') including a tracking target area of an object other than the object i.

Here, if A = B = 1, the mask processing target area is the tracking target area itself of the object other than the object i. The size (area) of the mask processing target area is a design matter that can greatly affect the tracking accuracy. For example, it is preferable to set a larger area within a range in which the probability of overlapping the tracking target area of the object i is equal to or less than a predetermined range.

Incidentally, as described above, even in the embodiment using three or more images, the object other than the object i at one time (for example, the newest time excluding the current time t, the intermediate time, or the oldest time) is the same as above. The masked area can be determined based on the tracked area of the object. In any case, the mask processing unit 111 (FIG. 2) performs the mask processing (pixel change processing) as described above on the mask processing target area determined as described above.

Next, as also shown in FIG. 3B, a reference image area is set for each of the masked time t and time t'(= t−1) images, and this reference image is set. Determine the area to be cut from the area. The image portion of the image area to be cut becomes an image area (image patch) that is later cut from each image and input to the classifier 11.

Here, when the reference image area is the tracking target area (c _x ^t ', c _y ^t ', w ^t ', h ^t ') at the time t'(= t−1), the cropping target area is, for example. It may be (c _x ^t ', c _y ^t ', C · w ^t ', D · h ^t '). Here, if C = D = 2, the area to be cut becomes an area having four times the area including the area to be tracked.

Incidentally, as described above, also in the embodiment using three or more images, an object other than the object i at one time (for example, the newest time excluding the current time t, the intermediate time, or the oldest time). The area to be cut can be determined based on the area to be tracked. In any case, next, an image portion (image patch) is cut out from the determined cutout target area, the image patch is input to the classifier 11, and the object i is tracked from the output of the classifier 11 at time t. The region is determined and the classifier 11 is trained.

As a matter of course, the reference image area that serves as a reference for the mask processing target area and the cut target area is not limited to the tracking target area at the time before the current time t. That is, the reference image area may be the same image area (with the same coordinate range) in the image at each time as described above, but it may be set so as to be a different image area for each time. For example, in the image at each time, the tracking target area at the previous time (for example, the time immediately before) may be used as the reference image area, and the image area at the time predicted from the trajectory information of the object position may be used as the reference image area. It is also possible to do.

FIG. 4 is a schematic view showing another embodiment of the pixel change processing (mask processing) according to the present invention.

In the present embodiment, as shown in FIG. 4A, the tracking target area of the object i, which is one tracking target, and the mask processing target area of another object are superimposed on the masked image at one time. If so, take measures to maintain or improve the identification accuracy. In fact, in the case of superimposing in this way, a part of the image information related to the object i is replaced with the pixel information related to the mask processing, so that the classifier 11 inputs such a masked image (or an image). You may not receive some of the correct information for identification or learning from the patch), or conversely accept the incorrect information.

Therefore, in the present embodiment, as shown in FIG. 4B, when even one of one or more existing mask processing target areas has a superposed area superimposed on the tracking target area of the object i, this ( Mask processing (pixel change processing) is not performed on one or more superposed areas. As a result, the classifier 11 can receive all the correct information in the tracking target area.

Further, as a modification mode, as shown in FIG. 4C, when even one of one or more existing mask processing target areas has a superposed area superimposed on the tracking target area of the object i, the object i It is also preferable to change the tracking target area of the above to a region excluding this (one or more) overlapping regions. By the way, such an area change process is carried out by the tracking area determination unit 112 (FIG. 2).

Here, the cropping target area for determining the image patch input to the classifier 11 is set based on the changed tracking target area. Further, when an image patch excluding such a superposed region is input to the classifier 11, the entire tracking target area of the object i is changed from the area estimated based on the output of the classifier 11. It may be decided by returning it.

FIG. 5 is a schematic view showing still another embodiment of the pixel changing process (mask process) according to the present invention.

In the present embodiment, when the times of the images are separated from each other by a predetermined value or more, particularly when three or more images are used, the mask processing target area does not deviate significantly from the actual image position of an object other than the object i. Take action to make it.

For example, as shown in FIG. 5A, an image at the current time t and an image at a time t'' (t'' <t) separated from the current time t by a predetermined time or more retroactively from the time series image group. A case where two or more image groups including and is taken out and mask processing is performed will be described. In this case, in the image at time t'', for example, the mask processing may be performed with the tracking target area of an object other than the object i as the mask processing target area, but on the other hand, in the image at time t, the mask processing may be performed. It is also preferable to perform masking on the approximate position range of these other objects.

Here, as the approximate position range of the other object at the time t, for example, as shown in FIG. 5B, the average velocity vector v (t'') around the time t'' in each other object. Is calculated, and the estimated movement vector r from time t'' to time t is calculated by the following equation (1) r ＝ v (t'') · (t''−t).
The area where the tracking target area of each other object at time t'' is translated by the estimated movement vector r of each other object is the area to be masked for each other object in the image at time t. May be.

Further, as a modification mode, it is also possible to predict the position at time t and determine the mask processing target area based on the locus information from time t'' to time t-1 for the position of each other object. Is.

Further, as a modification mode, when the classifier 11 is trained, the mask processing unit 111 (FIG. 2) performs mask processing on the deviation processing target area whose position deviates from the mask processing target area determined as described above. May be carried out. Here, the position deviation may be a deviation of a random amount of deviation within a predetermined range in a random direction, or may be a deviation of a predetermined amount of deviation in a preset predetermined direction. By learning the masked image (image patch) related to the deviation processing target area in advance, the classifier 11 should actually mask the masked area at the time of actual detection (identification). It is possible to perform more appropriate detection (identification) on a masked image (image patch) that is likely to be deviated from the position of the object to some extent.

Returning to FIG. 2, the image output unit 113 outputs an image or an image area at one or a plurality of times subjected to the mask processing (pixel change processing) as described above to the classifier 11. Here, the learning unit 113a of the image output unit 113 outputs the mask-processed image or the mask-processed image area (image patch) to the classifier 11 for learning the object i which is one tracking target.

On the other hand, the identification unit 113b of the image output unit 113 outputs the mask-processed image or the mask-processed image patch to the classifier for identifying the object i. Here, in performing a plurality of object tracking (multi-tracking), a mask process is performed in which the objects to be tracked are treated as objects i one by one, and the masked image (image patch) is used on the classifier 11. On the other hand, the identification of each object (determination of the position of the image area of the object) is sequentially performed.

By the way, the input of the mask-processed image (mask-processed image patch) may be performed only at the time of identification by the classifier 11. In this case, the learning of the classifier 11 is performed by inputting an image (image patch) that has not been masked. Further, the mask processed image (mask processed image patch) may be input only when the classifier 11 is learned, but the mask processed image (mask processed image patch) is input in both the learning and the identification. It is also preferable. That is, it is also preferable to perform discrimination on the masked image (image patch) by using the classifier 11 learned from the masked image (masked image patch).

Next, two embodiments of the non-online learning type and the online learning type in the classifier 11 (FIG. 2) will be described with reference to FIGS. 6 and 7, respectively.

FIG. 6 is a schematic view showing an embodiment of the classifier according to the present invention.

As shown in FIG. 6, the classifier 11 has the same embodiment.
(A) Convolutional layers as a plurality of first neural networks that input an image or an image area (image patch) at each time point and output feature information related to these features.
(B) Fully-Connected Layers as a second neural network that inputs a plurality of feature information output from these convolutional layers and outputs information related to the position of one tracking target. Including.

Here, the convolutional layer portion (a) has a function of imitating the function of simple cells in the visual cortex of an animal, and a convolutional process of sliding a kernel (weighted matrix filter) on an image to generate a feature map. To execute. By this convolution process, it is possible to extract basic features such as edges and gradients and obtain information on local correlation patterns while gradually reducing the resolution of the image.

For example, it is possible to use AlexNet using five convolutional layers as the convolutional layer portion. In this AlexNet, each convolutional layer is paired with a pooling layer, and the convolutional process and the pooling process are repeated. Here, the pooling process is a process that imitates the function of complex cells in the visual cortex of an animal, and the feature map (reaction of the convolution filter within a certain area) output from the convolutional layer is set to the maximum value or the average value. In summary, it is a process to ensure local translation invariance while reducing adjustment parameters. AlexNet is described, for example, in Krizhevsky, A., Sutskever, I., and Hinton, GE, "Imagenet classification with deep convolutional neural networks", Advances in Neural Information Processing Systems 25, 2012, pp. 1106-1114. There is.

Further, as the convolutional layer portion and the fully connected layer portion, for example, the neural network described in Non-Patent Document 2 may be adopted. Incidentally, in the present embodiment, the number of convolutional layer portions (first neural network) may be two, but in order to realize higher identification accuracy (tracking accuracy), the number should be three or more as shown in FIG. Is also preferable.

The classifier 11 is learning offline in advance before performing the tracking target detection (identification) process at each time point. Specifically
(A) A set of two or three or more images (or image patches) that move back and forth in time,
(B) Learning of the image output unit 113 of a data set including a four-dimensional vector (CU _C , CV _C , CW, CH) that defines the position and range of the correct answer tracking target area in each image (or each image patch). A large amount of data is input from unit 113a for learning.

Here, CU _C and CV _C are the object center in the u-axis direction and the object center in the v-axis direction of the correct image region, respectively. In addition, CW and CH are the width and height of the correct image area rectangle, respectively. Incidentally, if the input is an image patch, the coordinate system based on the CU _C values and CV _C value is a local coordinate system of the image patch. Further, it is also preferable that the input image (image patch) is a mask-processed image (image patch) subjected to the mask processing as described above.

It is also preferable that the discrimination model generated by the discriminator 11 trained in this way is stored in the discriminative model storage unit 104 (FIG. 2). For example, the stored discriminative model may be ported to an external discriminator for use.

Next, in the detection (identification) of a plurality of objects to be tracked, the plurality of convolutional layer portions of the classifier 11 are masked images (images) at time t from the identification unit 113b (FIG. 2) of the image output unit 113. Input a masked image (image patch) group including the patch). Here, it is also preferable that each convolutional layer unit inputs each of a plurality of times having a time interval set at the time of learning in the same time order (time allocation) as at the time of learning.

In FIG. 6, the masked image (image patch) at the current time t is input to the uppermost convolution layer portion in the figure, and the time t'(t'<t) is input to the second convolution layer portion from the top in the figure. The masked image (image patch) of is input, and the masked image (image patch) at time t''(t''<t') is input to the third convolution layer from the top in the figure. .. Here, in the present embodiment, at the time of learning, the uppermost convolutional layer portion in the figure, the second convolutional layer portion from the top, and the third convolutional layer portion from the top are at a certain time T and time (T−t + t, respectively. ') And the time (T−t + t'') image (image patch) or masked image (image patch) are input for learning.

Further, in the detection (identification) of a plurality of objects to be tracked, the classifier 11 (plurality of convolutional layers) sequentially inputs a masked image (image patch) for each of the plurality of objects as an identification target. Multi-tracking is realized by (outputting the identification result).

Then, over the total binding layer that inputs the feature quantity output from the convolutional layer entered the masking image (image patches) as described, finally, four-dimensional vector _{(U C, V C, W} , H ) Is output. Here, U _C and V _C are the object center in the u-axis direction and the object center in the v-axis direction of the tracking target region for the object to be identified in the input masked image (image patch), respectively. W and H are the width and height of the tracking target area rectangle, respectively. Incidentally, if the input is an image patch, the coordinate system based on the U _C value and V _C value is a local coordinate system of the image patch.

Further, as shown in FIG. 6, the image output unit 113 (FIG. 2) is changed from the convolution layer unit to the fully connected layer unit (second neural network) during learning and identification of the classifier 11. Information related to a time point in an image (image patch) corresponding to the output feature information (time interval Δt between images in FIG. 6) may be linked to this feature information and output. As a result, the classifier 11 can also learn information related to the time of a plurality of input images (image patches), and can perform more accurate detection (discrimination) based on the information. In addition, it is possible to secure a variety of times in a plurality of images (image patches) input to the classifier 11.

In any case, the classifier 11 inputs an image (image patch) that has been masked to an object other than the tracked object at that stage from the image output unit 113, so that the tracked object can be identified. The accuracy is improved and the occurrence of drift can be suppressed.

FIG. 7 is a schematic view showing another embodiment of the classifier according to the present invention.

According to FIG. 7, the classifier 11'of this embodiment has a configuration in which a support vector machine (SVM) capable of performing machine learning is connected to the output side of a convolutional neural network (CNN) including a convolutional layer.

The classifier 11'is applied to this image patch with respect to this mask processed image patch while learning the mask processed image area (image patch) input from the image output unit 113 every moment (at the current time t) online. A binary judgment is made as to whether or not the reflected object is a tracking target object. Specifically, while generating and updating the identification boundary surface in the feature space, the signed distance d from the identification boundary surface is calculated as the reliability, and it is determined whether or not the reliability is equal to or higher than a predetermined threshold value. It is. Regarding object tracking using such a classifier, for example, S. Hare, A. Saffari and PHS Torr, "Struck: Structured Output Tracking with Kernels", Publications of International Conference on Computer Vision (ICCV), 2011. Years, pp. 263-270.

The masked image patch (at the current time t) output by the image output unit 113 every moment can be a plurality of image patch candidates that can be the tracking target area of the correct answer at the time t. In this case, the classifier 11'determines the correct tracking target area from these candidates.

As described above, the classifier 11'inputs an image (image patch) that has been masked to an object other than the tracked object at that stage from the image output unit 113, so that the tracked object The identification accuracy is improved, and the occurrence of drift can be suppressed.

Returning to FIG. 2, the target position determination unit 114 determines the position of the tracked object at the current time t based on the output from the learned classifier 11. Specifically, for each of a plurality of objects are tracked on the basis of the four-dimensional vector output from the discriminator _{11 (U C, V C,} W, H), finally tracking in image at time t Calculate the target area (c _u ^t , c _v ^t , w ^t , h ^t ). Further, as a modification, the target position determination unit 114 may determine the position of the tracked object at the current time t based on the output from the classifier 11'(FIG. 7) learning online.

The tracking target management unit 115 determines these from the information of the tracking target area (c _u ^t , c _v ^t , w ^t , h ^t ) in each of the plurality of objects to be tracked, which is determined by the target position determination unit 114. For each of a plurality of objects, tracking history information (traffic line information) associated with a position in the world coordinate system Gx-Gy-Gz (stretched in real space) is generated and managed for each time. As one application example, this makes it possible to more accurately grasp the flow lines of a large number of clerk, customers, etc. who stay or move in the store.

Further, it is also preferable that the generated tracking history information (traffic line information) is stored in the target information storage unit 105 every time it is generated / updated or as appropriate. Further, the information may be transmitted to the external information processing device 3 via the communication control unit 121 and the communication interface 101.

As described in detail above, the present invention identifies a plurality of objects as tracking targets by performing "pixel change processing" on an object other than one object to be tracked. It eliminates or reduces the characteristics of the image or other object in the image area that is input to the vessel. As a result, this one object is confused with another object by training the classifier and / or having the classifier perform the discriminating process using the image or image region subjected to such processing. It is possible to continue to identify more reliably without doing so.

In other words, by using the "image object masking method" of the present invention described above, even when an object having a similar appearance exists near the object to be tracked, the occurrence of drift, which has been a problem in the past, can be caused. It becomes possible to suppress it.

By the way, this "image object mask method" makes it possible to suppress the increase in memory consumption (calculation cost) and the occurrence of drift, especially when applied to multi-tracking processing that does not perform online learning. It will be a very important technology to do.

Further, the configuration and method of the present invention include, for example, a monitoring system that monitors a place where a large number of people move, stay, and enter and exit, and entering, resting, and watching a game of a person in a shopping district or a commercial or service facility. It can be applied to various systems such as event participation and marketing research system for investigating the situation of movement.

In the various embodiments of the present invention described above, various changes, modifications and omissions in the technical idea and the scope of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Target tracking device 101 Communication interface 102 Image storage unit 103 Mask processing image storage unit 104 Identification model storage unit 105 Target information storage unit 11, 11'Identifier 111 Mask processing unit 112 Tracking area determination unit 113 Image output unit 113a Learning unit 113b Identification unit 114 Target position determination unit 115 Tracking target management unit 121 Communication control unit 2 Camera 3 Information processing device

Claims (12)

Based on the outputs from a plurality of images when included in the image group of sequences, which may include the tracking object or discriminator as input image region in said image, determining the position of each of said plurality of tracking object in the image A device that tracks the tracking target using a target positioning means .
An image included in the image group or a processing target area in an image area within the image, and for other tracking targets other than one tracking target, in an image related to a time point earlier than the image or at the past time point. A mask that performs pixel change processing that changes the processing target area determined based on the position determined in the image area in the image to a pixel pattern that eliminates or reduces the characteristics of the other tracking target. Processing means and
The image or the image area subjected to the pixel change processing at one time point, and the image or the image area subjected to the pixel change process at one or more time points prior to the one time point. and possess an image output means for outputting to the learned of the identifier,
The target position determining means has been trained by inputting the image or the image area that has undergone the pixel change processing at a plurality of time points, which is the one time point and the one or more time points before the time point. Determine the position of the one tracked object at that one point in time based on the output from the classifier.
Object-tracking device according to claim to Rukoto. The image output means also inputs information related to the plurality of time points into the trained classifier.
The target position determining means is from the trained classifier in which the image or the image area subjected to the pixel change processing at the plurality of time points and the information related to the plurality of time points are input together. The target tracking device according to claim 1, wherein the position of the one tracking target at the one time point is determined based on the output. The discriminator is a plurality of first neural networks, each of which inputs the image or the image area that has undergone the pixel change processing at the plurality of time points , and sets the characteristics of the image or the image area , respectively. a plurality of first neural networks generating feature information relating a plurality of the feature information output from the first neural network of the plurality of, and as input information relating to the plurality of time points, at the one time point The target tracking device according to claim 2 , further comprising a second neural network that outputs information relating to the position of the one tracking target. The trained classifier is the image or the image area that has undergone the pixel change processing at a certain time point, and the pixel change process that has been subjected to the pixel change process at one or more time points prior to the certain time point. Any one of claims 1 to 3, wherein the learning is performed by a learning data set including an image or the image area and information relating to the position of the tracking target as correct answer data. The target tracking device described in. The mask processing means has a pixel pattern determined based on the image or an image area other than the tracking target in the image area, or a predetermined pixel pattern for at least one processing target area related to the other tracking target. The target tracking device according to any one of claims 1 to 4 , wherein the change is made to. The mask processing means, as the pixel change process, with respect to the processing target area according to at least one of the other tracked from claim 1, characterized in that carrying out the process of blurring the appearance of the other tracked The target tracking device according to any one of 5 . When the processing target area related to the other tracking target has a superimposing area superimposed on the tracking target area including the position determined for the one tracking target, the mask processing means is concerned with the superimposing area. The target tracking device according to any one of claims 1 to 6 , wherein the pixel change process is not performed. When the processing target area related to the other tracking target has a superimposing area superimposed on the tracking target area including the position determined for the one tracking target, the tracking target area related to the one tracking target is used. The target tracking device according to any one of claims 1 to 7 , further comprising a tracking target area determination unit that changes to a region excluding the superposed region. The mask processing means according to any one of claims 1 to 8 is characterized in that when the discriminator is trained, the pixel change processing is performed on a determined area to be processed that is displaced from the processing target area. The target tracking device according to any one item. The image output means outputs the image or the image area at at least one time point after the pixel change processing is performed to the classifier for learning online.
The target position determining means, and characterized on the basis of an output from the discriminator which performed learning online, the that <br/> that to determine the position of one of the tracked in the one time point The target tracking device according to any one of claims 1 to 9 . Based on the outputs from a plurality of images when included in the image group of sequences, which may include the tracking object or discriminator as input image region in said image, determining the position of each of said plurality of tracking object in the image A program that activates a computer mounted on a device that tracks a tracking target using a target positioning means .
An image included in the image group or a processing target area in an image area within the image, and for other tracking targets other than one tracking target, in an image related to a time point earlier than the image or at the past time point. A mask that performs pixel change processing that changes the processing target area determined based on the position determined in the image area in the image to a pixel pattern that eliminates or reduces the characteristics of the other tracking target. Processing means and
The image or the image area that has undergone the pixel change processing at one time point, and the image or the image area that has undergone the pixel change process at one or more time points prior to the one time point. the causes the computer to function as an image output means for outputting to the learned of the identifier,
The target position determining means has been trained by inputting the image or the image area that has undergone the pixel change processing at a plurality of time points, which is the one time point and the one or more time points before the time point. Determine the position of the one tracked object at that one point in time based on the output from the classifier.
Object-tracking program characterized and this. Based on the outputs from a plurality of images when included in the image group of sequences, which may include the tracking object or discriminator as input image region in said image, determining the position of each of said plurality of tracking object in the image A target tracking method in a computer mounted on a device that tracks the tracking target using a target positioning means .
An image included in the image group or a processing target area in an image area within the image, and for other tracking targets other than one tracking target, in an image related to a time point earlier than the image or at the past time point. A step of performing pixel change processing for changing a processing target area determined based on a position determined in an image area in the image to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced. When,
The image or the image area subjected to the pixel change processing at one time point, and the image or the image area subjected to the pixel change process at one or more time points prior to the one time point. and a step of outputting to the learned of the identifier,
The image or the image area that has undergone the pixel change processing at a plurality of time points, which is a step performed by the target position determining means and is the one time point and the one or more time points before the time point. A target tracking method comprising a step of determining the position of the one tracked object at the one time point based on the input output from the trained classifier .

Images