JP6946231B2 - Object tracking device and object tracking method - Google Patents

Description

The present invention relates to an object tracking device and an object tracking method for tracking an object included in a moving image.

Conventionally, a device for tracking an object included in a moving image using a particle filter has been known. Patent Document 1 discloses a technique for tracking the contour shape of an object using a particle filter.

JP-A-2009-224924

By the way, when tracking an object included in a moving image using a particle filter, the likelihood of each particle after movement, which is estimated based on the observed value acquired from the frame image, is used. In this case, for example, when tracking an object whose size or orientation changes, the difference between the objects becomes large between the plurality of frame images, and the accuracy of estimating the likelihood deteriorates. Further, when tracking each of a plurality of similar objects, it becomes impossible to distinguish each of the plurality of objects due to the change in the size or orientation of the objects, and the accuracy of estimating the likelihood deteriorates. As a result, there is a problem that the accuracy of tracking the object is deteriorated.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide an object tracking device and an object tracking method capable of improving the accuracy of tracking an object included in a moving image.

The object tracking device according to the first aspect of the present invention includes a moving image acquisition unit that acquires a moving image generated by the imaging device, a front frame image that is a frame image included in the moving image, and the moving image. A plurality of processing layers included in a machine learning model capable of outputting the type of an object included in the moving image based on the input moving image for each of the current frame images which are the frame images after the previous frame image. Is commonly activated in both the treatment layer of the propagation control unit for propagating the image, the post-stage treatment layer selected from the plurality of treatment layers, and the pre-stage treatment layer which is the treatment layer immediately before the post-stage treatment layer. One or more current frame outputs output from the post-processing layer and the pre-processing layer based on the frame image, and one or more pre-frames output from the post-processing layer and the pre-processing layer based on the pre-frame image. An extraction unit that extracts the output, one or more current frame feature points are detected based on the one or more current frame outputs, and one or more previous frame feature points are detected based on the one or more previous frame outputs. A plurality of pre-transition particles are generated in association with the feature point detection unit to perform the previous frame image, and the positions of the one or more current frame feature points and the plurality of pre-transition particles are generated according to a predetermined object transition model. Based on the likelihood estimation unit that estimates the likelihood of each of the plurality of transition particles by comparing the positions of the plurality of transitioned post-transition particles, and the likelihood estimated by the likelihood estimation unit. It has a tracking unit that tracks an object included in the moving image based on a part of the post-transition particles selected from the plurality of post-transition particles.

In the extraction unit, a plurality of subsequent current frame outputs output from the subsequent processing layer by propagating the current frame image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers, and the above-mentioned Since the pre-frame image propagated in the order of the pre-processing layer and the post-processing layer, one or more post-stage current frame outputs and one or more post-stage current frame outputs that are commonly activated from the plurality of post-stage pre-frame outputs output from the post-stage processing layer. A plurality of pre-stage current frames output from the post-stage extraction unit that extracts the post-stage pre-frame output, and the pre-stage processing layer that has been a factor in activating the one or more post-stage current frame outputs and the one or more post-stage front frame outputs. The output and the pre-stage extraction unit that extracts one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs that are commonly activated among the plurality of pre-stage pre-frame outputs output from the pre-stage processing layer. You may have.

The pre-stage extraction unit has one or more pre-stage current frame outputs and one or more pre-stage pre-frames based on the activated size of the plurality of pre-stage current frame outputs and the plurality of pre-stage pre-frame outputs. The output may be extracted.

The machine learning model includes a convolutional neural network, and the post-processing layer may be any one of an output layer, a fully connected layer, a regularization layer, a pooling layer, and a convolutional layer.
The pretreatment layer may be any one of a fully bonded layer, a normalized layer, a pooling layer, a convolution layer, and an input layer.

When the last layer, which is the last treatment layer, is selected as the latter treatment layer among the plurality of treatment layers, the extraction unit commonly activates the one or more current frames in the last treatment layer. When there is no output and the one or more previous frame outputs, the one or more current frame outputs and the one or more previous frame outputs that are commonly activated in the processing layer before the last layer may be extracted. ..

The likelihood estimation unit may estimate that the closer the post-transition particles are to the positions of one or more current frame feature points, the higher the likelihood.

The likelihood estimation unit transitions the positions of the one or more current frame feature points and one or more pre-transition particles using the one or more previous frame feature points as particles according to the predetermined object transition model. The likelihood of each of the one or more post-transition particles may be estimated by comparing the positions of the post-transition particles.

In the case of estimating the likelihood, the likelihood estimation unit generates the plurality of pre-transition particles arranged in the front frame image based on a rule different from the case of estimating the likelihood from the next time onward, and the one or more. The likelihood of each of the plurality of post-transition particles may be estimated by comparing the position of the current frame feature point of the current frame with the positions of the plurality of post-transition particles.

The likelihood estimation unit compares the positions of the one or more current frame feature points corresponding to the one or more previous frame feature points with the positions of the plurality of post-transition particles, thereby performing the plurality of post-transition particles. The likelihood of each particle may be estimated.

The object tracking device further includes an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers, and the extraction unit receives the instruction received by the instruction receiving unit. The treated layer indicated by the above may be used as the post-stage treatment layer.

The extraction unit selects one of the plurality of processing layers as the post-stage processing layer, extracts the one or more current frame outputs and the one or more pre-frame outputs, and then selects the pre-stage processing layer. The processed layer may be selected as the subsequent processing layer to extract another one or more current frame outputs and one or more previous frame outputs.

The object tracking method according to the second aspect of the present invention includes a step of acquiring a moving image generated by an imaging device, a front frame image which is a frame image included in the moving image, and the front frame image included in the moving image. Each of the current frame images, which is the frame image after the frame image, propagates a plurality of processing layers included in the machine learning model capable of outputting the type of the object included in the moving image based on the input moving image. Based on the current frame image that is commonly activated in both the step, the post-stage treatment layer selected from the plurality of treatment layers, and the pre-stage treatment layer that is the treatment layer immediately before the post-stage treatment layer. Extract one or more current frame outputs output from the post-processing layer and the pre-processing layer and one or more pre-frame outputs output from the post-processing layer and the pre-processing layer based on the pre-frame image. A step, a step of detecting one or more current frame feature points based on the one or more current frame outputs, and a step of detecting one or more previous frame feature points based on the one or more previous frame outputs, and the previous step. A step of generating a plurality of pre-transition particles in association with a frame image, the positions of one or more current frame feature points, and a plurality of post-transitions in which the plurality of pre-transition particles are transitioned according to a predetermined object transition model. A step of estimating the likelihood of each of the plurality of post-transition particles by comparing the positions of the particles, and a part of the post-transition particles selected from the plurality of post-transition particles based on the estimated likelihood. Based on, it has a step of tracking an object included in the moving image.

In the extraction step, a plurality of post-stage current frame outputs output from the post-stage processing layer due to the propagation of the current frame image in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers, and One or more subsequent current frame outputs and one or more that are commonly activated from the plurality of subsequent pre-frame outputs output from the post-processing layer due to the propagation of the pre-frame image in the order of the pre-processing layer and the post-processing layer. A pre-stage extraction step for extracting the post-stage front frame output, and a plurality of pre-stage currents output from the pre-stage processing layer that have been a factor in activating the one or more post-stage current frame outputs and the one or more post-stage front frame outputs. A frame output and a post-stage extraction step for extracting one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs that are commonly activated among a plurality of pre-stage front frame outputs output from the pre-stage processing layer. May have.

After executing the pre-stage extraction step, the post-stage extraction step is performed by using the one or more pre-stage current frame outputs and the one or more pre-stage front frame outputs as the plurality of post-stage current frame outputs and the plurality of post-stage front frame outputs. You may do it.
The latter-stage extraction step and the first-stage extraction step may be executed for each of the plurality of processing layers.

According to the present invention, there is an effect that the accuracy of tracking an object included in a moving image can be improved.

It is a figure for demonstrating the outline of the object tracking process. It is a figure which shows an example of the structure of the machine learning model. It is a figure which shows the structure of the object tracking apparatus. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the detail of the object tracking process. It is a flowchart which shows the flow of the process performed by the object tracking apparatus. It is a flowchart which shows the flow of the process performed by the extraction part.

[Overview of Object Tracking Device 1]
FIG. 1 is a diagram for explaining an outline of the object tracking process. As a technique for estimating the internal state of an observation target that changes from moment to moment, there is a technique using a time series filter. The time series filter is a method of estimating the internal state of an object that cannot be directly observed based on the observation vector observed from the frame image included in the moving image, and is, for example, a particle filter.

In the particle filter, the probability distribution of the internal state of the observation target is expressed by the distribution of particles (hereinafter referred to as particles). In the particle filter, the posterior probability distribution of the state at the current time is used as the prior probability distribution of the state at the next time, and the template estimated from the state of the particles representing the prior probability distribution (sample set generated according to the prior probability distribution). The probability of each particle is obtained by comparing the observation (predicted sample) with the actual image at the next time (actual observation).

Then, in the particle filter, the posterior probability distribution of the particles is estimated from the obtained likelihood and the prior probability distribution. In the particle filter, the dynamically changing state of the observation target (for example, the tracking target) is sequentially estimated by repeating the above processing after the next time.

In the particle filter, the following processes (1) to (4) are performed. First, in the process (1), a predicted sample at a time one period ahead is generated for each sample (each particle). In the process (2), the weight (likelihood) is calculated for each prediction sample generated in the process (1), that is, the probability (likelihood) of obtaining the observation vector is estimated according to the observation model.

In the process (3), a plurality of particles are restored and extracted at a ratio proportional to the weight (likelihood). The posterior probability distribution (probability distribution of the internal state of the observation target) is obtained from the distribution of the particles restored and extracted in this way. In the process (4), the time is advanced and the process (1) is returned. At this time, the posterior probability distribution acquired in the process (3) is used as the prior probability distribution at the next time.

In this way, the particle filter estimates the parameters representing the state of the observation target, which change from moment to moment, by repeatedly predicting the prior probability distribution of the parameters representing the state of the observation target and calculating the posterior probability distribution. .. The particle filter is used, for example, to track the position of an object on a moving image. In this case, in the particle filter, the likelihood is calculated by comparing the observation (prediction sample) estimated from the parameter representing the position of the object with the actual observation (for example, the image captured by a camera or the like). By resampling the particles based on the calculated likelihood, the posterior probability distribution of the parameters representing the state of the observation target can be obtained. The parameter representing the state of the tracked object is, for example, a parameter representing the position of the object.

The object tracking device 1 is a device that tracks an object included in a moving image by estimating the likelihood of a moving destination vector of particles used in a particle filter using a machine learning model M. The object tracking device 1 is, for example, a computer. The object is a subject included in the frame image, for example, a human being or a vehicle.

The object tracking device 1 acquires a moving image generated by the imaging device ((1) in FIG. 1). The image pickup device is a camera, a smartphone, or the like capable of capturing a moving image. The moving image A shown in FIG. 1 is a moving image showing a human being, which is an object T, running from the left side to the right side. The frame image A1 of the first frame included in the moving image A shown in FIG. 1 is used as the front frame image, and the frame image A2 of the second frame is used as the current frame image. The front frame image is one of a plurality of frame images included in the moving image. The current frame image is a frame image after the previous frame image, and is a frame image for tracking an object. That is, the current frame image is a frame for which the likelihood is estimated next to the frame image for which the likelihood was estimated immediately before.

The object tracking device 1 inputs the acquired moving image A into the machine learning model M and propagates a plurality of processing layers included in the machine learning model M ((2) in FIG. 1). The machine learning model M is a model learned to output the type of the object included in the moving image based on the input moving image. Specifically, the machine learning model M outputs the type of the object shown in each frame image of the moving image for each frame image based on the input moving image.

FIG. 2 is a diagram showing an example of the configuration of the machine learning model M. The machine learning model M includes a convolutional neural network (CNN). In this case, the machine learning model M is an input layer M1, a first convolution layer M2, a second convolution layer M3, a first pooling layer M4, a normalization layer M5, a third convolution layer M6, and a second. It has a pooling layer M7, a first fully connected layer M8, a second fully connected layer M9, and an output layer M10. In the present specification, of the two adjacent processing layers, the processing layer on the upstream side when the current frame image and the front frame image propagate is referred to as a pre-stage processing layer, and the processing layer on the downstream side is referred to as a post-stage processing layer. ..

The treatment layers that can be the subsequent treatment layers are the first convolution layer M2, the second convolution layer M3, the first pooling layer M4, the normalization layer M5, the third convolution layer M6, the second pooling layer M7, and the second. It is one of the fully bonded layer M8 of 1, the second fully bonded layer M9, and the output layer M10. The processing layers that can be the pre-stage processing layers are the input layer M1, the first convolution layer M2, the second convolution layer M3, the first pooling layer M4, the normalized layer M5, the third convolution layer M6, and the second. It is any one of the pooling layer M7, the first fully bonded layer M8, and the second fully bonded layer M9. The object tracking device 1 inputs the acquired moving image into the machine learning model M, and forward-propagates each processing layer from the input layer M1 to the output layer M10, that is, infers the object in the moving image. Output the type of T.

Returning to FIG. 1, the object tracking device 1 has a calculation result in each processing layer from which the machine learning model M outputs the type of the object T (for example, human, male, female, etc.), that is, the degree of abstraction by deep learning. The feature points common to the current frame image and the previous frame image are detected by using the high feature amount of (FIG. 1 (3)). Here, the object tracking device 1 detects common feature points in an order opposite to the order in which each frame image is forward-propagated. By doing so, the object tracking device 1 can detect feature points based on features with a high degree of abstraction.

The object tracking device 1 detects the common feature points (for example, the feature points included in the face, body, limbs, etc.) to obtain the feature points included in the object T and the previous frame image shown in the current frame image. It is detected that there is a correspondence between the feature points included in the imaged object T. The correspondence relationship is a relationship in which the pixels included in the current frame image indicated by the feature points and the pixels included in the previous frame image indicated by the feature points match or approximate.

The object tracking device 1 generates particles in association with the previous frame image ((4) in FIG. 1). The object tracking device 1 predicts the movement of the object T (for example, the object T moves from the left side to the right side) and transitions the generated particles. The object tracking device 1 estimates the likelihood of each particle by comparing the position of the feature point of the current frame image corresponding to the detected feature point of the previous frame image with the position of the transitioned particle. ((5) in FIG. 1). The object tracking device 1 compares, for example, the position of the feature point on the face of the object T in the current frame image, which has a relationship consistent with the feature point of the previous frame image, with the position of the particles near the face of the object T. Thereby, the likelihood of the particles near the face of the object T is estimated. Similarly, the object tracking device 1 estimates the likelihood of each of a plurality of particles in the vicinity of the body, limbs, and the like.

The object tracking device 1 estimates the position of the object T included in the current frame image based on the estimated likelihood ((6) in FIG. 1). In this way, the object tracking device 1 tracks the object T by repeating the processes (1) to (6) of FIG. As a result, the object tracking device 1 can improve the accuracy of tracking the object included in the moving image by estimating the likelihood of each of the plurality of particles based on the feature amount having a high degree of abstraction.
Hereinafter, the details of the object tracking device 1 will be described.

[Configuration of Object Tracking Device 1]
FIG. 3 is a diagram showing the configuration of the object tracking device 1. The object tracking device 1 has an operation unit 11, a storage unit 12, and a control unit 13.

The operation unit 11 is an input device that accepts user operations.
The storage unit 12 is, for example, a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 12 stores various programs executed by the control unit 13.

The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 controls the function related to the object tracking device 1 by executing the program stored in the storage unit 12. By executing the program, the control unit 13 executes a moving image acquisition unit 131, a propagation control unit 132, an extraction unit 133, an instruction reception unit 136, a feature point detection unit 137, a selection unit 138, a likelihood estimation unit 139, and a tracking unit. It functions as a unit 140.
The moving image acquisition unit 131 acquires a moving image generated by the imaging device. The moving image acquisition unit 131 acquires, for example, a moving image captured by an imaging device connected by an HDMI (High-Definition Multimedia Interface) (registered trademark) cable. The moving image acquisition unit 131 may acquire the moving image stored in the storage unit 12. The moving image acquisition unit 131 inputs the acquired moving image to the propagation control unit 132.

The propagation control unit 132 propagates a plurality of processing layers included in the machine learning model M to each of the current frame image and the previous frame image included in the moving image input from the moving image acquisition unit 131. In the example shown in FIG. 2, the propagation control unit 132 first propagates each processing layer from the input layer M1 to the output layer M10 included in the machine learning model M to each of the plurality of frame images included in the moving image in order. Let me. Then, the propagation control unit 132 selects the current frame image and the previous frame image based on the type of the object for each frame image output from the machine learning model M. The propagation control unit 132 selects, for example, a frame image group in which objects of the same type are continuously captured among a plurality of frame images included in the moving image as the current frame image and the previous frame image. The propagation control unit 132 inputs information indicating the selected current frame image and the previous frame image to the extraction unit 133.

The extraction unit 133 is commonly activated in both the post-stage processing layer selected from the plurality of processing layers and the pre-stage processing layer which is the processing layer immediately before the post-stage processing layer, and the subsequent stage is based on the current frame image. One or more current frame outputs output from the processing layer and the pre-stage processing layer and one or more pre-frame outputs output from the post-stage processing layer and the pre-stage processing layer are extracted based on the pre-frame image. The details of the extraction process performed by the extraction unit 133 will be described later, but the extraction unit 133 is a part of the current frame output and the previous frame output which are a part of the current frame output commonly activated in the subsequent processing layer. The rear stage extraction unit 134 that extracts a certain rear stage front frame output, the front stage current frame output that is a part of the current frame output that is commonly activated in the front stage processing layer, and the front stage front frame output that is a part of the previous stage front frame output. It has a pre-stage extraction unit 135 for extraction.

The current frame output and the previous frame output extracted by the extraction unit 133 are information indicating the activated unit among the plurality of units included in the processing layer. A unit is one or more pixels contained in a frame image. The definition of activation may be, for example, when the product of the output value of the unit or the output value of the unit and the weight of the connection of the unit exceeds a predetermined threshold value, or a predetermined number or a predetermined number in descending order of output. It may be included in the ratio. Further, in the processing layer other than the fully connected layer, for example, it may be included in a predetermined number or a predetermined ratio in descending order of output for each channel. The channel is the output calculated by convolution for each filter.

It is preferable that the extraction unit 133 selects the last treatment layer, which is the last treatment layer, as the post-treatment layer among the plurality of treatment layers. However, there are cases where there is no current frame output and previous frame output that are commonly activated in the rearmost layer. Therefore, the extraction unit 133 selects one or more current frame outputs and one or more pre-frame outputs that are commonly activated in the rearmost layer when the rearmost layer is selected as the subsequent processing layer among the plurality of processing layers. If there is no such, one or more current frame outputs and one or more previous frame outputs that are commonly activated in the processing layer before the last layer may be extracted.

For example, when the extraction unit 133 selects the output layer M10, which is the rearmost layer, as the post-processing layer, there is no one or more current frame outputs and one or more previous frame outputs that are commonly activated in the output layer M10. And. In this case, the extraction unit 133 repeatedly searches for one or more current frame outputs and one or more previous frame outputs that are commonly activated for each processing layer before the output layer M10. The extraction unit 133 is, for example, when there is one or more current frame outputs and one or more previous frame outputs that are commonly activated in the second fully connected layer M9, which is the processing layer immediately before the output layer M10. The fully bonded layer M9 of 2 is selected as the post-treatment layer. Then, the extraction unit 133 extracts one or more current frame outputs and one or more front frame outputs that are commonly activated in the second fully connected layer M9 selected as the post-processing layer. By doing so, the extraction unit 133 can associate the objects reflected in each of the current frame image and the previous frame image even when there are few matching areas.

The extraction unit 133 may select the processing layer specified by the user as the subsequent processing layer. Specifically, the instruction receiving unit 136 receives an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 11. Then, the extraction unit 133 uses the processing layer indicated by the instruction received by the instruction reception unit 136 as the subsequent processing layer. In the example shown in FIG. 2, the extraction unit 133 uses the second fully connected layer M9 indicated by the instruction received by the instruction receiving unit 136 as the subsequent processing layer when the user selects the second fully connected layer M9. use. The extraction unit 133 inputs the extracted current frame output and the previous frame output to the feature point detection unit 137.

The feature point detection unit 137 detects one or more current frame feature points based on one or more current frame outputs, and detects one or more previous frame feature points based on one or more previous frame outputs. Specifically, the feature point detection unit 137 first searches for a corresponding feature point based on one or more current frame outputs and one or more previous frame outputs. Then, the feature point detection unit 137 detects one or more current frame feature points based on one or more current frame outputs and one or more previous frame feature points based on one or more previous frame outputs. The feature point detection unit 137 inputs the detected current frame feature point and the previous frame feature point to the selection unit 138.

The selection unit 138 is composed of one or more current frame feature points and one or more previous frame feature points specified by the feature point detection unit 137, and some current frame feature points and some previous frame feature points based on mutual correspondence. Select a feature point. Specifically, the selection unit 138 removes the erroneously detected correspondence and selects one or more current frame feature points and one or more previous frame feature points based on the correspondence after the removal. False detection of the correspondence relationship is a state in which the correspondence relationship between the current frame feature point and the previous frame feature point is inconsistent. The selection unit 138 removes the correspondence by, for example, narrowing down based on the RANSAC (Random Sampling Consensus) method or the least squares median (LMedS: Least Median of Square) method.

The likelihood estimation unit 139 estimates the likelihood to be used in the particle filter based on one or more current frame feature points and one or more previous frame feature points detected by the feature point detection unit 137. The details of the likelihood estimation process performed by the likelihood estimation unit 139 will be described later, but the likelihood estimation unit 139 first generates a plurality of pre-transition particles in association with the previous frame image, and the generated plurality of pre-transition particles. Is transitioned according to a predetermined object transition model. Then, the likelihood estimation unit 139 compares the positions of one or more current frame feature points corresponding to one or more previous frame feature points with the positions of the plurality of transitioned post-transition particles, thereby causing a plurality of transitions. Estimate the likelihood of each post-particle.

The likelihood estimation unit 139 compares the positions of some of the current frame feature points corresponding to some of the previous frame feature points selected by the selection unit 138 with the positions of the plurality of post-transition particles. The likelihood of each particle after the transition may be estimated. The predetermined object transition model is a motion model that models the characteristics of the motion of the object, and is, for example, a constant velocity linear motion model, an autoregressive model, or the like. The likelihood estimation unit 139 inputs the likelihood of each of the estimated plurality of post-transition particles to the tracking unit 140.

The tracking unit 140 sets an object included in the moving image based on some of the post-transition particles selected from the plurality of post-transition particles based on the likelihood of each of the plurality of post-transition particles estimated by the likelihood estimation unit 139. Chase. Specifically, the tracking unit 140 first selects some post-transition particles from the plurality of post-transition particles based on the magnitude of the likelihood of each of the plurality of post-transition particles estimated by the likelihood estimation unit 139. .. The tracking unit 140 selects a predetermined number of post-transition particles in descending order of the likelihood of each of the plurality of post-transition particles estimated by the likelihood estimation unit 139. A predetermined quantity is set in the object tracking device 1 in advance. Then, the tracking unit 140 estimates the position of the object included in the current frame image based on the parameter calculated by the weighted average of some of the selected post-transition particles. The parameters are, for example, coordinate values indicating the positions of objects included in the current frame image.

The tracking unit 140 uses a plurality of tracking units 140 to track an object included in a frame image after the current frame image based on the magnitude of the likelihood of each of some post-transition particles selected from the plurality of post-transition particles. Select pre-transition particles. Specifically, the tracking unit 140 duplicates the post-transition particles having a high likelihood and erases the post-transition particles having a low likelihood. The tracking unit 140 may select a predetermined number of post-transition particles based on the magnitude of the likelihood of each of some post-transition particles selected from the plurality of post-transition particles.

[Extraction process]
Subsequently, the extraction process performed by the extraction unit 133 will be described. As described above, the extraction unit 133 has a rear-stage extraction unit 134 and a front-stage extraction unit 135. The post-stage extraction unit 134 outputs a plurality of post-stage current frames and pre-frame images output from the post-stage processing layer because the current frame image propagates in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers. Is propagated in the order of the pre-processing layer and the post-processing layer, so that one or more post-stage current frame outputs and one or more post-stage pre-frame outputs that are commonly activated from the plurality of post-stage pre-frame outputs output from the post-stage processing layer. Is extracted.

The pre-stage extraction unit 135 is output from a plurality of pre-stage current frame outputs output from the pre-stage processing layer and output from the pre-stage processing layer, which are factors that activate one or more post-stage current frame outputs and one or more post-stage front frame outputs. Of the plurality of previous stage previous frame outputs, one or more previous stage current frame outputs and one or more previous stage previous frame outputs that are commonly activated are extracted.

4 to 9 are diagrams for explaining the extraction process performed by the extraction unit 133. 4 to 9 show a state of propagation from the pre-stage processing layer to the post-stage processing layer. In FIGS. 4 to 9, the connecting line connecting the units shown by the solid line indicates that there was an output from the connecting unit, and the connecting line shown by the broken line indicates that there was no output from the connecting unit. The thickness of the line indicating the connecting line indicates the magnitude of the output from the unit to be connected.

In the case of FIG. 4, the post-stage treatment layer is a treatment layer (for example, an output layer or a fully connected layer) or a treatment layer before the last layer selected by the extraction unit 133 (such as a fully connected layer or a pooling layer). Yes, the pre-stage treatment layer is a treatment layer immediately before the post-stage treatment layer (for example, a fully bonded layer or a pooling layer). In FIG. 4, it is assumed that the post-stage processing layer is the output layer M20 and the pre-stage processing layer is the fully connected layer M19.

FIG. 4A is a state before extraction, and FIG. 4B is a state after extraction. In the current frame image, units U5 and U8 are activated in the output layer M20, and units U2, U5, U6, U7 and U8 are activated in the fully connected layer M19. In the previous frame image, units U3 and U5 are activated in the output layer M20, and units U2, U4, U5 and U8 are activated in the fully connected layer M19.

In this case, the post-stage extraction unit 134 includes units U5 and U8 which are the post-stage current frame outputs output from the output layer M20 which is the post-stage processing layer, and units U3 which are the post-stage front frame outputs output from the output layer M20. Compare U5. Then, the rear stage extraction unit 134 extracts the unit U5 of the rear stage current frame output and the unit U5 of the rear stage front frame output that are commonly activated.

Subsequently, the pre-stage extraction unit 135 includes units U2, U5, U6, which are the pre-stage current frame outputs output from the fully connected layer M19, which is the pre-stage processing layer, which is a factor that activates the unit U5 of the rear-stage current frame output. And the units U2, U5, and U8, which are the front frame outputs of the front stage, which are the factors that activate the unit U5 of the front frame output of the rear stage, are compared. Then, the pre-stage extraction unit 135 extracts the units U2 and U5 of the pre-stage current frame output and the units U2 and U5 of the pre-stage front frame output that are commonly activated.

When the extraction unit 133 extracts the output from the output layer M20 to the fully connected layer M19, the extraction unit 133 extracts the output for the next processing layer. Specifically, the extraction unit 133 repeats the process of extracting the current frame output and the previous frame output that are commonly activated for each processing layer in the order opposite to the order propagated by the propagation control unit 132. conduct. More specifically, the extraction unit 133 selects one of the plurality of processing layers as the post-processing layer, extracts one or more current frame outputs and one or more pre-frame outputs, and then uses the extraction unit 133 as the pre-processing layer. The selected processing layer is selected as the subsequent processing layer, and another one or more current frame outputs and one or more previous frame outputs are extracted. By doing so, the extraction unit 133 can improve the accuracy of comparison with the current frame image and the previous frame image.

FIG. 5 shows a state in which the image is propagated from the pre-stage processing layer to the post-stage processing layer based on the current frame image. FIG. 6 shows a state in which the image is propagated from the pre-stage processing layer to the post-stage processing layer based on the pre-frame image. In the case of FIGS. 5 and 6, the post-stage treatment layer is a fully-bonded layer M18, and the front-stage treatment layer is a treatment layer other than the fully-bonded layer (for example, a pooling layer or a convolution layer). In FIGS. 5 and 6, the pretreatment layer will be described as the pooling layer M17. Further, in FIGS. 5 and 6, the pretreatment layer has three channels. The first channel in the upper row includes units U11, U12, U13, U14, and U15. The second channel in the middle stage includes units U21, U22, U23, U24, and U25. The lower third channel includes units U31, U32, U33, U34, and U35.

In the current frame image, in the pooling layer M17, the units U13 included in the first channel and the units U21 and U24 included in the second channel are activated. In the front frame image, in the fully connected layer M18, the units U22, U24, U25 included in the second channel and the units U32, U33 included in the third channel are activated.

The front-stage extraction unit 135 activates the front-stage current frame output output from the pooling layer M17, which is the front-stage processing layer, and the unit U5 of the rear-stage front frame output, which are factors that activate the unit U5 of the rear-stage current frame output. The previous frame output from the pooling layer M17, which is a factor, is compared. The pre-stage extraction unit 135 checks for the presence or absence of an activated unit, and unit U13 included in the first channel of the activated pre-stage current frame output, U21 and U24 included in the second channel, and the pre-stage pre-frame output. Focus on the units U22, U24, U25 included in the second channel and U32, U33 included in the third channel.

Then, since the channel in which the unit activated in both the previous stage current frame output and the previous stage previous frame output exists is the second channel, the front stage extraction unit 135 is set to the second channel of the previous stage current frame output. The units U21, U24 included and the units U22, U24, U25 included in the second channel of the front frame output of the previous stage are extracted.

In the case of FIG. 7, the post-stage treatment layer is the pooling layer M16, and the front-stage treatment layer is a treatment layer other than the pooling layer (for example, a convolution layer or a normalized layer). In FIG. 7, the pre-stage processing layer will be described as the convolution layer M15. Further, in FIG. 7, the pre-stage processing layer will be described as having one channel. In the current frame image, the pooling layer M16 has the unit U5 activated, and the convolution layer M15 has the units U3 and U5 activated. In the pre-frame image, the pooling layer M16 has the unit U3 activated, and the convolution layer M15 has the units U3 and U4 activated.

Here, in the pooling layer that compresses the image, the extraction unit 133 extracts the output based on the degree of activation for each channel among the plurality of units bonded to the pooling layer from the immediately preceding processing layer. do. Specifically, the pre-stage extraction unit 135 has one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs based on the activated size of the plurality of pre-stage current frame outputs and the plurality of pre-stage pre-frame outputs. Extract the frame output. The pre-stage extraction unit 135 extracts, for example, one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs that are most activated for each channel among the plurality of pre-stage current frame outputs and the plurality of pre-stage pre-frame outputs. do.

In this case, the post-stage extraction unit 134 selects the pooling layer M16 selected as the pre-stage processing layer in the immediately preceding extraction process, and outputs from the post-stage current frame output unit U5 and the pooling layer M16 output from the pooling layer M16. The unit U3 of the post-stage front frame output is extracted. Then, the front-stage extraction unit 135 has the front-stage current frame output unit U5 and the front-stage front, which are the most activated for each channel among the rear-stage current frame output units U3 and U5 and the rear-stage front frame output units U3 and U4. The frame output unit U4 is extracted. By doing so, the pre-stage extraction unit 135 can specify a characteristic region in the image.

In the case of FIG. 8, the post-stage treatment layer is a convolution layer M14, and the front-stage treatment layer is another treatment layer including the convolution layer (for example, a normalized layer or a pooling layer). In FIG. 8, the pre-stage processing layer will be described as the normalization layer M13. Further, in FIG. 8, the pre-stage processing layer will be described as having one channel. In the current frame image, the convolution layer M14 has the unit U5 activated, and the normalized layer M13 has the units U3, U5, and U6 activated. In the pre-frame image, the convolution layer M14 has the unit U3 activated, and the normalized layer M13 has the units U3, U4, and U5 activated.

In this case, the post-stage extraction unit 134 selects the convolution layer M14 selected as the pre-stage processing layer in the immediately preceding extraction process, and outputs from the post-stage current frame output unit U5 and the convolution layer M14 output from the convolution layer M14. The unit U3 of the post-stage front frame output is extracted.

Subsequently, the front-stage extraction unit 135 includes the front-stage current frame output output from the normalization layer M13, which is the front-stage processing layer, which is a factor that activates the unit U5 of the rear-stage current frame output, and the unit U4 of the rear-stage front frame output. The pre-stage pre-frame output output from the normalization layer M13, which is the pre-stage processing layer that has become a factor for activating the above, is compared. Here, when the post-stage processing layer is a convolutional layer, the pre-stage extraction unit 135 has the pre-stage current frame output and the pre-stage of the plurality of units of the pre-stage processing layer that are coupled to the units extracted from the post-stage processing layer by the post-stage extraction unit 134. Extract units that are relatively the same position as the previous frame output and have the same channel. In this case, the pre-stage extraction unit 135 has the units U5, U6, and the pre-stage front of the pre-stage current frame output as units having relatively the same position in the pre-stage current frame output and the pre-stage front frame output and having a common channel. The frame output units U3 and U4 are extracted.

In the case of FIG. 9, the post-stage treatment layer is the normalization layer M12, and the front-stage treatment layer is a treatment layer other than the normalization layer (for example, a convolution layer or a pooling layer). In FIG. 9, the pretreatment layer will be described as the pooling layer M11. Further, in FIG. 9, the pre-stage processing layer will be described as having one channel. In the current frame image, the unit U5 is activated in the normalized layer M12. In the pre-frame image, the normalized layer M12 has the unit U3 activated.

Here, in the normalization layer that preprocesses the image, the extraction unit 133 is the center of the plurality of units included in the pretreatment layer that is bound to the unit that is activated in the post-processing layer. Extract the unit. In this case, the post-stage extraction unit 134 extracts the unit U5 of the post-stage current frame output output from the normalization layer M12 selected as the post-stage processing layer and the unit U3 of the post-stage front frame output output from the normalization layer M12. do.

Then, the front-stage extraction unit 135 extracts the central unit U5 from the units of the pooling layer M11 coupled to the unit U5 of the rear-stage current frame output output from the normalization layer M5. Similarly, the front-stage extraction unit 135 extracts the central unit U3 from the units of the pooling layer M11 coupled to the unit U3 of the rear-stage front frame output output from the normalization layer M12.

It is preferable that the extraction unit 133 repeats the above-mentioned extraction process up to the input layer. However, the extraction unit 133 may end the extraction process at an intermediate process layer (for example, a pooling layer, a normalization layer, or the like) without performing the extraction process up to the first process layer. In this way, the extraction unit 133 can extract an output having a high degree of abstraction by performing the extraction process in the order opposite to the order in which the propagation control unit 132 propagates.

[Object tracking process]
Subsequently, the object tracking process will be described. FIG. 10 is a diagram for explaining the details of the object tracking process. The white circles shown in FIG. 10 are particles generated by the likelihood estimation unit 139 or the tracking unit 140, and the black circles are the feature points detected by the feature point detection unit 137. The positions of the white circles and the black circles indicate the positional relationship between the white circles and the black circles located in the frame image. For example, when the feature point F1 of the front frame is used as a reference, the particles P2 are the closest and the particles P3 are the farthest.

First, the likelihood estimation unit 139 generates a plurality of pre-transition particles in association with the pre-frame image ((1) in FIG. 10). As shown in FIG. 10, the likelihood estimation unit 139 generates pre-transition particles P1, P2, and P3 in the pre-frame image. Specifically, the likelihood estimation unit 139 arranges the pre-transition particles P1, P2, and P3 in the pre-frame image when the pre-transition particles P1, P2, and P3 selected by the tracking unit 140 are present. On the other hand, the likelihood estimation unit 139 generates pre-transition particles P1, P2, and P3 and arranges them in the previous frame image when the particles generated by the tracking unit 140 do not exist, that is, when the likelihood is estimated first. ..

Subsequently, the likelihood estimation unit 139 predicts the movement of the object by making the arranged plurality of pre-transition particles transition according to a predetermined object transition model ((2) in FIG. 10). The likelihood estimation unit 139 shifts the pre-transition particles P1, P2, and P3 to the positions of the post-transition particles P4, P5, and P6, for example, according to a constant velocity linear motion model.

Subsequently, the likelihood estimation unit 139 compares the positions of one or more current frame feature points corresponding to one or more previous frame feature points with the positions of the plurality of post-transition particles after the transition. The likelihood of each of the plurality of post-transition particles is estimated ((3) in FIG. 10). Specifically, the likelihood estimation unit 139 estimates that the likelihood is increased for post-transition particles close to the positions of one or more current frame feature points corresponding to one or more previous frame feature points. The likelihood estimation unit 139 has a likelihood magnitude of “P5> P4> with respect to the post-transition particles P4, P5, and P6 in the order of proximity to the current frame feature point F2 corresponding to the previous frame feature point F1. It is estimated to be "P6".

The likelihood estimation unit 139 compares the positions of one or more current frame feature points with the positions of a plurality of post-transition particles regardless of whether or not they correspond to one or more previous frame feature points. , The likelihood of each of the plurality of post-transition particles may be estimated. The likelihood estimation unit 139 may estimate, for example, the closeness between the post-transition particles and the current frame feature points closest to the post-transition particles among a plurality of current frame feature points in the current frame image as the likelihood. .. In this case, the likelihood estimation unit 139 may estimate the reciprocal of the distance between the position of the current frame feature point and the position of the post-transition particles as the likelihood. Further, the likelihood estimation unit 139 may estimate, for example, the average closeness between the post-transition particles and one or more current frame feature points as the likelihood. Further, the likelihood estimation unit 139 determines the closeness between the post-transition particles and the current frame feature points corresponding to the nearest previous frame feature points of the pre-transition particles corresponding to the post-transition particles. May be.

By the way, when the likelihood estimation unit 139 first estimates the likelihood, there are no plurality of post-transition particles whose likelihood is estimated as a reference for arranging the plurality of pre-transition particles. Therefore, when the likelihood estimation unit 139 first estimates the likelihood, the likelihood estimation unit 139 generates a plurality of pre-transition particles randomly arranged in the pre-frame image, and the one or more pre-transition particles corresponding to one or more pre-frame feature points. By comparing the positions of the current frame feature points with the positions of the plurality of post-transition particles, the likelihood of each of the plurality of post-transition particles is estimated.

By randomly arranging the initial pre-transition particles in the entire front frame image in this way, the tracking unit 140 can estimate the position of the object based on the likelihood estimated by the likelihood estimation unit 139. However, when the pre-transition particles are randomly arranged in the entire previous frame image, the proportion of the pre-transition particles arranged at the position of the object becomes small, and the likelihood estimation unit 139 may deteriorate the accuracy of estimating the likelihood. There is. Therefore, the likelihood estimation unit 139 determines the positions of one or more current frame feature points corresponding to one or more previous frame feature points and one or more pre-transition particles using one or more previous frame feature points as particles. The likelihood of each of one or more post-transition particles may be estimated by comparing the positions of the post-transition particles that have been transitioned according to the object transition model of.

For example, the likelihood estimation unit 139 transitions the position of the current frame feature point F2 corresponding to the previous frame feature point F1 and the pre-transition particles using the previous frame feature point F1 as particles according to a predetermined object transition model. The likelihood of each of one or more post-transition particles may be estimated by comparing the positions of the post-transition particles. By using feature points based on features with a high degree of abstraction as the initial pre-transition particles, the proportion of pre-transition particles placed at the position of the object is higher than when the pre-transition particles are randomly placed. The number increases, and the likelihood estimation unit 139 can improve the accuracy of estimating the likelihood.

Subsequently, the tracking unit 140 sets the current frame image based on some of the post-transition particles selected from the plurality of post-transition particles based on the likelihood of each of the plurality of post-transition particles estimated by the likelihood estimation unit 139. The position of the contained object is estimated ((4) in FIG. 10). The tracking unit 140 selects a predetermined number of post-transition particles in the order of increasing likelihood of post-transition particles estimated by the likelihood estimation unit 139, for example, “P5> P4> P6”. For example, when the predetermined quantity is two, the tracking unit 140 selects the post-transition particles P4 and P5 in descending order of the likelihood of the post-transition particles estimated by the likelihood estimation unit 139.

Then, the tracking unit 140 estimates the position of the object included in the current frame image based on the parameters calculated by the weighted average of some of the selected post-transition particles P4 and P5. For example, the tracking unit 140 may indicate the estimated position of the object with a square frame in the current frame image.

Subsequently, the tracking unit 140 tracks an object included in the frame image after the current frame image based on the magnitude of the likelihood of each of some of the post-transition particles selected from the plurality of post-transition particles. A plurality of pre-transition particles to be used are selected ((5) in FIG. 10). The tracking unit 140 duplicates the post-transition particles P5 having a high likelihood, such as the post-transition particles P5, P7, and P8, and erases the post-transition particles P6 having a low likelihood. In this way, the object tracking device 1 tracks the object included in the moving image by repeating the object tracking process.

[Processing of object tracking device 1]
Subsequently, the flow of processing performed by the object tracking device 1 will be described. FIG. 11 is a flowchart showing the flow of processing performed by the object tracking device 1. This flowchart starts when the object tracking device 1 receives an operation of storing a moving image in the storage unit 12 and executing a process of compressing the moving image by the user.

The moving image acquisition unit 131 acquires the moving image stored in the storage unit 12 (S1). The moving image acquisition unit 131 inputs the acquired moving image to the propagation control unit 132. The propagation control unit 132 transfers each of the current frame image and the previous frame image, which are frame images included in the moving image input from the moving image acquisition unit 131, from the input layer M1 included in the machine learning model M to the output layer M10. It is propagated to the plurality of processing layers in order from the input layer M1 (S2).

The extraction unit 133 performs a process of extracting one or more current frame outputs and one or more front frame outputs that are commonly activated in both the processing layers of the post-stage processing layer and the front-stage processing layer (S3). FIG. 12 is a flowchart showing the flow of processing performed by the extraction unit 133. The extraction unit 133 determines whether or not the instruction receiving unit 136 has received an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 11 (S31).

When the extraction unit 133 determines that the instruction receiving unit 136 has received the instruction (YES in S31), the extraction unit 133 selects the processing layer indicated by the instruction received by the instruction receiving unit 136 as the subsequent processing layer (S32). When, for example, the extraction unit 133 determines that the instruction receiving unit 136 has received the instruction indicating the first fully connected layer M8, the extraction unit 133 processes the first fully connected layer M8 indicated by the instruction received by the instruction receiving unit 136 in a subsequent stage. Used as a layer. On the other hand, when the extraction unit 133 determines that the instruction reception unit 136 is not accepting the instruction (NO in S31), one or more presents that are commonly activated in the rearmost layer (for example, the output layer M10). It is determined whether or not there is a frame output and one or more previous frame outputs (S33).

When the extraction unit 133 determines that there is one or more current frame outputs and one or more previous frame outputs that are commonly activated in the output layer M10 (YES in S33), the output layer M10 which is the last layer. Is used as a post-treatment layer (S34). On the other hand, when the extraction unit 133 determines that there is no one or more current frame outputs and one or more previous frame outputs commonly activated in the output layer M10 (NO in S33), the extraction unit 133 is before the output layer M10. For each processing layer, one or more current frame outputs and one or more previous frame outputs that are commonly activated are repeatedly searched. Then, the extraction unit 133 uses a processing layer having one or more current frame outputs and one or more front frame outputs (for example, the second fully connected layer M9) that are commonly activated as the subsequent processing layer (for example, the second fully connected layer M9). S35). The extraction unit 133 is one or more current frames output from the post-stage processing layer and the pre-stage processing layer based on the current frame image, which are commonly activated in both the selected post-stage processing layer and the pre-stage processing layer. Based on the output and the pre-frame image, the post-stage processing layer and one or more pre-frame outputs output from the pre-stage processing layer are extracted.

Specifically, first, the post-stage extraction unit 134 is commonly activated from a plurality of post-stage current frame outputs output from the selected post-stage processing layer and a plurality of post-stage pre-frame outputs output from the selected post-stage processing layer. One or more subsequent-stage current frame outputs and one or more subsequent-stage front frame outputs that have been converted are extracted (S36). Then, the front-stage extraction unit 135 has a plurality of front-stage current frames output from the front-stage processing layer that has been a factor in activating one or more rear-stage current frame outputs and one or more rear-stage front-frame outputs extracted by the rear-stage extraction unit 134. From the output and the plurality of pre-stage pre-frame outputs output from the pre-stage processing layer, one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs that are commonly activated are extracted (S37).

Subsequently, the extraction unit 133 determines whether or not there is another processing layer before the pre-stage processing layer (S38). When the extraction unit 133 determines that there is another treatment layer (for example, the third convolution layer M6) before the pre-stage treatment layer (for example, the second pooling layer M7) (YES in S38), the first The pooling layer M7 of 2 is selected as the post-treatment layer (S39), and the treatment is returned to S36. On the other hand, when the extraction unit 133 determines that there is no other processing layer before the pre-stage processing layer (for example, the input layer M1) (NO in S38), the extracted one or more current frame outputs and one or more current frame outputs. The previous frame output is input to the feature point detection unit 137, and the extraction process is completed.

Returning to FIG. 11, the feature point detection unit 137 searches for a corresponding feature point based on one or more current frame outputs and one or more previous frame outputs, and one based on one or more current frame outputs having a corresponding relationship. The above current frame feature points and one or more previous frame feature points based on one or more previous frame outputs are detected (S4). Subsequently, the selection unit 138 determines whether or not there are inappropriate feature points in the current frame feature point and the previous frame feature point detected by the feature point detection unit 137 (S5). The selection unit 138 narrows down based on, for example, the RANSAC method.

When the selection unit 138 determines that the current frame feature point and the previous frame feature point have an inappropriate feature point (YES in S5), the selection unit 138 has an inappropriate feature point, that is, an erroneously detected current corresponding relationship. The frame feature points and the front frame feature points are removed (S6), and some current frame feature points and some front frame feature points based on the correspondence after the removal are selected.

The likelihood estimation unit 139 removes the correspondence when the selection unit 138 determines that there are no inappropriate feature points in the current frame feature point and the previous frame feature point (NO in S5) or erroneously detected. Later, it is determined whether or not there are a plurality of pre-transition particles generated by the tracking unit 140 (S7). When the likelihood estimation unit 139 determines that there are no plurality of pre-transition particles generated by the tracking unit 140 (NO in S7), the likelihood estimation unit 139 generates a plurality of pre-transition particles (S8). The likelihood estimation unit 139 may use one or more previous frame feature points as pre-transition particles.

When the likelihood estimation unit 139 determines that there are a plurality of pre-transition particles generated by the tracking unit 140 (YES in S7), or after generating a plurality of pre-transition particles, a plurality of pre-transition particles are generated in the previous frame image. Pre-transition particles are placed (S9). For example, in the plurality of pre-transition particles generated by the tracking unit 140, the likelihood estimation unit 139 arranges the pre-transition particles in the front frame image with reference to the position of the post-transition particles selected by the tracking unit 140. On the other hand, the likelihood estimation unit 139 arranges the pre-transition particles at the positions of one or more pre-transition feature points in the generated plurality of pre-transition particles.

Subsequently, the likelihood estimation unit 139 transitions a plurality of pre-transition particles arranged in the pre-frame image in the current frame image according to a predetermined object transition model (S10). The likelihood estimation unit 139 transitions a plurality of pre-transition particles arranged in the pre-frame image according to, for example, a constant velocity linear motion model. Subsequently, the likelihood estimation unit 139 compares the positions of one or more current frame feature points corresponding to one or more previous frame feature points with the positions of the plurality of post-transition particles after the transition. The likelihood of each of the plurality of post-transition particles is estimated (S11).

Subsequently, the tracking unit 140 sets the current frame image based on some of the post-transition particles selected from the plurality of post-transition particles based on the likelihood of each of the plurality of post-transition particles estimated by the likelihood estimation unit 139. The position of the contained object is estimated (S12). After that, the tracking unit 140 is used to track an object included in the frame image after the current frame image based on the magnitude of the likelihood of each of some post-transition particles selected from the plurality of post-transition particles. A plurality of pre-transition particles are selected (S13).

[Effect in Embodiment]
As described above, the object tracking device 1 propagates a plurality of processing layers included in the machine learning model M to each of the current frame image and the previous frame image included in the acquired moving image. The object tracking device 1 outputs one or more current frame outputs and one or more pre-frame outputs that are commonly activated in both the processing layers of the post-stage processing layer and the pre-stage processing layer in the reverse order of the propagation order. Extraction is performed for each processing layer, and the corresponding current frame feature points and previous frame feature points are detected. Then, the object tracking device 1 moves the moving image based on the likelihood estimated by comparing the position of the current frame feature point corresponding to the previous frame feature point with the position of the post-transition particle that predicted the movement of the object. Track the objects contained in.

By doing so, the object tracking device 1 estimates the movement of the object between the frame images by using the machine learning model M including the convolutional neural network and obtaining the features with a high degree of abstraction by deep learning. be able to. As a result, the object tracking device 1 can improve the accuracy of tracking the object included in the moving image.

The object tracking device 1 can track individual customers in, for example, in-store behavior analysis. By properly designing the system model of the particle filter, the object tracking device 1 can separate and track individual customers even when a plurality of customers are mixed. In addition, the object tracking device 1 can track individual soccer players in, for example, a soccer game. The object tracking device 1 can separate and track individual soccer players by properly designing the system model of the particle filter.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist thereof. be. For example, the specific embodiment of the distribution / integration of the device is not limited to the above embodiment, and all or a part thereof may be functionally or physically distributed / integrated in any unit. Can be done. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination has the effect of the original embodiment together.

1 Object tracking device 11 Operation unit 12 Storage unit 13 Control unit 131 Moving image acquisition unit 132 Propagation control unit 133 Extraction unit 134 Post-stage extraction unit 135 Front-stage extraction unit 136 Instruction reception unit 137 Feature point detection unit 138 Selection unit 139 Likelihood estimation unit 140 Tracking Department

Claims (16)

A moving image acquisition unit that acquires moving images generated by an imaging device,
A front frame image which is a frame image included in the moving image and a current frame image which is a frame image after the previous frame image included in the moving image are added to the moving image based on the input moving image. A processing layer included in a machine learning model that can output the types of included objects, and a propagation control unit that propagates multiple processing layers including multiple units.
A frame output indicating the unit that is commonly activated in both the post-stage treatment layer selected from the plurality of treatment layers and the pre-stage treatment layer that is the treatment layer immediately before the post-stage treatment layer. one or more of the previous frame is the frame output indicating the unit based on the basis of the frame image is the frame output indicating the unit is activated one or more current frame output to the previous frame image is activated An extractor that extracts the output and
A feature point detection unit that detects one or more current frame feature points based on the one or more current frame outputs and detects one or more previous frame feature points based on the one or more previous frame outputs.
After a plurality of transitions in which a plurality of pre-transition particles are generated in association with the pre-frame image, the positions of the one or more current frame feature points and the plurality of pre-transition particles are transitioned according to a predetermined object transition model. A likelihood estimation unit that estimates the likelihood of each of the plurality of post-transition particles by comparing the positions of the particles,
A tracking unit that tracks an object included in the moving image based on some of the post-transition particles selected from the plurality of post-transition particles based on the likelihood estimated by the likelihood estimation unit.
Object tracking device with. The extraction unit
A plurality of subsequent stages of the frame output indicating the unit activated in the subsequent stage processing layer by propagating the current frame image in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. Commonly from the frame output and the plurality of post-stage front frame outputs which are the frame outputs indicating the unit activated in the post-stage processing layer by propagating the pre-frame image in the order of the pre-stage processing layer and the post-stage processing layer. A post-stage extraction unit that extracts one or more active post-stage current frame outputs and one or more post-stage front frame outputs,
A plurality of pre-stage current frame outputs, which are the frame outputs indicating the unit activated in the pre-stage processing layer that has been a factor in activating the one or more post-stage current frame outputs and the one or more post-stage pre-frame outputs. And one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs that are commonly activated among the plurality of pre-stage pre-frame outputs that are the frame outputs indicating the unit that is activated in the pre-stage processing layer. Pre-stage extraction unit to extract and
Have,
The object tracking device according to claim 1. The pre-stage extraction unit has one or more pre-stage current frame outputs and one or more pre-stage pre-frames based on the activated size of the plurality of pre-stage current frame outputs and the plurality of pre-stage pre-frame outputs. Extract the output,
The object tracking device according to claim 2. The machine learning model includes a convolutional neural network.
The post-treatment layer is any one of an output layer, a fully connected layer, a normalized layer, a pooling layer, and a convolution layer.
The object tracking device according to claim 2 or 3. The pretreatment layer is any one of a fully bonded layer, a normalized layer, a pooling layer, a convolution layer, and an input layer.
The object tracking device according to claim 4. The extraction unit indicates the unit that is commonly activated in the last layer when the last layer, which is the last treatment layer, is selected as the latter treatment layer among the plurality of treatment layers. When there is no one or more current frame outputs and one or more previous frame outputs, which are outputs, the one or more which is the frame output indicating the unit which is commonly activated in the processing layer before the last layer. Extracts the current frame output of and one or more of the previous frame outputs.
The object tracking device according to any one of claims 1 to 5. The likelihood estimation unit estimates that the closer the post-transition particles are to the positions of one or more current frame feature points, the higher the likelihood.
The object tracking device according to any one of claims 1 to 6. The likelihood estimation unit transitions the positions of the one or more current frame feature points and one or more pre-transition particles using the one or more previous frame feature points as particles according to the predetermined object transition model. The likelihood of each of the one or more post-transition particles is estimated by comparing the positions of the post-transition particles.
The object tracking device according to any one of claims 1 to 7. In the case of estimating the likelihood, the likelihood estimation unit generates the plurality of pre-transition particles arranged in the front frame image based on a rule different from the case of estimating the likelihood from the next time onward, and the one or more. By comparing the position of the current frame feature point of the current frame with the position of the plurality of post-transition particles, the likelihood of each of the plurality of post-transition particles is estimated.
The object tracking device according to any one of claims 1 to 8. The likelihood estimation unit compares the positions of the one or more current frame feature points corresponding to the one or more previous frame feature points with the positions of the plurality of post-transition particles, thereby performing the plurality of post-transition particles. Estimate the likelihood of each particle,
The object tracking device according to any one of claims 1 to 9. Further, it has an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers.
The extraction unit uses the processing layer indicated by the instruction received by the instruction receiving unit as the subsequent processing layer.
The object tracking device according to any one of claims 1 to 10. The extraction unit selects one of the plurality of processing layers as the post-stage processing layer, extracts the one or more current frame outputs and the one or more pre-frame outputs, and then selects the pre-stage processing layer. The processed layer is selected as the subsequent processing layer, and another one or more current frame outputs and one or more previous frame outputs are extracted.
The object tracking device according to any one of claims 1 to 11. The steps to acquire the moving image generated by the image pickup device and
A front frame image which is a frame image included in the moving image and a current frame image which is a frame image after the previous frame image included in the moving image are added to the moving image based on the input moving image. A processing layer included in a machine learning model that can output the type of the included object, and a step of propagating a plurality of processing layers including a plurality of units, respectively.
A frame output indicating the unit that is commonly activated in both the post-stage treatment layer selected from the plurality of treatment layers and the pre-stage treatment layer that is the treatment layer immediately before the post-stage treatment layer. one or more of the previous frame is the frame output indicating the unit based on the basis of the frame image is the frame output indicating the unit is activated one or more current frame output to the previous frame image is activated Steps to extract the output and
A step of detecting one or more current frame feature points based on the one or more current frame outputs and detecting one or more previous frame feature points based on the one or more previous frame outputs.
A step of generating a plurality of pre-transition particles in association with the pre-frame image,
By comparing the positions of the one or more current frame feature points with the positions of the plurality of post-transition particles obtained by transitioning the plurality of pre-transition particles according to a predetermined object transition model, the plurality of post-transition particles. Steps to estimate each likelihood and
A step of tracking an object included in the moving image based on some of the post-transition particles selected from the plurality of post-transition particles based on the estimated likelihood.
Object tracking method with. The extraction step
A plurality of subsequent stages of the frame output indicating the unit activated in the subsequent stage processing layer by propagating the current frame image in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. Commonly from the frame output and the plurality of post-stage front frame outputs which are the frame outputs indicating the unit activated in the post-stage processing layer by propagating the pre-frame image in the order of the pre-stage processing layer and the post-stage processing layer. A pre-stage extraction step that extracts one or more activated current frame outputs and one or more post-stage pre-frame outputs,
A plurality of pre-stage current frame outputs, which are the frame outputs indicating the unit activated in the pre-stage processing layer that has been a factor in activating the one or more post-stage current frame outputs and the one or more post-stage pre-frame outputs. And one or more pre-stage current frame outputs and one or more pre-stage pre-frame outputs that are commonly activated among the plurality of pre-stage pre-frame outputs that are the frame outputs indicating the unit that is activated in the pre-stage processing layer. The latter extraction step to extract and
13. The object tracking method according to claim 13. After executing the pre-stage extraction step, the post-stage extraction step is performed by using the one or more pre-stage current frame outputs and the one or more pre-stage front frame outputs as the plurality of post-stage current frame outputs and the plurality of post-stage front frame outputs. Run,
The object tracking method according to claim 14. The latter-stage extraction step and the first-stage extraction step are executed for each of the plurality of processing layers.
The object tracking method according to claim 14 or 15.

Images