JP7316236B2 - Skeletal tracking method, device and program - Google Patents

Description

The present invention relates to a skeleton tracking method, apparatus and program, and more particularly to a skeleton tracking method, apparatus and program for estimating a skeleton by tracking joint points of a person on time-series images.

Skeletal tracking is a technique for estimating a person's skeleton and its movements in time-series images. Especially, when it is assumed that multiple people exist in time-series images, there are roughly two approaches. .

The first approach, as disclosed in Non-Patent Documents 1 and 2, detects the joint points of a person in each frame and optimizes the intra-frame and inter-frame joint points for all detected joint points in multiple frames. It is a bottom-up approach that performs joint point tracking and skeleton estimation simultaneously by finding connections.

Non-Patent Document 1 describes (1) joint point detection, (2) confidence estimation (joint point adjacent confidence and joint point tracking confidence), and (3) confidence base A technique is disclosed for outputting the result of skeletal tracking after performing tracking estimation in .

The joint point adjacency confidence is the confidence that each joint point in the frame is a joint point that is directly connected by a limb or the like, such as a wrist and an elbow. The joint point tracking confidence is the confidence that each joint point between frames is the same joint point. Confidence-based tracking estimation considers a graph in which all the obtained joint points are nodes and the estimated confidence is the edge weight. to estimate

In Non-Patent Document 2, skeleton tracking is estimated each time a frame image is read, and joint points between two frames are tracked more effectively. That is, the joint points are detected from the frame of interest, then the confidence that the joint points in the frame of interest are adjacent to each other is estimated, and the tracking confidence is estimated from the joint points in the immediately preceding frame. Each degree of certainty is obtained from a joint point presence probability map, a joint point adjacent direction vector map (PAF), and a joint point tracking direction vector map (TAF) output by a model obtained by learning.

In particular, the TAF includes a motion representation of the joint points and is calculated based on the previous and current PAFs and the previous TAF. In Non-Patent Document 2, by using the previous TAF, the current motion can be estimated based on the past motion. Finally, we consider a graph in which the joint points of the frame of interest and the previous frame are nodes, and the estimated confidence is the weight of each edge. presume.

The second approach, as disclosed in Non-Patent Document 3, is a top-down approach that performs person detection and skeleton estimation in each frame, followed by tracking for each person.

In Non-Patent Document 3, skeleton estimation is performed for each bounding box obtained by human detection in each frame, and the positions of the obtained joint points in the next frame are estimated by optical flow. Tracking is then performed by associating persons whose estimated positions are closest to joint point positions actually obtained in the next frame.

In this approach, skeleton estimation in each frame is essential, and Patent Document 1 discloses a method for robust skeleton estimation even in a complex scene in which various objects are included in an image by setting a skeleton model in advance. ing.

JP 2017-091377 A

Iqbal, Umar, Anton Milan, and Juergen Gall. "Posetrack: Joint multi-person pose estimation and tracking." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. Raaj, Yaadhav, et al. "Efficient Online Multi-Person 2D Pose Tracking with Recurrent Spatio-Temporal Affinity Fields." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. Xiao, Bin, Haiping Wu, and Yichen Wei. "Simple baselines for human pose estimation and tracking." Proceedings of the European Conference on Computer Vision (ECCV). 2018. Cao, Zhe, et al. "OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields." arXiv preprint arXiv:1812.08008 (2018). He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. Doering, Andreas, Umar Iqbal, and Juergen Gall. "Joint flow: Temporal flow fields for multi person tracking." arXiv preprint arXiv:1805.04596 (2018). Bewley, Alex, et al. "Simple online and realtime tracking." 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 2016.

There is a concern that the top-down approach to which Non-Patent Document 3 belongs increases the computational cost in proportion to the number of people detected in each frame. In addition, it has been confirmed that human detection and skeleton estimation fail in difficult situations such as when multiple people are close to each other. Patent Document 1, which is a skeleton estimation method, does not have a mechanism for solving the above-described situation in which skeleton estimation is difficult, and does not consider tracking processing in the first place.

In the bottom-up approaches disclosed in Non-Patent Documents 1 and 2, skeleton estimation and tracking of all detected persons are simultaneously estimated, so an increase in the number of persons has relatively little effect on processing time. Furthermore, since optimal solutions are sought for both skeleton estimation and tracking, there is room for reducing the accuracy degradation of one by the other.

However, Non-Patent Document 1 is an offline process for estimating skeleton tracking for a plurality of frames at the same time. In addition, there is a direct joint point correspondence between the current frame and past frames, and this correspondence process is computationally expensive. Therefore, there is a problem that it is difficult to use in situations where real-time performance is required.

In Non-Patent Document 2, tracking is performed only between the previous and next frames, and instead of directly using the information of the previous frames, motion information and appearance features in the previous frame are used for inference in the current frame. Recursive use reduces matching processing and enables online real-time processing.

However, tracking based on correspondence between the preceding and following frames is susceptible to adverse effects when occlusions occur frequently. In addition, since the two types of vectors representing the movement direction of the joint point (TAF) and the connection direction of the joint point (PAF) are used to associate the joint points, even if the positions are similar, joints that look very different Even points may be mistaken for each other. Therefore, problems remain, particularly in situations such as sports videos where multiple people are close together and mutual shading occurs.

An object of the present invention is to solve the above technical problems, and even if the joint points being tracked cannot be temporarily detected from the frame image due to shielding, etc., the tracking can be continued with high accuracy and an accurate skeleton can be obtained. An object of the present invention is to provide a skeleton tracking method, apparatus and program capable of realizing tracking.

In order to achieve the above object, the present invention is characterized by having the following configuration in a skeleton tracking device for tracking a human skeleton based on the tracking result of joint points.

(1) Joint point detection means for detecting joint point positions and appearance feature quantities from each frame of time-series images, and for each joint point with a detection history, the position and appearance feature quantity histories are recorded as a joint point tracking flow. storage means for storing; confidence estimation means for estimating confidence that each joint point detected from a frame image corresponds to each joint point tracking flow based on each position and appearance feature quantity; and skeleton estimation means for estimating a skeleton based on a combination of corresponding joint points and the joint point tracking flow.

(2) Based on the estimation results of the skeleton, the positions of the joint points are updated and registered in the joint point tracking flow with the highest degree of confidence, and the appearance information of the joint points is added to the joint with the highest degree of confidence. An updating means is provided to add history to the point tracking flow.

(3) When updating the joint point tracking flow, the updating means determines that the history of the appearance feature amount whose similarity to the appearance feature amount to be added exceeds a predetermined reference value is already registered in the joint point tracking flow. The appearance feature amount is not added.

(4) When the number of appearance feature quantity histories registered in the joint point tracking flow exceeds a prescribed number, the updating means deletes one of the appearance feature quantity pairs whose similarity exceeds a prescribed reference value. .

(5) The confidence estimating means calculates the degree of similarity between the appearance feature amount of the joint point and the history of each appearance feature amount of the joint point tracking flow, and calculates any of the average value, the maximum value and the median value of each degree of similarity. to estimate confidence based on

According to the present invention, the following effects are achieved.

(1) For each joint point that has a detection history, the history of its position and appearance feature amount is stored as a joint point tracking flow, and the certainty that each joint point detected from the frame image corresponds to each joint point tracking flow is Estimates based on each position and appearance feature, and estimates the human skeleton based on the combination of the joint points with the highest confidence and the joint point tracking flow, so the joint points are masked during tracking. Even if it cannot be detected temporarily from the frame image for some reason, it is possible to continue tracking with high accuracy and accurately estimate the skeleton.

(2) Based on the estimation results of the skeleton, the positions of the joint points are updated and registered in the joint point tracking flow with the highest degree of confidence, and the appearance information of the joint points is added to the joint with the highest degree of confidence. It is added as history to the point tracking flow so that the joint point tracking flow can retain the latest position and appearance information.

(3) When updating the joint point tracking flow, the updating means determines that the history of the appearance feature amount whose similarity to the appearance feature amount to be added exceeds a predetermined reference value is already registered in the joint point tracking flow. Since the appearance feature amount is not added, the history of the appearance feature amount can be diversified, and robust estimation of the degree of certainty against changes in the appearance feature amount is possible.

(4) When the number of appearance feature quantity histories registered in the joint point tracking flow exceeds a prescribed number, the updating means deletes one of the appearance feature quantity pairs whose similarity exceeds a prescribed reference value. Therefore, it is possible to reduce the number of histories of appearance feature values registered in the joint point tracking flow while providing diversity, and at the same time, it is possible to reduce the storage capacity and calculation cost required for tracking processing. Become.

(5) The confidence estimating means calculates the degree of similarity between the appearance feature amount of the joint point and the history of each appearance feature amount of the joint point tracking flow, and calculates any of the average value, the maximum value and the median value of each degree of similarity. Since the degree of certainty is estimated based on , the degree of similarity between the joint point appearance feature amount and the history of each appearance feature amount in the joint point tracking flow can be obtained practically and quantitatively.

1 is a diagram showing the configuration of main parts of a skeleton estimation device according to an embodiment of the present invention; FIG. FIG. 4 is a diagram showing an example of definition of joint points; FIG. 10 is a diagram showing the configuration of a joint point tracking flow list; 4 is a flow chart showing a procedure for estimating a tracking confidence; FIG. 4 is a diagram schematically showing a method of estimating a tracking certainty; FIG. 10 is a diagram showing an example of graph generation and subgraph division in which joint points and joint point tracking flows are regarded as nodes;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing the configuration of the main parts of a skeleton estimation apparatus 1 according to one embodiment of the present invention. In this embodiment, it is assumed that skeleton estimation is performed on time-series images in which a plurality of people are captured. However, the time-series images shall continue for a period longer than one frame, and shall not change scenes.

Such a skeleton estimation device can be configured by installing an application (program) that implements each function on at least one general-purpose computer or server equipped with a CPU, a memory, an interface, and a bus connecting them. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is made into hardware or ROMized.

The image input unit 101 receives RGB time-series images of a person as a subject. The joint point detection unit 102 acquires time-series images from the image input unit 101 on a frame-by-frame basis, and according to the definition of joint points shown in an example in FIG. to detect

Any existing technique can be used to detect joint points. For example, a method similar to that of Non-Patent Document 4, which is a skeleton estimation method, can be used. That is, a method may be used in which a frame image is input to a pre-trained CNN model, an existence probability map of each joint point is output by type, and then coordinates with high probability are used as the positions of the joint points. In this embodiment, the type id, the information of the position p, and the appearance feature amount f are acquired for each joint point and provided to the certainty estimation unit 103 .

Here, the joint point type id is information for identifying whether each joint point is a right wrist, a right elbow, or the like, which constitutes the skeleton. The position p is information representing the position of each joint point on the frame image. The appearance feature amount f is information representing image features in a local range centering on the joint point of the frame image, and two typical calculation methods will be described below.

(1) Method using CNN

As disclosed in Non-Patent Document 5, it is calculated by inputting an image (for example, a 5×5 rectangular area centered on the detection position) near the detection position of the joint point to a pre-trained CNN model.

(2) A method that utilizes the processing in the previous stage

If the joint point detection unit 102 uses the same means as in Non-Patent Document 4, the output of the intermediate layer of the CNN model is considered to correspond to the appearance feature amount of the input image. The feature amount of the corresponding portion is cut out as the visible feature amount.

The storage unit 105 stores a joint point tracking flow list 105a and joint point adjacent information 105b. The joint point tracking flow list 105a includes a plurality of joint point tracking flows F generated for each joint point having a detection history, as shown in FIG. Each joint point tracking flow F is composed of a history of the identifier ID of the person to whom the joint point belongs, the joint point type id, the position p, and the appearance feature amount f.

The appearance feature history is a set of appearance features f extracted when the same joint point was detected in each past frame. consists of In FIG. 3, for example, for each joint point, the appearance feature value f at the frame position where the joint point could be detected is schematically marked with a circle, and the frame position where the joint point could not be detected is schematically marked with a space symbol. there is

As the position p, position coordinates in the frame image in which the joint point was last detected can be used. It should be noted that motion information can be reflected in the position of the joint point tracking flow F as the transition of the position so far. The joint point adjacency information 105b is information specifying the adjacency relationship of each joint point defined in the joint point tracking flow 105a.

The certainty estimator 103 includes a joint point adjacent certainty estimator 103a and a joint point tracking certainty estimator 103b. The joint point adjacent certainty estimating unit 103a estimates the certainty that the joint points detected by the joint point detecting unit 102 are adjacent according to the definition of the adjacency relationship between the joint points.

Any existing technique can be used for this confidence estimation. For example, the method disclosed in Non-Patent Document 4, which is a skeleton estimation method, may be adopted. In other words, input an image to a pre-trained CNN model, output a vector map representing the connection between adjacent joint points on the image, and output the direction vector between the two points between the detected joint points. This is a method of obtaining the degree of adjacency confidence from the degree of similarity of vectors. However, regardless of this estimation method, all the degrees of certainty between joint points whose adjacency relationship is not defined are set to 0.

The joint point tracking confidence estimation unit 103b associates each joint point detected from the current frame image with each joint point tracking flow list F of the joint point tracking flow 105a held in the storage unit 105 as a tracking target. Estimate the confidence that

FIG. 4 is a flowchart showing a tracking confidence estimation procedure by the joint point tracking confidence estimation unit 103b, and FIG. 5 is a diagram schematically showing a tracking confidence estimation method.

In step S1, attention is focused on one of the joint points detected by the joint point detection unit 102 from the current frame image. In step S2, one of the joint point tracking flows F having the matching joint point type id among the plurality of joint point tracking flows F registered in the joint point tracking flow list is focused on.

In step S3, the degree of certainty that the joint point of interest is associated with the joint point tracking flow F of interest as a tracking target is estimated based on the similarity of the positions and appearance features of both. In this embodiment, the degrees of certainty between joint points of different joint point types (however, left and right are not included) and the joint point tracking flow are all set to 0 regardless of the degree of similarity between positions and appearance feature amounts. Therefore, the real confidence is estimated based on the similarity between the position p and the appearance feature f.

The positional similarity S1 can be calculated based on the following equation (1) using the Euclidean distance d between the position of the joint point and the position of the joint point tracking flow list.
S1=1/(1+d) (1)

The appearance feature value similarity S2 is defined by the appearance feature value of the joint points detected in the current frame as f0, and the history of each appearance feature value registered in the joint point tracking flow list as [f1, f2..., fn]. , it can be represented by the mean, median or maximum value of the cosine similarity [f0f1, . . . ,f0fn]. However, it is assumed that the history of each appearance feature amount [f0, .

As described above, in this embodiment, the similarity between the appearance feature amount of the joint point and the history of each appearance feature amount of the joint point tracking flow is calculated, and any of the average value, the maximum value, and the median value of each similarity is calculated. Since the degree of certainty is estimated based on this, the degree of similarity between the two can be obtained practically and quantitatively.

The degree of certainty between the joint point of interest and the joint point tracking flow of interest is the combined similarity of the similarity of the position and the appearance feature amount, which is the position similarity S1 and the appearance feature amount similarity S2 obtained as described above. can be represented by the product of Note that when the joint point tracking flow is not stored in the storage unit, such as in the first round of the iterative process, the tracking certainty is not calculated.

In step S4, it is determined whether the certainty estimation of all the joint point tracking flows whose joint point type id matches the joint point of interest has been completed. If not completed, the process returns to step S2, and the above processing is repeated while switching the joint point tracking flow of interest.

When the certainty estimation of all the joint point tracking flows whose joint point type id matches the joint point of interest is completed, the process proceeds to step S5, and all the joint points and all joint point tracking flows whose joint point type id matches. is completed. If not completed, the process returns to step S1, and the above processing is repeated while switching the joint point of interest. When all certainty estimations are completed, the process is terminated.

The certainty-based skeleton tracking estimation unit 104 includes a graph generating unit 104a, a subgraph dividing unit 104b, an updating unit 104c, and an appearance feature amount pruning unit 104d, and combines joint points having a correspondence relationship with the highest certainty and the joint point tracking flow. Estimate the skeleton based on the combination of

As shown in FIG. 6, the graph generation unit 104a generates a graph regarding all joint points and joint point tracking flows as nodes. The edge weight between each node is defined as follows.

That is, (1) the weight of the edge between the joint point nodes is the certainty estimated by the joint point adjacent certainty estimator 103a. (2) The weight of the edge between the joint point node and the joint point tracking flow node is the confidence estimated by the joint point tracking confidence estimation unit 103b. (3) The weight of the edge between the joint point tracking flow nodes is the adjacency confidence between the joint point tracking flows, which will be described later. Note that when the joint point tracking flow is not accumulated in the storage unit 105, such as in the first round of the iterative process, the graph is alternatively constructed from only the edges between the joint point nodes.

The subgraph division unit 104b divides the graph generated as described above into optimal subgraphs under the constraints on the configuration of the joint points of the human body, thereby obtaining the skeleton tracking result for each person. In this embodiment, the generated graph is divided into subgraphs for each person by the same method as in Non-Patent Documents 1, 2, and 6, for example.

The update unit 104c updates the joint point tracking flow 105a and the joint point adjacent information 105b based on the skeleton tracking result for each person. In this embodiment, the person IDs of a plurality of joint point tracking flows that are mutually linked as the same person are updated to the most frequently occurring person IDs among the person IDs already registered in each joint point tracking flow. If there is no linked joint point tracking flow, a new person ID is set.

The updating unit 104c also additionally registers the appearance feature amount f of the joint point in the corresponding joint point tracking flow F as a history of the appearance feature amount. The updating unit 104c further updates the position P of the corresponding joint point tracking flow F based on the position p of the joint point. At this time, if position motion information is registered in relation to the position, the coordinate position of the next frame is predicted based on this motion information and the current position p, and the prediction result corresponds to the corresponding joint point tracking flow. It is updated and registered as the position of F.

For example, as in Non-Patent Document 7, a Kalman filter can be used to predict the motion of the joint point, and the position of the joint point in the next frame can be predicted. Alternatively, after obtaining the image of the next frame, the joint point positions in the next frame may be predicted by a method using optical flow similar to Non-Patent Document 3 or a method using TAF similar to Non-Patent Document 2.

On the other hand, for a joint point that is included in the skeleton of any person but for which the corresponding joint point tracking flow F has not been registered, the joint point tracking flow F corresponding to the joint point is newly registered. In the joint point tracking flow F, the position of the joint point is registered as its position, and the appearance feature amount f of the joint point is registered as the history of the appearance feature amount.

When the number of appearance feature amounts registered in the joint point tracking flow F exceeds a certain number, the appearance feature amount pruning unit 104d deletes appearance feature amounts in order of oldest registration timing. Alternatively, two of the appearance feature amounts registered in the joint point tracking flow F are extracted as an appearance feature amount pair, and one of the appearance feature amount pairs whose similarity exceeds a predetermined reference value is deleted.

When adding the appearance feature quantity f detected from the current frame image to the joint point tracking flow F, the appearance feature quantity pruning unit 104d further compares the appearance feature quantity f to be added and the history of the registered appearance feature quantity f. The degree of similarity is compared, and additional registration of the appearance feature quantity f whose degree of similarity with any of the histories of the already registered appearance feature quantity f exceeds a predetermined reference value is restricted. Alternatively, by replacing the history of registered appearance feature amounts whose similarity exceeds a predetermined reference value, it is possible to diversify the appearance feature amounts to be registered in the joint point tracking flow while limiting the storage capacity, and at the same time, it becomes necessary for the tracking process. It is also possible to reduce the calculation cost.

As described above, according to this embodiment, for each joint point having a detection history, the position and the history of the appearance feature amount are stored as a joint point tracking flow, and each joint point detected from the frame image is stored in each joint point tracking flow. is estimated based on each position and appearance feature amount, and the skeleton of a person is estimated based on the combination of the joint point having the correspondence relationship with the highest confidence and the joint point tracking flow. Therefore, even if the joint points being tracked cannot be temporarily detected from the frame image due to shielding or the like, highly accurate tracking can be continued, and accurate skeleton estimation is possible.

Reference numerals 101: image input unit, 102: joint point detection unit, 103: confidence estimation unit, 103a: joint point adjacent confidence estimation unit, 103b: joint point tracking confidence estimation unit, 104: confidence-based skeleton tracking estimation unit, 104a Graph generation unit 104b Subgraph division unit 104c Update unit 104d Appearance feature amount pruning unit 105 Storage unit 105a Joint point tracking flow list 105b Joint point adjacent information

Claims (12)

A skeleton tracking device for estimating a human skeleton based on tracking results of joint points,
joint point detection means for detecting the position of a joint point from each frame of a time-series image and an appearance feature quantity, which is an image feature of a local range centering on the position of the joint point detected in each frame;
storage means for storing, for each joint point having a detection history, a history of appearance feature amounts, which is a set of positions and appearance feature amounts, as a joint point tracking flow;
confidence estimating means for estimating the confidence that each joint point detected from a frame image corresponds to each joint point tracking flow based on the similarity of each position and appearance feature quantity;
skeleton estimating means for estimating the skeleton of each person based on a combination of joint points having a correspondence relationship with the highest degree of confidence and the joint point tracking flow,
wherein the certainty estimation means estimates the certainty based on the similarity between a joint point appearance feature amount detected from a frame image and a set of joint point appearance feature amounts having a detection history. Device. 2. The skeleton tracking apparatus according to claim 1, further comprising updating means for updating the joint point tracking flow based on the skeleton estimation result. The updating means updates and registers the position of the joint point in the joint point tracking flow having the correspondence relationship with the highest degree of confidence, and the appearance information of the joint point as a history in the joint point tracking flow having the correspondence relationship with the highest degree of confidence. 3. The skeletal tracking device according to claim 2, further comprising: The update means does not add the appearance feature amount if a history of the appearance feature amount whose degree of similarity with the appearance feature amount to be added exceeds a predetermined reference value is already registered in the joint point tracking flow. 4. The skeleton tracking device according to claim 3. The updating means is characterized in that, when the number of histories of appearance feature amounts registered in the joint point tracking flow exceeds a predetermined number, one of the appearance feature amount pairs whose degree of similarity exceeds a predetermined reference value is deleted. A skeleton tracking device according to any one of claims 2 to 4. The certainty estimating means calculates a similarity between the joint point appearance feature amount and the history of each appearance feature amount of the joint point tracking flow, and calculates the degree of similarity based on one of the average value, the maximum value and the median value of each similarity. 6. The skeleton tracking device according to any one of claims 1 to 5, wherein the confidence factor is estimated by In a skeleton tracking method in which a computer estimates a human skeleton based on joint point tracking results,
Detecting the position of the joint point from each frame of the time-series images and the appearance feature quantity , which is an image feature of a local range centering on the position of the joint point detected in each frame,
For each joint point with a detection history, the position and the history of the appearance feature amount, which is a set of appearance feature amounts, are stored as a joint point tracking flow,
estimating the confidence that each joint point detected from the frame image corresponds to each joint point tracking flow based on the similarity of each position and appearance feature,
estimating the skeleton of each person based on the combination of the joint points having the correspondence relationship with the highest degree of confidence and the joint point tracking flow ;
A skeleton tracking method, wherein the degree of certainty is estimated based on a degree of similarity between a joint point appearance feature amount detected from a frame image and a set of joint point appearance feature amounts having a detection history. 8. The skeleton tracking method according to claim 7, wherein the joint point tracking flow is updated based on the estimation result of the skeleton. When updating the joint point tracking flow, the position of the joint point is updated and registered in the joint point tracking flow having the correspondence relationship with the highest degree of confidence, and the appearance information of the joint point is registered to the joint having the correspondence relationship with the highest degree of confidence. 9. The skeleton tracking method according to claim 8, wherein the point tracking flow is added as a history. In a skeleton tracking program for estimating a human skeleton based on joint point tracking results,
a procedure for detecting, from each frame of a time-series image, the position of a joint point and an appearance feature quantity , which is an image feature of a local range centering on the position of the joint point detected in each frame;
a procedure for storing, for each joint point having a detection history, a history of appearance feature amounts, which is a set of positions and appearance feature amounts, as a joint point tracking flow;
a procedure for estimating the degree of confidence that each joint point detected from a frame image corresponds to each joint point tracking flow based on the similarity of each position and appearance feature quantity;
causing a computer to execute a procedure for estimating the skeleton of each person based on the combination of the joint points having the correspondence relationship with the highest degree of confidence and the joint point tracking flow ;
The step of estimating the degree of certainty is characterized by estimating the degree of certainty based on the degree of similarity between the joint point appearance feature amount detected from the frame image and a set of joint point appearance feature amounts having a detection history. skeleton tracking program. 11. The skeleton tracking program according to claim 10, further comprising a step of updating a joint point tracking flow based on said skeleton estimation result. In the updating procedure, the position of the joint point is updated and registered in the joint point tracking flow having the correspondence relationship with the highest degree of confidence, and the appearance information of the joint point is recorded in the joint point tracking flow having the correspondence relationship with the highest degree of confidence. 12. The skeleton tracking program according to claim 11, wherein the program is added as

Images