JP6906273B2 - Programs, devices and methods that depict the trajectory of displacement of the human skeleton position from video data - Google Patents

Description

The present invention relates to a technique for estimating a person's behavior from video data.

Conventionally, there is an image processing technique that realizes robust and highly accurate person tracking regardless of whether or not it is occlusion between people (see, for example, Patent Document 1). According to this technique, when matching the tracking result of the past frame with the detection result of the person area, the similarity is compared as the area of the interpersonal occlusion state separately from the similarity criterion of comparing as the area of a single person. By using the criteria, it is possible to robustly match even when interpersonal occlusion occurs.

There is also a technique for tracking people in a video by dividing them into a group area where people in the video are dense and an individual area other than that (see, for example, Patent Document 2). According to this technique, the tracking process of a person is performed only in the personal area, not in the collective area. That is, when a person is absorbed from the individual area into the group area, the tracking process is stopped, and when the person is separated from the group area into the individual area, the tracking process is started again. In particular, the trajectory of the coordinates in the group area of the person is determined based on the coordinates of the person in the individual area and the numbers written on the clothes.

Further, unlike a person, there is also a technique for tracking the movement of an object (see, for example, Patent Document 3). According to this technique, continuous images in a time series are input to a neural network, and the features of each of the input images are compared with the features extracted by the neural network to match the similarity. Then, the identification information and the position information of one or more objects appearing in the subsequent image, which match the one or more objects appearing in the image at the previous time as tracking candidates, are output as the identification result. As a result, the tracking of the object is displayed on the display.

Further, there is also a technique for displaying the locus of each human skeleton (see, for example, Non-Patent Document 1 and Patent Document 4). There is also a technique for displaying the locus of a fingertip as a specific part in a virtual space (see, for example, Non-Patent Document 2). According to these techniques, the movement locus of the extracted skeleton points can be displayed.

Japanese Unexamined Patent Publication No. 2017-182295 Re-table 2016-139906 JP-A-2018-026108 Japanese Unexamined Patent Publication No. 2015-061579

Zhe Cao etc. "Realtime Multi-Person 2D Pose Optimization using Part Affinity Fields", Carnegie Mellon University, [online], [Search June 16, 2018], Internet <URL: https://arxiv.org/pdf /1611.08050.pdf ＞ Yuki Tanaka, Yasuyuki Kono, "Aerial drawing system of fingertip trajectory by tracking head position / posture from camera image and sensor", Kwansei Gakuin University, Information Processing Society of Japan Research Report, Vol.2015-HCI-162 No.7, [ online], [Search June 16, 2018], Internet <URL: https://ipsj.ixsq.nii.ac.jp/ej/?action=repository_uri&item_id=141307&file_id=1&file_no=1> "I tried OpenPose, which can detect bones from videos and photos", [online], [Search June 16, 2018], Internet <URL: http://hackist.jp/?p=8285>

According to the techniques described in Non-Patent Documents 1 and 2 and Patent Document 4 described above, the movement loci of all skeletons can be displayed.
However, depending on the behavior of the person (for example, in the case of violent movement), the loci of the skeletons often overlap each other, and the displayed movement locus is visually difficult for the user to see.
On the other hand, the inventor of the present application thought that the movement locus of the skeleton that the user wants to pay attention to may differ depending on the behavior of the person.

Therefore, an object of the present invention is to provide a program, an apparatus, and a method for estimating a person's behavior from video data and drawing a trajectory of displacement of the skeleton position according to the behavior.

According to the present invention, it is a program that makes a computer function so as to draw a trajectory of displacement of a person's skeleton position from video data.
A skeleton setting table in which one or more skeletons for which a trajectory should be drawn are associated in advance for each action of a person,
A skeleton recognition means that extracts multiple skeleton positions in time series from video data,
A behavior estimation means for estimating a person's behavior from the displacement amount of the skeleton position in a time series in the first predetermined period,
A skeleton identifying means that identifies one or more skeletons corresponding to the behavior recognized by the behavior estimating means using the skeleton setting table.
It is characterized in that the computer functions as a skeleton displacement depiction means for depicting the displacement of the skeleton position in a time series in a second predetermined period as a locus extracted by the skeleton recognition means for the skeleton identified by the skeleton identification means. ..

According to other embodiments in the program of the present invention
It is also preferable that the skeleton displacement depiction means causes the computer to function so as to superimpose the displacement of the skeleton position in the time series in the second predetermined period as a locus on the video data.

According to other embodiments in the program of the present invention
The first predetermined period of the behavior estimation means is from the pre-estimation time point tn to the estimation time point t.
The second predetermined period of the skeletal displacement depiction means is
Post-estimation time point t + 1 to t + k, or
It is also preferable to operate the computer so that the pre-estimation time point tm to the estimation time point t to the post-estimation time point t + k.

According to other embodiments in the program of the present invention
It also has a first skeletal displacement table in which the displacement amounts of each skeletal position in the first predetermined period are arranged in chronological order for each action of the person.
It is also preferable that the behavior estimation means uses the first skeletal displacement table for each behavior to make the computer function to search for behaviors similar to the skeletal displacement of the time series in the first predetermined period.

According to other embodiments in the program of the present invention
It also has a second skeletal displacement table in which the displacement amounts of each skeletal position in the second predetermined period are arranged in chronological order for each action of the person.
As the skeleton setting table, it is also preferable to make the computer function so as to set a skeleton having a time-series displacement amount based on a predetermined rule by using the second skeleton displacement table in the estimated behavior.

According to other embodiments in the program of the present invention
It is also preferable to make the computer function so that the predetermined rule of the skeleton setting table is preset by the user.

According to other embodiments in the program of the present invention
It is also preferable that the predetermined rule of the skeleton setting table causes the computer to function so as to set a skeleton in which the displacement amount in the predetermined period is larger than the predetermined condition or a skeleton in which the displacement amount in the predetermined period is smaller than the predetermined condition.

According to the present invention, it is a device for drawing a trajectory of displacement of a person's skeleton position from video data.
A skeleton setting table in which one or more skeletons for which a trajectory should be drawn are associated in advance for each action of a person,
A skeleton recognition means that extracts multiple skeleton positions in time series from video data,
A behavior estimation means for estimating a person's behavior from the displacement amount of the skeleton position in a time series in the first predetermined period,
A skeleton identifying means that identifies one or more skeletons corresponding to the behavior recognized by the behavior estimating means using the skeleton setting table.
It is characterized by having a skeleton displacement depiction means for describing the displacement of the skeleton position in a time series in a second predetermined period as a locus, which is extracted by the skeleton recognition means for the skeleton specified by the skeleton identification means.

According to the present invention, it is a method of drawing a skeleton trajectory of an apparatus for drawing a trajectory of displacement of a person's skeleton position from video data.
The device has a skeleton setting table in which one or more skeletons for which a locus should be drawn are associated in advance for each action of the person.
The device is
The first step of extracting multiple skeleton positions in chronological order from video data,
The second step of estimating the behavior of a person from the displacement amount of the skeleton position in the time series in the first predetermined period, and
Using the skeleton setting table, a third step of identifying one or more skeletons corresponding to the behavior recognized by the second step, and
The skeleton identified by the third step is characterized by executing the fourth step of drawing the displacement of the skeleton position in the time series in the second predetermined period as a locus, which is extracted by the first step. do.

According to the program, device and method of the present invention, the behavior of a person can be estimated from the video data, and the locus of displacement of the skeleton position according to the behavior can be drawn.

It is a functional block diagram of the recognition device in this invention. It is explanatory drawing which shows the skeleton position recognized by the skeleton recognition part. It is explanatory drawing which shows the 1st Embodiment of the skeleton recognition part and behavior estimation part in this invention. It is explanatory drawing which shows the 2nd Embodiment of the skeleton recognition part and behavior estimation part in this invention. It is explanatory drawing which shows the displacement of the skeleton position in time series in this invention. It is explanatory drawing which shows the 1st Embodiment which determines the skeleton setting table. It is explanatory drawing which shows the 2nd Embodiment which determines the skeleton setting table.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional configuration diagram of the recognition device according to the present invention.

According to FIG. 1, the recognition device 1 of the present invention can depict a trajectory of displacement of a person's skeleton position from video data.
The recognition device 1 may be a smartphone or a mobile terminal equipped with a camera, and inputs video data of a person. Further, the recognition device 1 may receive video data via an access network such as a mobile phone network or a wireless LAN.
Of course, the recognition device 1 is not limited to a smartphone or the like, and may be, for example, a fixedly installed Web camera. Further, the video data captured by the Web camera may be recorded on the SD card, and the recorded video data may be input to the recognition device 1.
In the following, the recognition device 1 will be described as a smartphone equipped with a camera, but may function as a server connected to the Internet, for example.

According to FIG. 1, the recognition device 1 has a skeleton setting table 10, a skeleton recognition unit 11, a behavior estimation unit 12, a skeleton identification unit 13, and a skeleton displacement depiction unit 14. These functional components can be realized by executing a program that makes the computer mounted on the device function. Further, the processing flow of these functional components can be understood as a method of drawing the skeleton trajectory of the device.

[Skeletal recognition unit 11]
The skeleton recognition unit 11 extracts a plurality of skeleton positions of a person in time series from the video data. The video data for extracting the skeleton is based on two dimensions and may be captured by a general Web camera. The extracted skeleton position is output to the behavior estimation unit 12.

FIG. 2 is an explanatory diagram showing the skeleton position recognized by the skeleton recognition unit.

Specifically, the skeleton recognition unit 11 uses a skeleton model such as OpenPose (registered trademark) to extract characteristic points of the human skeleton (see, for example, Non-Patent Document 3). OpenPose is software that can detect multiple human body / hand / face key points in real time from images, and is published by GitHub. For example, 18 key points can be detected in the entire human body shown in the captured image.
According to FIG. 2, one person is reflected in the video data. In the case of OpenPose, the two-dimensional coordinate points and reliability of each of the 18 skeleton positions (Nose, Neck, RShoulder, RElbow, ...) are linked in each frame.

[Behavior estimation unit 12]
The behavior estimation unit 12 estimates the behavior of the person from the displacement amount of the skeleton position in the time series in the first predetermined period. Specifically, the behavior estimation unit 12 has constructed a learning model for deep learning in advance based on teacher data in which "behavior" is associated with the displacement amount of the skeletal position in the time series of the video data.
Then, the behavior estimation unit 12 recognizes the "behavior" from the time-series skeleton position of the video data by using the learning model. The behavior estimation unit 12 recognizes a person's behavior such as "kicking", "sitting", and "dancing" from the time-series skeletal position of the person.

FIG. 3 is an explanatory diagram showing a first embodiment of the skeleton recognition unit and the behavior estimation unit in the present invention.
According to FIG. 3, both the skeleton recognition unit 11 and the behavior estimation unit 12 may be configured by, for example, OpenPose. In the case of OpenPose, the score is calculated for each action by class classification. That is, by inputting the "video data", the "behavior" having the highest score can be estimated.

FIG. 4 is an explanatory diagram showing a second embodiment of the skeleton recognition unit and the behavior estimation unit in the present invention.

According to FIG. 4, a plurality of skeleton positions are recognized only by the skeleton recognition unit 11, for example, OpenPose. Further, the behavior estimation unit 12 is associated with the first skeleton displacement table 121.

[First Skeleton Displacement Table 121]
The first skeleton displacement table 121 arranges the displacement amount of each skeleton position in the first predetermined period in which the action can be specified for each "action" of the person in chronological order. According to FIG. 4, for example, the amount of displacement from the previous time in three-dimensional (x, y, z) coordinates is shown.
According to FIG. 4, the first predetermined period is from the pre-estimation time point tn to the estimated time point t ([tn] is a time point arbitrary n hours before the estimated time point t).
The behavior estimation unit 12 uses the first skeleton displacement table 121 to determine the time-series skeleton displacement of the first predetermined period from the change in the displacement amount of each skeleton position for each action up to the estimation time t in the video data. Select the most similar "behavior" that is similar. Of course, the action is not limited to one action, and a plurality of actions may be selected in descending order of similarity.

According to the first skeleton displacement table of FIG. 4, it can be understood that the displacement amount of the position of the skeleton 10 “right ankle (RAnkle)” is the largest in the case of the action “kicking”.
In the case of the action "sitting", it can be understood that the displacement amount of the entire skeletal position is relatively small.
In the case of the action "dancing", it can be understood that the displacement amount of the position of the skeleton 8 "right hip (Hip)" is relatively large.

[Skeletal setting table 10]
The skeleton setting table 10 is associated with one or more skeletons for which a locus should be drawn in advance for each action of the person.
According to FIG. 1 described above, the skeleton setting table 10 is associated with the skeleton number as follows for each action of the person.
Action "Kick"-> Skeleton number 10 "Right ankle"
Action "Sit"-> Skeleton No. 4 "Right wrist"
Action "Dancing"-> Skeleton number 0 "Nose"

[Skeletal identification part 13]
The skeleton identification unit 13 identifies one or more skeletons corresponding to the behavior recognized by the behavior estimation unit 12 by using the skeleton setting table 10.
For example, when the action "kicking" is estimated by the action estimation unit 12, the skeleton number 10 "right ankle" is selected using the skeleton setting table 10.

[Skeletal displacement depiction unit 14]
The skeleton displacement depiction unit 14 describes the displacement of the skeleton position in the time series in the second predetermined period extracted by the skeleton recognition unit 11 as a locus for the skeleton specified by the skeleton identification unit 13.
Further, the skeleton displacement depiction unit 14 superimposes the time-series displacement of the skeleton position as a locus on the video data.

Here, the second predetermined period may be any of the following.
(1) Post-estimation time point t + 1 to t + k ([t + k] is a time point arbitrary k hours after the estimated time point t)
In this case, the "behavior" is estimated from the pre-estimation time point tn to the estimation time point t, and the displacement trajectory of the skeleton position is drawn from the post-estimation time point t + 1 to the time point t + k for the skeleton number specified from the behavior. do. That is, the displacement of the skeletal position is depicted from the stage after the behavior is estimated.
(2) Pre-estimation time point tm ~ Estimated time point t ~ Post-estimation time point t + k
In this case, the "behavior" is estimated from the pre-estimation time point tn to the estimation time point t, and the displacement of the skeleton position from the pre-estimation time point tm to the post-estimation time point t + k that goes back to the past for the skeleton number specified from the behavior. Depict the trajectory of. That is, the displacement of the skeletal position is depicted from the stage before the behavior is estimated ([tm] is a time point arbitrary m hours before the estimation time point t).

FIG. 5 is an explanatory diagram showing the time-series displacement of the skeleton position in the present invention.

(1) Between the pre-estimation time point t-2 and the estimated time point t, the estimated score of the action "kicking" gradually increases from the displacement amount of the skeleton position, and the action "kicking" exceeds the estimated threshold value at the time point t. It is output. At this time, the skeleton 10 “right ankle” corresponding to the action “kicking” is specified using the skeleton setting table 10.
(2) The locus of the position of the skeleton 10 “right ankle” is recorded between t + 1 and t + 3 at the time point after estimation. The depiction of the locus of skeleton 10 "right ankle" may be updated gradually or may be displayed at any time point t + 4.
According to FIG. 5, the displacement of the skeleton position is depicted in a two-dimensional image, but of course, it may be a three-dimensional image.

<Skeleton setting in the skeleton setting table 10>
The skeleton setting table 10 sets a skeleton that is a time-series displacement amount based on a predetermined rule by using the second skeleton displacement table 101 in the specified action.

FIG. 6 is an explanatory diagram showing a first embodiment for determining a skeleton setting table.

According to FIG. 6, a second skeleton displacement table 101 and a predetermined rule are shown in order to determine the skeleton setting table 10.
[Second skeletal displacement table 101]
The second skeleton displacement table 101 arranges the displacement amount of each skeleton position in the second predetermined period in chronological order for each action of the person.
The second skeleton displacement table 101 is past statistical data, and represents the displacement amount of each skeleton position according to the elapsed time for each action. According to FIG. 6, for each of the actions “kicking”, “sitting”, and “dancing”, the amount of displacement of each skeletal position of the person is recorded according to the elapsed time.
[Predetermined rule]
The predetermined rule sets a skeleton in which the displacement amount in the predetermined period is larger than the predetermined condition, or a skeleton in which the displacement amount in the predetermined period is smaller than the predetermined condition.
According to FIG. 6, it is assumed that, for example, "the skeleton having the largest displacement within a predetermined period" is defined as a predetermined rule.

In this case, the skeleton having the largest displacement within a predetermined period is registered in the skeleton setting table 10 for each action.
In the case of the action "kicking", the skeleton 10 "right ankle" having the largest displacement within a predetermined period is registered.
In the case of the action "sitting", the skeleton 4 "right wrist" having the largest displacement within a predetermined period is registered.
In the case of the action "dancing", the skeleton 0 "nose" having the largest displacement within a predetermined period is registered.
In this way, the skeleton setting table 10 is registered according to the predetermined rule.

Here, the "predetermined period" for determining the predetermined rule in association with each other is arbitrary.
For example, it may be a post-estimation time point t + 1 to t + 2 that is after the behavior estimation time point t.
For example, it may be t-1 to t + 2 before the estimation of the behavior.
That is, the numerical value of the displacement amount of each skeleton position from the pre-estimation time point tn to the estimated time point t in the second skeleton displacement table 101 may be exactly the same as that of the first skeleton displacement table.

FIG. 7 is an explanatory diagram showing a second embodiment for determining the skeleton setting table.

According to FIG. 7, as compared with FIG. 6, the predetermined rule of the skeleton setting table 10 is preset by the user. That is, the skeleton setting table 10 is registered according to different rules for each user.
The second skeleton displacement table 101 is the same as in FIG.

According to the predetermined rule for each user in FIG. 7, the settings are as follows.
(User A)
"Kick": Skeleton with the largest amount of displacement in the first predetermined period "Sit": Skeleton with the smallest amount of displacement in the first predetermined period "Dancing": Skeleton with the largest amount of displacement in the second predetermined period (user) B)
"Kick": Skeleton with the smallest amount of displacement in the first predetermined period "Sit": Skeleton with the largest amount of displacement in the first predetermined period "Dancing": Skeleton with the largest amount of displacement in the second predetermined period The predetermined period can also be set arbitrarily. As a result, not only the skeleton number based on intense behavior or quiet behavior can be detected in a short time, but also the skeleton number based on intense behavior or quiet behavior can be detected in a long time.

According to the user A, when the action "kicking" is estimated, the skeleton 10 "right ankle" having the largest displacement amount in the first predetermined period is selected. In this case, the displacement trajectory of the skeleton 10 "right ankle" is depicted.
Further, according to the user B, when the action "kicking" is estimated, the skeleton 1 "neck" having the smallest displacement amount in the first predetermined period is selected. In this case, the locus of displacement of the skeleton 1 "neck" is drawn.

As described in detail above, according to the program, device, and method of the present invention, the behavior of a person can be estimated from the video data, and the locus of displacement of the skeleton position according to the behavior can be drawn.
Further, according to the present invention, the skeletal position according to the estimated behavior can be automatically specified based on a predetermined rule. That is, the locus of displacement of the skeleton position that the user should pay attention to can be drawn according to a predetermined rule.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Recognition device 10 Skeleton setting table 101 Second skeleton displacement table 11 Skeleton recognition unit 12 Behavior estimation unit 121 First skeleton displacement table 13 Skeleton identification unit 14 Skeleton displacement depiction unit

Claims (9)

A program that makes a computer function to depict the trajectory of displacement of a person's skeleton position from video data.
A skeleton setting table in which one or more skeletons for which a trajectory should be drawn are associated in advance for each action of a person,
A skeleton recognition means that extracts a plurality of skeleton positions in time series from the video data,
A behavior estimation means for estimating a person's behavior from the displacement amount of the skeleton position in a time series in the first predetermined period,
A skeleton specifying means that identifies one or more skeletons corresponding to the behavior recognized by the behavior estimating means using the skeleton setting table.
It is characterized in that the computer functions as a skeleton displacement depiction means for depicting the displacement of the skeleton position in a time series in a second predetermined period as a locus extracted by the skeleton recognition means for the skeleton identified by the skeleton identification means. Program to be. The first aspect of the present invention, wherein the skeleton displacement depiction means causes the computer to function so as to superimpose the displacement of the skeleton position in the time series in the second predetermined period as a locus on the video data. program. The first predetermined period of the behavior estimation means is from the pre-estimation time point tn to the estimation time point t.
The second predetermined period of the skeletal displacement depiction means is
Post-estimation time point t + 1 to t + k, or
The program according to claim 1 or 2, wherein the computer functions so that the pre-estimation time point tm to the pre-estimation time point t to the post-estimation time point t + k. It also has a first skeletal displacement table in which the displacement amounts of each skeletal position in the first predetermined period are arranged in chronological order for each action of the person.
The claim is characterized in that the behavior estimation means makes a computer function to search for a behavior similar to a time-series skeletal displacement in a first predetermined period by using a first skeletal displacement table for each behavior. The program described in 3. It also has a second skeletal displacement table in which the displacement amounts of each skeletal position in the second predetermined period are arranged in chronological order for each action of the person.
The skeleton setting table is characterized in that the computer functions to set a skeleton that is a time-series displacement amount based on a predetermined rule by using a second skeleton displacement table in the estimated behavior. The program according to 3 or 4. The program according to claim 5, wherein the predetermined rule of the skeleton setting table causes the computer to function as if it is preset by a user. The predetermined rule of the skeleton setting table is characterized in that the computer functions to set a skeleton in which the displacement amount in a predetermined period is larger than a predetermined condition or a skeleton in which the displacement amount in a predetermined period is smaller than the predetermined condition. The program according to claim 6. It is a device that draws the trajectory of displacement of the skeleton position of a person from video data.
A skeleton setting table in which one or more skeletons for which a trajectory should be drawn are associated in advance for each action of a person,
A skeleton recognition means that extracts a plurality of skeleton positions in time series from the video data,
A behavior estimation means for estimating a person's behavior from the displacement amount of the skeleton position in a time series in the first predetermined period,
A skeleton specifying means that identifies one or more skeletons corresponding to the behavior recognized by the behavior estimating means using the skeleton setting table.
The skeleton identified by the skeleton specifying means is characterized by having a skeleton displacement depiction means that describes the displacement of the skeleton position in a time series in a second predetermined period as a locus, which is extracted by the skeleton recognition means. Device. It is a method of drawing the skeleton trajectory of a device that draws the trajectory of the displacement of the skeleton position of a person from video data.
The device has a skeleton setting table in which one or more skeletons for which a locus should be drawn are associated in advance for each action of a person.
The device is
The first step of extracting a plurality of skeleton positions in time series from the video data,
The second step of estimating the behavior of a person from the displacement amount of the skeleton position in the time series in the first predetermined period, and
Using the skeleton setting table, a third step of identifying one or more skeletons corresponding to the behavior recognized by the second step, and
The skeleton identified by the third step is characterized by executing the fourth step of drawing the displacement of the skeleton position in the time series in the second predetermined period as a locus, which is extracted by the first step. How to describe the skeletal trajectory of the device.

Images