JP7060549B2 - Coaching application construction equipment, methods and programs - Google Patents

Description

The present invention relates to a coaching application construction device, method and program, and more particularly to a coaching application construction device, method and program for constructing a coaching application based on training images of specialists such as coaches and trainers.

Training may be conducted at home or at a facility such as a training gym for the purpose of maintaining or promoting health. Expert guidance is indispensable for safe and effective training, but it is difficult to receive expert guidance in home training, and it becomes a self-styled training.

The training gym is equipped with training equipment and specialized instructors are stationed there. However, it is not easy because you have to go to the training gym. In addition, in order to receive one-on-one instruction, a reasonable cost burden will be required, and time constraints will increase.

In response to such technical problems, Patent Document 1 displays a model still image showing operation states at a plurality of timings of a predetermined operation, and a photographed moving image of a player demonstrating the predetermined operation. Disclosed is a technique for displaying a plurality of captured still images extracted from an image and displaying an image selected by an operator as a corresponding still image corresponding to the model still image from the captured still image side by side with the model still image. ing.

In Patent Document 2, a user receives a playback instruction of a training video including a predetermined number of repetitive exercises, the training video can be played based on the playback instruction, and a portion of the repetitive exercise is extracted from the training video. A technique for controlling the reproduction of a part of repetitive exercise based on the viewing condition of a training video is disclosed.

Japanese Patent No. 6180011 Japanese Unexamined Patent Publication No. 2018-114228

Z. Cao, T. Simon, S. Wei and Y. Sheikh, "Realtime Multi-person 2D Pose Optimization Using Part Affinity Fields," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp . 1302-1310.

If you want to analyze the training video with an application and evaluate the training automatically, it is necessary to extract the movement of the part that is the point of evaluation from the training video, and further quantify the movement of each part and evaluate it individually. .. However, specialists such as coaches and trainers may empirically and sensuously grasp the movement of each part, and implement such subjective and potential evaluation criteria in coaching applications and use them for evaluation. It wasn't easy.

In addition, even if the evaluation criteria for the movement of each part can be quantitatively grasped, advanced expertise is required to incorporate this into the application and build a coaching application, and coaches who do not specialize in application construction It is a huge burden for the trainer.

An object of the present invention is to provide a coaching application construction device, method and program capable of solving the above technical problems and constructing a coaching application only by shooting a training scene of a specialist such as a coach or a trainer.

In order to achieve the above object, the present invention is characterized in that the following configuration is provided in a coaching application construction device that evaluates training by comparing the amount of movement of an evaluation site extracted from a training video with an evaluation threshold. There is.

(1) Means for extracting skeletal information from the training video, means for determining the evaluation site based on the skeletal information, means for calculating the amount of movement of the evaluation site, and for evaluating the evaluation site based on the amount of movement. It is provided with a means for setting the evaluation threshold value of.

(2) As a means for determining the evaluation site, the site where the amount of movement exceeds a predetermined reference value is determined as the evaluation site.

(3) An interface that accepts correction operations for the set evaluation threshold value is provided so that the evaluation threshold value can be corrected according to the correction operation.

(4) Further means for displaying the amount of movement and the evaluation threshold in a graphic manner are provided.

According to the present invention, the following effects are achieved.

(1) Since the evaluation threshold value when evaluating the training of the evaluation target can be set based on the training video of the coach etc. as a model, the coaching application can set the evaluation threshold value that the coach etc. grasps empirically and sensuously. Can be set easily and quantitatively.

(2) Since the evaluation site to be evaluated is set based on the amount of movement of each site based on the skeletal information extracted from the training video as a model, the evaluation site can be set appropriately.

(3) While the evaluation threshold is set automatically, the manual correction is also possible, so the evaluation threshold can be optimized based on the experience and intuition of coaches and the like.

(4) Since the movement amount and the evaluation threshold are displayed graphically, the evaluation threshold automatically set based on the movement amount can be visually recognized.

It is a figure which showed the structure of the main part of the coaching application construction apparatus 1 which concerns on one Embodiment of this invention. It is a figure which showed the extraction example of the skeleton information. It is a figure which showed the example which sets the evaluation threshold value of training based on the movement amount P. It is a figure which showed the configuration example of the network to which this invention is applied. It is a figure which showed an example of a coaching application construction screen. It is a sequence flow showing how to build a coaching application and how to apply it. It is a figure (the 1) which showed the method of changing the evaluation threshold value Pref. It is a figure (No. 2) which showed the method of changing the evaluation threshold value Pref. It is a figure which showed the display example of a message. It is a sequence flow showing how to build a coaching application and other application methods.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a configuration of a main part of a coaching application construction device 1 according to an embodiment of the present invention, which is a training video acquisition unit 101, a person area estimation unit 102, a skeleton information extraction unit 103, and an evaluation site determination. A coaching application 107 to be used by a user is constructed including a unit 104, an evaluation threshold setting unit 105, a coaching application construction unit 106, a display 108, and an input interface 109.

Such a coaching application construction device 1 can be configured by mounting an application (program) that realizes each function described later on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or programmed.

The training video acquisition unit 101 acquires a camera image of a training performed by a training expert such as a coach or a trainer as a model training video in frame units. The person area estimation unit 102 extracts a person area from each frame image. For example, SSD (Single Shot Multibox Detector) can be used to extract the person area.

As shown in FIG. 2, the skeleton information extraction unit 103 extracts the skeleton of a person object from each frame image, and registers the position information and the connection state with other skeletons as skeleton information. The existing skeleton extraction technique (Cascaded Pyramid Network) can be used to extract the skeleton information.

The skeleton extraction method is not limited to the method of targeting the person region extracted in advance as described above. For example, as disclosed in Non-Patent Document 1, for a feature map extracted from a frame image, a Confidence Map that encodes the position of a body part and a Part Affinity Fields (PAFs) that encode the connectivity between the body parts are provided. A skeleton model is constructed by sequentially applying the two sequential prediction processes used and estimating the position and connectivity of the body parts of the person object (user) extracted from the frame image with a single inference using a bottom-up approach. You may.

At this time, by implementing a process that excludes the connectivity of body parts extracted from different partial regions, the position and connectivity of the body parts are estimated for each partial region, that is, for each user. become able to.

The evaluation site determination unit 104 includes the motion amount calculation unit 104a, calculates the motion amount P of each site such as the hand, foot, head, knee joint, and elbow joint based on the extracted skeletal information, and the motion amount P is predetermined. The part that exceeds the reference value or the part where the movement amount P is in the top N best is set as the part (evaluation part) to be evaluated at the time of coaching. For example, in the case of a squat training image, the movement amount P of the knee joint exceeds the reference value, so the knee joint is set as the evaluation site.

The evaluation threshold setting unit 105 includes a plurality of threshold value setting units including the first to fourth threshold value setting units 105a to 105d, analyzes the movement amount P of the set evaluation part, and moves the evaluation part during coaching. Set the evaluation threshold Pref to evaluate.

In the present embodiment, the video used for constructing the coaching application is a training video of an expert, and the calculated movement amount P is considered to be a sufficient amount of movement for a general user to obtain a training effect. Therefore, in the present embodiment, it is considered that a sufficient effect can be obtained if the user can achieve some percentage of the movement amount P at the time of coaching.

For example, in a squat training video, bending and stretching of the knee joint are repeated. Therefore, as shown in FIG. 3, the knee joint that repeatedly moves is set as the evaluation site, and the maximum value of the amount of movement P to one side is set. The minimum value, the average value, the predetermined percentile value, etc. of are set in the evaluation threshold Pref_1 of the amount of movement P to one side.

Similarly, the maximum value, the average value, a predetermined percentile value, or the like of the minimum value of the movement amount P to the other side is set in the evaluation threshold value Pref_2 of the movement amount to the other side. In the illustrated example, the evaluation threshold Pref_1 is set to 165 ° based on the maximum value of the movement amount P, and the evaluation threshold Pref_2 is set to 110 ° based on the minimum value. In addition, if the cycle T of the repetitive motion is acquired, the cycle T ± Δt can be set in the threshold range of the cycle T.

In the present embodiment, the set value of the movement amount P and the evaluation threshold value Pref based on the movement amount P is displayed on the display 108 in the format as shown in FIG. 3, so that the coach or the like evaluates the movement amount P. The relationship with the threshold Pref can be confirmed on the graphic display.

As described with reference to FIG. 3, the first threshold value setting unit 105a determines that the amount of movement P when the skeleton repeatedly moves exceeds the reference value (or enters the upper N best, and so on). For the evaluated site, the threshold value Pref_1 of the movement amount P to one side and the threshold value Pref_2 of the movement amount P to the other side are set as described above based on the movement amount P to one side and the other side of the repetitive motion, respectively. And set.

The second threshold value setting unit 105b sets the evaluation threshold value Tref of the cycle T based on the calculation result of the cycle T of the repeat motion for the evaluation site where the cycle T when the skeleton repeatedly moves exceeds the reference value, for example, Tref = Set T ± ΔT (hereinafter, the evaluation threshold is represented by Pref). The third threshold value setting unit 105c sets the evaluation threshold value Pref of the displacement amount based on the calculation result of the displacement amount for the evaluation portion where the displacement amount of the skeleton exceeds the reference value. The fourth threshold value setting unit 105d sets the evaluation threshold value Pref of the rotation angle based on the calculation result of the rotation angle with respect to the evaluation portion where the rotation angle of the skeleton exceeds the reference value.

The threshold value adjusting unit 105e provides a man / machine interface for finely adjusting the evaluation threshold value Pref or the like automatically set based on the movement amount P as described above by a graphic operation from the input interface 109 by the coach or a numerical input operation. .. Such a fine adjustment function can be used to correct an automatically set evaluation threshold value Pref or the like when it deviates from a general appropriate value or a coach's subjective appropriate value.

The coaching application construction unit 106 analyzes the training video of the user and evaluates the training based on whether or not the movement amount P of each evaluation site exceeds the evaluation threshold value Pref set by the evaluation threshold value setting unit 105. Build a coaching application 107.

The coaching application 107 includes a training video acquisition function 107a, a person area estimation function 107b, a skeleton information extraction function 107c, a movement amount acquisition function 107d, and an evaluation function 107e, and is mounted on the server 3 in the network configuration shown in FIG. , Evaluate the training video of the user acquired from the user terminal 2 via the network. The user terminal 2 can be replaced with a smartphone or tablet terminal having a camera function, a display and a wireless communication function.

In the coaching application 107, the training video acquisition function 107a acquires a camera image of the training demonstrated by the user as a training video of the evaluation target person in frame units. The person area estimation unit 107b extracts the user's person area from each frame image. The skeleton information extraction function 107c extracts the user's skeleton from each frame image, and registers the position information and the connection state with other skeletons as skeleton information.

The movement amount acquisition function 107d measures the movement amount P for each evaluation site based on the skeleton information of the user. The evaluation function 107e compares the measurement result of the movement amount P with the evaluation threshold Pref set by the evaluation threshold setting unit 105, and if there is an evaluation part where the movement amount P is lower than the evaluation threshold Pref, the movement amount with respect to the evaluation part. A message or the like indicating that P is insufficient is returned to the user terminal 2 to call attention to the user.

FIG. 5 is a diagram showing a display example of the coaching application construction screen displayed on the display 108, and is a diagram showing a training video playback area A1 for playing a training video, a training video for inputting operations such as playing and stopping the training video. It is equipped with an operation area A2, a condition setting area A3 for setting training conditions, a movement amount visualization area A4 for visualizing the movement amount of the evaluation site in the training video, and a count condition registration area A5 for registering the count conditions for the number of trainings.

FIG. 6 is a sequence flow showing a method of constructing a coaching application and a coaching method using the coaching application.

At time t1, a training video of a specialist such as a coach or trainer training is uploaded. At time t2, the training video is taken into the coaching application construction device 1 by the training video acquisition unit 101. At time t3, the person area is estimated and extracted by the person area estimation unit 102, and is displayed in the training video reproduction area A1 of the display 108 as shown in FIG. At time t4, the skeleton information extraction unit 103 extracts the skeleton information from the person area, and as shown in FIG. 5, the skeleton information is superimposed and displayed on the training video of the training video reproduction area A1.

At time t5, the evaluation site determination unit 104 calculates the movement amount P of each site based on the skeletal information, and the site where the movement amount P exceeds a predetermined reference value is determined as the evaluation site. At this time, the movement amount P of the determined evaluation site is graphically displayed in the movement amount visualization area A4 of the display 108. For example, if the training is a squat exercise, the knee is determined as the evaluation site, so the change in knee angle is displayed in chronological order.

At time t6, the evaluation threshold setting unit 105 automatically sets an evaluation threshold Pref for evaluating the movement of each evaluation part during coaching with respect to the determined evaluation part, and the movement amount P is displayed graphically. It is superimposed.

At time t7, when the coach or the like operates the threshold value adjusting unit 105e from the input interface 109, specifies the threshold value Pref1 or Pref2 of the movement amount P as shown in FIGS. At t8, the automatically set threshold values Pref1 and Pref2 are finely adjusted in response to the operation.

At time t9, a coaching application is constructed based on the evaluation site and its evaluation threshold Pref. In the present embodiment, the automatically set evaluation site and the evaluation thresholds Pref1 and Pref_2 are registered as count conditions, and the registration result is displayed in the count condition registration area A5. In the example of FIG. 5, the angle is defined as "right hip-right knee ⇔ right knee-right ankle" as the count condition of the "knee" of the evaluation site, and Pref1 and Pref2 are "165 °" and "110", respectively. It is visualized that it is set to "°".

After that, at time t10, the user uploads his / her own training video, and at time t11, the training video is acquired by the coaching application. At time t12, the person area is estimated from each frame image. At time t13, the skeleton information of each evaluation site is extracted from the person area. At time t14, the amount of movement P of each evaluation site is calculated based on the skeletal information.

At time t15, the amount of movement P of each evaluation site is compared with the evaluation threshold Pref, and the training of each evaluation site is evaluated based on the magnitude relationship thereof. At time t16, for example, "Please raise your arm." Is delivered to the user terminal 2 as a message regarding the evaluation result of the training. At time t17, as shown in FIG. 9, the message is output from the display or speaker of the user terminal 2.

In the above embodiment, the constructed coaching application is mounted on the server 3 on the network, and the training video of the user is acquired from the user terminal 2 and evaluated on the server 3, thereby increasing the ability of the user terminal 2. Regardless, it has become possible to realize coaching by application.

However, the present invention is not limited to this, and if the processing capacity of the user terminal 2 is sufficiently high, the coaching application is implemented in the user terminal 2 and the user terminal 2 is trained as shown in FIG. May be able to be evaluated independently. By doing so, since the load related to the training evaluation is distributed to each user terminal 2, the load of the server 3 can be kept low even when the training of a large number of users is evaluated at the same time.

1 ... coaching application construction device, 2 ... user terminal, 3 ... server, 101 ... training video acquisition unit, 102 ... person area estimation unit, 103 ... skeleton information extraction unit, 104 ... evaluation site determination unit, 104a ... movement amount calculation unit, 105 ... evaluation threshold setting unit, 106 ... coaching application construction unit, 107 ... coaching application

Claims (13)

In a coaching application construction device that evaluates training by comparing the amount of movement of the evaluation site extracted from the training video with the evaluation threshold value.
A means of extracting skeletal information from training videos,
A means of determining the evaluation site based on skeletal information,
A means to calculate the amount of movement of the evaluation site,
A means for setting an evaluation threshold value for evaluating an evaluation site based on the amount of movement thereof is provided .
The means for determining the evaluation site is a device for constructing a coaching application, characterized in that a site where the amount of movement exceeds a predetermined reference value is determined as the evaluation site . In a coaching application construction device that evaluates training by comparing the amount of movement of the evaluation site extracted from the training video with the evaluation threshold value.
A means of extracting skeletal information from training videos,
A means of determining the evaluation site based on skeletal information,
A means to calculate the amount of movement of the evaluation site,
A means for setting an evaluation threshold value for evaluating an evaluation site based on the amount of movement thereof is provided.
The means for calculating the amount of movement of the evaluation site calculates the amount of movement to one side and the amount of movement to the other side of the repetitive motion for the evaluation site that repeats the motion.
The means for setting the evaluation threshold is characterized in that the evaluation threshold of the movement amount to one side and the evaluation threshold of the movement amount to the other side are set based on the respective movement amounts to the one side and the other side, respectively. A device for building a coaching application. In a coaching application construction device that evaluates training by comparing the amount of movement of the evaluation site extracted from the training video with the evaluation threshold value.
A means of extracting skeletal information from training videos,
A means of determining the evaluation site based on skeletal information,
A means to calculate the amount of movement of the evaluation site,
A means for setting an evaluation threshold value for evaluating an evaluation site based on the amount of movement thereof is provided.
The means for calculating the amount of movement of the evaluation site calculates the cycle of the repetitive motion as the amount of motion for the evaluation site that repeatedly moves.
The means for setting the evaluation threshold is a device for constructing a coaching application, which comprises setting an evaluation threshold for a cycle. The means for calculating the amount of movement of the evaluation site calculates the amount of movement to one side and the amount of movement to the other side of the repetitive motion for the evaluation site that repeats the motion.
The means for setting the evaluation threshold is characterized in that the evaluation threshold of the movement amount to one side and the evaluation threshold of the movement amount to the other side are set based on the respective movement amounts to the one side and the other side, respectively. The apparatus for constructing a coaching application according to claim 1 . The means for calculating the amount of movement of the evaluation site calculates the cycle of the repetitive motion as the amount of motion for the evaluation site that repeatedly moves.
The apparatus for constructing a coaching application according to claim 1 , wherein the means for setting the evaluation threshold value is to set the evaluation threshold value of the cycle. The means for calculating the amount of movement of the evaluation site calculates the amount of displacement as the amount of movement for the evaluation site to be displaced.
The apparatus for constructing a coaching application according to claim 1 , wherein the means for setting the evaluation threshold value is to set the evaluation threshold value of the displacement amount. The means for calculating the amount of movement of the evaluation site calculates the rotation angle of the rotating evaluation site as the amount of movement.
The apparatus for constructing a coaching application according to claim 1 , wherein the means for setting the evaluation threshold value is to set the evaluation threshold value of the rotation angle. An interface that accepts correction operations for the set evaluation threshold, and
The apparatus for constructing a coaching application according to any one of claims 1 to 7 , further comprising a means for correcting an evaluation threshold value according to the correction operation. The apparatus for constructing a coaching application according to claim 8 , further comprising means for displaying the amount of movement and the evaluation threshold in a graphic manner. In a method of constructing a coaching application in which a computer evaluates training by comparing the amount of movement of an evaluation site extracted from a training video with an evaluation threshold value.
The procedure for extracting skeletal information from the model training video,
The procedure for determining the evaluation site based on skeletal information and
The procedure for calculating the amount of movement of the evaluation site and
Including a procedure for setting an evaluation threshold for evaluating an evaluation site based on the amount of movement thereof.
The method for constructing a coaching application is characterized in that, in determining the evaluation site, a site in which the amount of movement exceeds a predetermined reference value is determined as the evaluation site . Including a procedure for accepting a correction operation for the set evaluation threshold.
The method for constructing a coaching application according to claim 10 , wherein the evaluation threshold value is corrected according to the correction operation. In a coaching application construction program that evaluates training by comparing the amount of movement of the evaluation site extracted from the training video with the evaluation threshold.
The procedure for extracting skeletal information from the model training video,
The procedure for determining the evaluation site based on skeletal information and
The procedure for calculating the amount of movement of the evaluation site and
Let the computer perform the procedure of setting the evaluation threshold value for evaluating the evaluation site based on the amount of movement.
In determining the evaluation site, a program for constructing a coaching application is characterized in that a site where the amount of movement exceeds a predetermined reference value is determined as the evaluation site . Including a procedure for accepting a correction operation for the set evaluation threshold.
The program for constructing a coaching application according to claim 12 , wherein the evaluation threshold value is corrected according to the correction operation.

Images