JP7614504B2 - Technique recognition method, technique recognition program, and gymnastics scoring support system - Google Patents

Description

The present invention relates to a technique recognition method, a technique recognition program, and a gymnastics scoring support system.

In various sports, as technical advances in competitions progress, it is becoming increasingly difficult to fairly and accurately evaluate athletes' performances by visual inspection. For this reason, in recent years, technology that can automatically evaluate athletes' performances has been developed.

Specifically, the above technology recognizes each basic movement performed by an athlete by, for example, identifying the start and end poses of the basic movement from time-series data of the athlete's performance. Then, the above technology recognizes the type of technique performed by the athlete based on, for example, the order of each recognized basic movement (see, for example, Patent Documents 1 and 2).

International Publication No. 2019/116495 JP 2020-038440 A

For example, when an athlete performs multiple techniques in succession, the start and end poses of each technique may differ from the start and end poses when each technique is performed individually. Therefore, the above-mentioned technology may not be able to accurately recognize each basic movement performed by the athlete, and may not be able to accurately recognize the type of technique performed by the athlete.

In one aspect, the present invention aims to provide a technique recognition method, technique recognition program, and gymnastics scoring support system that enable accurate recognition of techniques performed by athletes.

In one aspect of the embodiment, a computer executes a process of acquiring a plurality of frames arranged in a time series, each of which includes a posture of a person, identifying a segment frame from the plurality of frames that satisfies a first condition, identifying a group of segment frames from the plurality of frames that consecutively satisfies a second condition, segmenting the plurality of frames into a plurality of groups by using the identified plurality of frames as segmentation points, identifying a basic movement performed by the person for each of the segmented plurality of groups, and identifying the type of technique performed by the person based on the combination and order of the basic movements for each of the identified plurality of groups.

In one aspect, it enables accurate recognition of the techniques performed by the athlete.

FIG. 1 is a diagram illustrating a configuration of an information processing system 10. As shown in FIG. FIG. 2 is a diagram illustrating the hardware configuration of the information processing device 1. As shown in FIG. FIG. 3 is a functional block diagram of the information processing device 1. FIG. 4 is a flow chart for explaining an outline of the technique recognition process in the first embodiment. FIG. 5 is a diagram for explaining a schematic specific example of the first embodiment. FIG. 6 is a diagram for explaining a schematic specific example of the first embodiment. FIG. 7 is a flow chart illustrating the details of the technique recognition process in the first embodiment. FIG. 8 is a flow chart illustrating the details of the technique recognition process in the first embodiment. FIG. 9 is a flow chart illustrating the details of the technique recognition process in the first embodiment. FIG. 10 is a flow chart illustrating the details of the technique recognition process in the first embodiment. FIG. 11 is a diagram for explaining a specific example of the time-series data 131. As shown in FIG. FIG. 12 is a diagram illustrating a specific example of the first segmentation condition information 132. FIG. 13 is a diagram illustrating a specific example of the second segmentation condition information 133. FIG. 14 is a diagram illustrating a specific example of the basic exercise condition information 134. As shown in FIG. FIG. 15 is a diagram illustrating a specific example of the technique action sequence information 135. As shown in FIG. FIG. 16 is a diagram illustrating a specific example of the first certification condition information 136. As shown in FIG. FIG. 17 is a diagram illustrating a specific example of the second certification condition information 137. As shown in FIG. FIG. 18 is a diagram illustrating a specific example of the technique recognition result information 138. As shown in FIG. FIG. 19 is a diagram illustrating a specific example according to the first embodiment. FIG. 20 is a diagram illustrating a specific example according to the first embodiment. FIG. 21 is a diagram illustrating a specific example according to the first embodiment. FIG. 22 is a diagram illustrating a specific example of the technique time information 139. As shown in FIG.

[Configuration of Information Processing System in First Embodiment]
First, a description will be given of the configuration of the information processing system 10. FIG.

The information processing system 10 has, for example, a sensor 2 that captures images of the movements of a player P (hereinafter also referred to as human P), an information processing device 1 that recognizes techniques performed by the player P, and an output device 5 such as a monitor that outputs the results of the technique recognition by the information processing device 1.

Sensor 2 is, for example, a 3D (dimension) sensor, and is placed in a position that allows each movement of athlete P during the performance to be recognized.

Specifically, the sensor 2 is, for example, a sensor that measures the distance to each observation point of the player P, and continuously generates data indicating the three-dimensional coordinates of each observation point (hereinafter also referred to as three-dimensional data) and transmits the data to the information processing device 1.

The information processing device 1 is, for example, configured with one or more physical machines, and recognizes the techniques performed by the player P by using the three-dimensional data transmitted from the sensor 2.

Specifically, the information processing device 1 generates data (hereinafter simply referred to as time-series data) in which skeletal data indicating multiple feature points constituting the skeleton of the player P is arranged in time series from, for example, the three-dimensional data continuously transmitted from the sensor 2.

The information processing device 1 then recognizes each basic exercise performed by the player P, for example, by identifying the start pose and end pose of the basic exercise from the generated time-series data. Furthermore, the information processing device 1 recognizes the type of technique performed by the player P, for example, based on the order of each recognized basic exercise.

For example, when player P performs multiple techniques in succession, the start pose and end pose of each technique may differ from the start pose and end pose when each technique is performed alone. Therefore, the information processing device 1 may not be able to accurately recognize each basic exercise performed by player P, and may not be able to accurately recognize the type of technique performed by player P.

The information processing device 1 in this embodiment therefore identifies frames (hereinafter also referred to as segment frames) that satisfy a first condition from among the multiple frames included in the time series data. Specifically, the information processing device 1 identifies, for example, from among the multiple frames included in the time series data, a frame that corresponds to any of multiple first postures pre-stored in a storage unit (not shown) as a segment frame. The multiple first postures are, for example, postures that correspond to the start pose and end pose of each basic movement.

The information processing device 1 also identifies, from among the frames included in the time series data, a plurality of frames that consecutively satisfy the second condition (hereinafter also referred to as a segment frame group). Specifically, the information processing device 1, for example, identifies, from among the frames included in the time series data, a plurality of frames that consecutively correspond to any of a plurality of second postures pre-stored in a storage unit (not shown), as a segment frame group. The plurality of second postures are, for example, postures that correspond to each pose (hereinafter also referred to as an intermediate pose) between the end pose of each technique and the start pose of the technique performed following each technique when a plurality of techniques are performed in succession.

Then, the information processing device 1 segments the multiple frames into multiple groups based on the identified segment frames and segment frame groups. Specifically, the information processing device 1 segments the multiple frames into multiple groups, for example, by setting the multiple frames as segment frames and segment frame groups, respectively, as segmentation points. Then, the information processing device 1 identifies the basic exercise performed by the player P for each of the segmented groups.

Then, the information processing device 1 identifies the type of technique included in the performance performed by the player P based on the combination and order of the basic exercises for each of the identified groups.

That is, the information processing device 1 in this embodiment identifies frames that can be determined to correspond to the start pose or end pose of each basic exercise that constitutes each technique as segment frames. Furthermore, when multiple techniques are being performed consecutively, the information processing device 1 identifies, for example, one or more frames that can be determined to correspond to the period between the end pose of the last basic exercise that constitutes the first technique and the start pose of the first basic exercise that constitutes the second technique as a segment frame group. The information processing device 1 then uses not only the segment frames but also the segment frame groups as segment points of the basic movements to identify each of the multiple basic movements included in the multiple frames.

As a result, the information processing device 1 in this embodiment is able to segment multiple frames for each frame corresponding to each basic movement, even when multiple techniques are performed consecutively, and is able to accurately recognize each of the basic movements performed by the player P. This allows the information processing device 1 to accurately recognize the type of technique performed by the player P.

[Hardware configuration of information processing system]
Next, a description will be given of the hardware configuration of the information processing system 10. FIG.

As shown in FIG. 2, the information processing device 1 has a processor, a CPU 101, a memory 102, an I/O interface 103, and a storage medium 104. Each part is connected to each other via a bus 105.

The storage medium 104 has, for example, a program storage area (not shown) that stores a program 110 for performing a process for recognizing techniques performed by the player P (hereinafter also referred to as technique recognition processing). The storage medium 104 also has, for example, a storage unit 130 (hereinafter also referred to as information storage area 130) that stores information used when performing the technique recognition processing. The storage medium 104 may be, for example, a HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The CPU 101 executes the program 110 loaded from the storage medium 104 to the memory 102 to perform technique recognition processing.

The I/O interface 103 is, for example, an interface device such as a network interface card, and is capable of accessing the sensor 2 and the output device 5.

[Functions of the information processing system]
Next, the functions of the information processing system 10 will be described.

As shown in FIG. 3, the information processing device 1 realizes various functions including a data input unit 111, a first segment identification unit 112, a second segment identification unit 113, an exercise recognition unit 114, a technique recognition unit 115, a first technique certification unit 116, a second technique certification unit 117, a data output unit 118, and a time determination unit 119, by organic cooperation between hardware such as a CPU 101 and a memory 102 and a program 110.

In addition, the information processing device 1 stores, for example, time series data 131, first segmentation condition information 132, second segmentation condition information 133, basic movement condition information 134, technique movement sequence information 135, first certification condition information 136, second certification condition information 137, technique recognition result information 138, and technique time information 139 in the information storage area 130.

The data input unit 111 continuously receives, for example, three-dimensional data (not shown) transmitted from the sensor 2. Then, the data input unit 111 generates time series data 131 from each of the three-dimensional data transmitted from the sensor 2. The time series data 131 is time series data consisting of multiple frames each including position information of multiple feature points corresponding to each of multiple parts of the player P's body. The data input unit 111 then stores the generated time series data 131 in the information storage area 130.

The first segment identification unit 112, for example, identifies a segment frame that satisfies a first condition from among the multiple frames constituting the time series data 131 stored in the information storage area 130. Specifically, the first segment identification unit 112, for example, identifies a frame that corresponds to any of the conditions included in the first segment condition information 132 stored in the information storage area 130 from among the multiple frames included in the time series data 131 as a segment frame. The first segment state information 132 is information that includes the conditions of a frame that corresponds to a segment frame.

The second segment identification unit 113, for example, identifies a segment frame group consisting of a plurality of frames that consecutively satisfy the second condition, among the plurality of frames constituting the time series data 131 stored in the information storage area 130. Specifically, the second segment identification unit 113, for example, identifies, as a segment frame group, a plurality of frames that consecutively correspond to any of the conditions included in the second segment condition information 133 stored in the information storage area 130, among the plurality of frames included in the time series data 131. The second segment state information 133 is information that includes the conditions of the frames that fall into the segment frame group.

The movement recognition unit 114 segments the multiple frames constituting the time series data 131 into multiple groups, for example, based on the segment frames identified by the first segment identification unit 112 and the segment frame groups identified by the second segment identification unit 113. Specifically, the movement recognition unit 114 segments the multiple frames constituting the time series data 131 into multiple groups, for example, by treating the multiple frames as segment frames and the segment frame groups as segment points.

Then, the exercise recognition unit 114 recognizes (identifies) the basic exercises performed by the player P for each of the segmented groups. Specifically, the exercise recognition unit 114 refers to the basic exercise condition information 134 stored in the information storage area 130, for example, and recognizes the basic exercises corresponding to each of the multiple groups. The basic exercise condition information 134 is information that includes the conditions of the group that corresponds to the basic exercise.

The motion recognition unit 114 may perform segmentation such that a segment frame is not included in either of the two groups segmented by the segment frame, or may perform segmentation such that a segment frame is included in one of the two groups segmented by the segment frame. The motion recognition unit 114 may perform segmentation such that a segment frame group is not included in either of the two groups segmented by the segment frame group, or may perform segmentation such that a segment frame group is included in one of the two groups segmented by the segment frame group.

The technique recognition unit 115 recognizes (identifies) candidates for techniques performed by the player P (hereinafter also referred to as technique candidates) from, for example, the combination and order of basic exercises for each of the multiple groups recognized by the exercise recognition unit 114. Specifically, the technique recognition unit 115, for example, refers to technique movement order information 135 stored in the information storage area 130 and recognizes technique candidates corresponding to the multiple groups. The technique movement order information 135 is information that includes conditions for the combination and order of basic exercises that correspond to the technique candidates.

For example, for each of the candidate techniques recognized by the technique recognition unit 115, the first technique certification unit 116 identifies the certification conditions (hereinafter also referred to as the third conditions) of the first certified technique that are satisfied by the posture of the player P in the basic exercise corresponding to each candidate technique. Specifically, the first technique certification unit 116, for example, refers to the first certification condition information 136 stored in the information storage area 130, and identifies a combination of the first certified technique and the certification conditions corresponding to each candidate technique. The first certification condition information 136 is information that includes the certification conditions corresponding to each of the first certified techniques.

For example, for each of the technique candidates recognized by the technique recognition unit 115, the second technique certification unit 117 determines whether the order of each technique candidate and other technique candidates performed immediately before or after each technique candidate satisfies a condition (hereinafter also referred to as the fourth condition) corresponding to the certification condition specified by the first technique certification unit 116 for each technique candidate. Then, for example, if it is determined that the order corresponding to each candidate satisfies the condition, it identifies the type of each technique candidate as the type of technique actually performed by the player P (hereinafter also referred to as the second certified technique). Specifically, the second technique certification unit 117, for example, refers to the second certification condition information 137 stored in the information storage area 130 and determines whether or not to certify each technique candidate as a second certified technique. The second certification condition information 137 is information that includes the certification conditions of the first certified technique that corresponds to the second certified technique.

The data output unit 118 outputs, for example, information indicating the type of technique identified by the second technique recognition unit 117 to the output device 5.

The time determination unit 119 determines, for example, the start time and end time of the technique identified by the second technique certification unit 117. Specifically, the time determination unit 119 determines, for example, the recognition time by the sensor 2 of the frame corresponding to the start pose of each technique or the time corresponding to the frame number of the frame corresponding to the start pose of each technique as the start time of the technique. The time determination unit 119 also determines, for example, the recognition time by the sensor 2 of the frame corresponding to the end pose of each technique or the time corresponding to the frame number of the frame corresponding to the end pose of each technique as the end time of the technique. Then, the data output unit 118 outputs, for example, the start time and end time of the technique determined by the time determination unit 119 to the output device 5 together with information indicating the technique identified by the second technique certification unit 117.

[Outline of the first embodiment]
Next, an outline of the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flow chart for explaining an outline of the technique recognition process in the first embodiment.

As shown in FIG. 4, the information processing device 1 waits, for example, until the timing for technique recognition arrives (NO in S11). The timing for technique recognition may be, for example, the timing when the operator inputs information to perform technique recognition processing via an operation terminal (not shown).

Then, when the timing for technique recognition arrives (YES in S11), the information processing device 1 acquires, for example, the time series data 131 stored in the information storage area 130 (S12).

Next, the information processing device 1 identifies segment frames that satisfy the first condition, for example, from among the multiple frames included in the time series data 131 acquired in the processing of S12 (S13).

The information processing device 1 also identifies a segment frame group consisting of multiple consecutive frames that satisfy the second condition, for example, from among the multiple frames included in the time series data 131 acquired in the processing of S12 (S14).

Furthermore, the information processing device 1 segments the multiple frames included in the time series data 131 acquired in the process of S12 into multiple groups based on, for example, the segment frame identified in the process of S13 and the segment frame group identified in the process of S14 (S15). Specifically, the information processing device 1 segments the multiple frames included in the time series data 131 acquired in the process of S12 into multiple groups by setting each of the segment frame identified in the process of S13 and the segment frame group identified in the process of S14 as segmentation points.

Then, the information processing device 1 identifies the basic exercises performed by the player P for each of the multiple groups segmented in the processing of S15 (S16).

Then, the information processing device 1 identifies the type of technique performed by the player P based on the combination and order of the basic exercises for each group identified in the processing of S15 (S17).

As a result, the information processing device 1 in this embodiment is able to segment multiple frames for each frame corresponding to each basic movement, even when multiple techniques are performed consecutively, and is able to accurately recognize each of the basic movements performed by the player P. This allows the information processing device 1 to accurately recognize the type of technique performed by the player P.

[Specific example of outline of first embodiment]
Next, a specific example of the outline of the first embodiment will be described. Figures 5 and 6 are diagrams for explaining a specific example of the outline of the first embodiment.

Specifically, the example shown in FIG. 5 shows a case where frames corresponding to times t1, t2, t3, t4, t5, t6, and t7 are identified as segment frames.

In this case, in the process of S15, the information processing device 1 segments the time series data 131 recognized between time t0 (the start time of the performance by athlete P) and time t7, for example, into time periods from time t0 to time t1, from time t1 to time t2, from time t2 to time t3, from time t3 to time t4, from time t4 to time t5, from time t5 to time t6, and from time t6 to time t7.

Next, in the processing of S16 and S17, the information processing device 1 recognizes that, for example, basic exercises (1), (2), and (3) were performed from time t0 to time t1, from time t1 to time t2, and from time t2 to time t3, respectively, and further recognizes that technique (1) in which basic exercises (1), (2), and (3) were performed in this order was performed. The information processing device 1 also recognizes that, for example, basic exercises (4), (5), and (6) were performed from time t4 to time t5, from time t5 to time t6, and from time t6 to time t7, respectively, and further recognizes that technique (2) in which basic exercises (4), (5), and (6) were performed in this order was performed.

In other words, as shown in FIG. 5, when each technique is performed independently, the information processing device 1 recognizes the type of each technique by segmenting the time series data 131 using segmentation frames as segmentation points.

On the other hand, the example shown in FIG. 6 shows a case where frames corresponding to times t11, t12, t14, t15, and t16 are identified as segment frames, and multiple frames corresponding to time period T13 are identified as a segment frame group. Specifically, the example shown in FIG. 6 shows a case where technique (1) and technique (2) described in FIG. 5 are performed consecutively.

In this case, in the process of S15, the information processing device 1 segments the time series data 131 recognized between time t10 (the start time of the performance by athlete P) and time t16, for example, into time periods from time t10 to time t11, from time t11 to time t12, from time t12 to the start time of time period T13, from the end time of time period T13 to time t14, from time t14 to time t15, and from time t15 to time t16.

Next, in the processing of S16 and S17, the information processing device 1 recognizes that, for example, basic exercises (1), (2), and (3) were performed from time t10 to time t11, from time t11 to time t12, and from time t12 to the start time of time zone T13, respectively, and further recognizes that technique (1) in which basic exercises (1), (2), and (3) were performed in this order was performed. The information processing device 1 also recognizes that, for example, basic exercises (4), (5), and (6) were performed from time zone T13 to time t14, from time t14 to time t15, and from time t15 to time t16, respectively, and further recognizes that technique (2) in which basic exercises (4), (5), and (6) were performed in this order was performed.

In other words, as shown in FIG. 6, when multiple techniques are performed consecutively, the information processing device 1 segments the time series data 131 by using not only a segment frame consisting of one frame, but also a segment frame group consisting of multiple frames.

As a result, the information processing device 1 is able to accurately recognize each of the basic movements performed by the player P, even if the start and end poses of each technique when multiple techniques are performed consecutively differ from the start and end poses of each technique when each technique is performed independently, and is able to accurately recognize the type of technique performed by the player P.

[Details of the First Embodiment]
Next, the details of the first embodiment will be described. Figs. 7 to 10 are flow charts for explaining the details of the technique recognition process in the first embodiment. Figs. 11 to 22 are diagrams for explaining the details of the technique recognition process in the first embodiment. In the following, the case where the athlete P performs a front wheel and a front floating support spinning handstand in succession will be explained, but the present invention is not limited to this. Specifically, in this embodiment, the athlete P may perform other gymnastics competitions such as a front somersault with two twists or a front somersault, or the athlete P may perform other judged competitions.

As shown in FIG. 5, the data input unit 111 waits, for example, until the timing for recognizing a technique arrives (NO in S21).

When the timing for technique recognition arrives (YES in S21), the data input unit 111 acquires, for example, one frame included in the time-series data 131 stored in the information storage area 130 in chronological order (S22). Specifically, the time-series data 131 is, for example, time-series data consisting of multiple frames each including skeletal data SK of the player P, as shown in FIG. 11.

Then, the first segment identification unit 112, for example, refers to the first segmentation condition information 132 stored in the information storage area 130, and determines whether the frame acquired in the process of S22 matches the segment frame conditions (S23). A specific example of the first segmentation condition information 132 will be described below.

[Specific example of first segment condition information]
FIG. 12 is a diagram illustrating a specific example of the first segmentation condition information 132.

The first segmentation condition information 132 shown in FIG. 12 has a "segment ID" that identifies each segmentation point, and a "segmentation condition" in which conditions corresponding to each segmentation point are set.

Specifically, in the first segmentation condition information 132 shown in FIG. 12, the information in the first row has "11" set as the "segment ID," and "head down, center of body closest to perpendicular to the ground, angle between arm and center of body > 170 degrees" set as the "segmentation condition."

In addition, in the first segmentation condition information 132 shown in FIG. 12, the information in the second line has "12" set as the "segment ID," and "head up, center of body closest to perpendicular to the ground, angle between arm and center of body > 170 degrees" set as the "segmentation condition." Explanation of other information included in FIG. 12 will be omitted.

That is, for example, if the posture of player P indicated by the skeletal data SK included in the frame acquired in the processing of S22 matches any of the conditions set in the "segmentation condition" in the first segmentation condition information 132 shown in FIG. 12, the first segment identification unit 112 determines that the frame acquired in the processing of S22 is a segmented frame.

In addition, the "segmentation condition" in each piece of information included in the first segmentation condition information 132 in FIG. 12 is set to "the center of the body is closest to perpendicular to the ground". Frames that satisfy this "the center of the body is closest to perpendicular to the ground" cannot be identified for each frame, but can be identified by, for example, comparing multiple frames that satisfy conditions other than "the center of the body is closest to perpendicular to the ground" (for example, "head down" or "angle between the arm and the center of the body > 170 degrees"). Therefore, for example, in the process of S23, the first segment identification unit 112 determines whether the frame acquired in the process of S22 satisfies a condition other than "the center of the body is closest to perpendicular to the ground" among the conditions set in the "segmentation condition" in the first segmentation condition information 132 in FIG. 12.

Returning to FIG. 7, if it is determined that the frame acquired in the processing of S22 matches the segment frame condition (YES in S23), the first segment identification unit 112 acquires, for example, one more frame in chronological order from the time series data 131 stored in the information storage area 130 (S24).

Then, the first segment identification unit 112, for example, refers to the first segmentation condition information 132 stored in the information storage area 130 and determines whether the frame acquired in the processing of S24 matches the segment frame conditions (S25).

As a result, if it is determined that the frame acquired in the processing of S22 matches the segment frame conditions (YES in S25), the first segment identification unit 112 performs the processing from S24 onwards again.

In other words, in the process of S25, the first segment identification unit 112 identifies one or more frames that consecutively satisfy conditions other than "the center of the body is closest to being perpendicular to the ground" set in the "segmentation condition" of each piece of information included in the first segmentation condition information 132 in FIG. 12, for example.

On the other hand, if it is determined that the frame acquired in the process of S22 does not match the segment frame conditions (NO in S25), the first segment identification unit 112 identifies a segment frame, for example, from the frame acquired in the immediately preceding process of S22 to the frame acquired in the immediately preceding process of S24 (S26).

Specifically, for example, by referring to the first segmentation condition information 132 described in FIG. 12, from among the frames obtained in the immediately preceding processing of S22 to the frames obtained in the immediately preceding processing of S24, a frame that satisfies the "center of the body is closest to being perpendicular to the ground" set as the "segmentation condition" in the first segmentation condition information 132 described in FIG. 12 is identified as the segmentation frame.

In other words, in the process of S26, the first segment identification unit 112 identifies, as a segment frame, a frame that further satisfies "the center of the body is closest to perpendicular to the ground" from among one or more frames that consecutively satisfy conditions other than "the center of the body is closest to perpendicular to the ground" set in the "segmentation condition" of each piece of information included in the first segmentation condition information 132 in FIG. 12.

Furthermore, if it is determined in the process of S23 that the frame acquired in the process of S22 does not match the conditions for the segment frame (NO in S23), the second segment identification unit 113, as shown in FIG. 8, for example, refers to the second segmentation condition information 133 stored in the information storage area 130, and determines whether or not the frame acquired in the process of S22 matches the conditions for the frames included in the segment frame group (S31). Specific examples of the second segmentation condition information 133 are described below.

[Specific example of second segmentation condition information]
FIG. 13 is a diagram illustrating a specific example of the second segmentation condition information 133.

The second segmentation condition information 133 shown in FIG. 13 has a "segment ID" that identifies each segmentation point, and a "segmentation condition" in which conditions corresponding to each segmentation point are set.

Specifically, in the second segmentation condition information 133 shown in FIG. 13, the information in the first line has "21" set as the "segment ID," and "segmentation condition" set as "head up, center of body perpendicular to the ground at 45 degrees forward or backward, angle between arm and center of body > 90 degrees." Explanation of other information included in FIG. 13 will be omitted.

That is, for example, if the posture of player P indicated by the skeletal data SK included in the frame acquired in the processing of S22 matches any of the information set in the "segmentation condition" in the second segmentation condition information 133 shown in FIG. 13, the second segment identification unit 113 determines that the frame acquired in the processing of S22 is a frame included in the segment frame group.

Returning to FIG. 7, if it is determined that the frame acquired in the processing of S22 does not match the conditions of the segment frame group (NO in S31), the first segment identification unit 112 performs the processing from S22 onwards again.

On the other hand, if it is determined that the frame acquired in the processing of S22 matches the segment frame condition (YES in S31), the second segment identification unit 113 acquires, for example, one more frame in chronological order from the time series data 131 stored in the information storage area 130 (S32).

Then, the second segment identification unit 113, for example, refers to the second segmentation condition information 133 stored in the information storage area 130 and determines whether the frame acquired in the processing of S32 matches the conditions of the segment frame group (S33).

As a result, if it is determined that the frame acquired in the processing of S32 matches the conditions of the segment frame group (YES in S33), the second segment identification unit 113 performs the processing from S32 onwards again.

On the other hand, if it is determined that the frame acquired in the process of S32 does not meet the segment frame conditions (NO in S33), the second segment identification unit 113 identifies, for example, a group of frames from the frame acquired in the immediately preceding process of S22 to the frame acquired in the immediately preceding process of S32 as a segment frame group (S34).

Then, after processing of S26 or S34, the movement recognition unit 114 identifies, for example, the movement of player P in a frame group (group) segmented by the segment frame or segment frame group identified in the immediately preceding processing of S26 or S34 and the segment frame or segment frame group identified in the processing of S26 or S34 performed immediately before the immediately preceding S26 or S34, as a candidate for a basic movement (hereinafter also referred to as a basic movement candidate) (S35).

Then, the exercise recognition unit 114, for example, refers to the basic exercise condition information 134 stored in the information storage area 130 and determines whether the basic exercise candidate identified in the process of S35 matches the conditions of the basic exercise (S36). A specific example of the basic exercise condition information 134 will be described below.

[Specific examples of basic exercise condition information]
FIG. 14 is a diagram illustrating a specific example of the basic exercise condition information 134. As shown in FIG.

The basic exercise condition information 134 shown in FIG. 14 includes a "basic exercise ID" that identifies each basic exercise, a "start pose" in which a segment ID corresponding to the start pose of each basic exercise is set, an "end pose" in which a segment ID corresponding to the end pose of each basic exercise is set, and a "basic exercise condition" in which a condition corresponding to each basic exercise is set.

Specifically, in the basic exercise condition information 134 shown in FIG. 14, the information in the first row has "1" set as the "basic exercise ID," "11" set as the "start pose," "12" set as the "end pose," and "0 degrees < forward rotation amount < 180 degrees, minimum waist angle > 170 degrees, minimum elbow angle > 170 degrees" set as the "basic exercise condition."

In the basic exercise condition information 134 shown in FIG. 14, the information in the second row has "2" set as the "basic exercise ID", "12" set as the "start pose", "11" and "21" set as the "end pose", and "0 degrees < forward rotation amount < 180 degrees, minimum waist angle > 170 degrees, minimum elbow angle > 170 degrees" set as the "basic exercise condition". Explanation of the other information included in FIG. 14 will be omitted.

In other words, for example, if the combination of the start pose, end pose, and state of the player P in the basic exercise candidate identified in the processing of S25 matches any of the combinations of information set in the "start pose," "end pose," and "basic exercise conditions" in the basic exercise condition information 134 shown in FIG. 14, the exercise recognition unit 114 determines that the basic exercise candidate identified in the processing of S25 matches the conditions of the basic exercise.

Returning to FIG. 7, if it is determined that the basic movement candidate identified in the process of S25 does not match the conditions for the basic movement (NO in S36), the first segment identification unit 112 performs the processes from S22 onwards again.

On the other hand, if it is determined that the basic movement candidate identified in the processing of S25 matches the conditions for a basic movement (YES in S36), the exercise recognition unit 114 stores, for example, the basic movement candidate identified in the processing of S25 as a basic movement in a basic movement list (not shown) (S37).

In this case, the exercise recognition unit 114 identifies the basic exercise candidate identified in the processing of S25 as one of the basic exercises performed by player P.

The technique recognition unit 115 then refers to the technique movement order information 135 stored in the information storage area 130, for example, and determines whether the combination and order of the basic movements stored in the basic movement list match the conditions of the technique candidates (hereinafter also referred to as technique candidates) (S38). A specific example of the technique movement order information 135 will be described below.

[Specific example of technique motion sequence information]
FIG. 15 is a diagram illustrating a specific example of the technique action sequence information 135. As shown in FIG.

The technique movement order information 135 shown in FIG. 15 includes a "technique ID" that identifies each technique, and "movement order conditions" that set conditions for the movement order corresponding to each technique (the combination and order of the "IDs" of the basic movements corresponding to each technique).

Specifically, in the technique movement order information 135 shown in FIG. 15, the information in the first row has "1" set as the "Technique ID" and "1, 2" set as the "Movement order condition."

In addition, in the technique movement order information 135 shown in FIG. 15, the information in the second row has "2" set as the "technique ID" and "3, 4, 5" set as the "movement order condition." Explanation of the other information included in FIG. 15 will be omitted.

In other words, for example, if a basic exercise with a "basic exercise ID" of "1" and a basic exercise with a "basic exercise ID" of "2" are stored in that order in the basic exercise list, the technique recognition unit 115 determines that the combination and order of the basic exercises stored in the basic exercise list matches the conditions for a technique candidate (information in the technique movement sequence information 135 shown in FIG. 15 where "Technique ID" is "1").

Returning to FIG. 7, if it is determined that the combination and order of the basic movements stored in the basic movement list does not match the conditions for the technique candidate (NO in S38), the first segment identification unit 112 performs the processing from S22 onwards again.

On the other hand, if it is determined that the combination and order of the basic exercises stored in the basic exercise list meets the conditions for the technique candidate (YES in S38), the first technique certification unit 116, as shown in FIG. 9, for example, refers to the first certification condition information 136 stored in the information storage area 130, and identifies the combination of the technique candidate determined to meet the conditions in the processing of S38 and the certification condition of that technique candidate as a combination of the first certified technique and the certification condition (S41). Specific examples of the first certification condition information 136 are described below.

[Specific example of first certification condition information]
FIG. 16 is a diagram illustrating a specific example of the first certification condition information 136. As shown in FIG.

The first certification condition information 136 shown in FIG. 16 has a "skill ID" that identifies each technique, an "implementation state" that indicates the implementation state of each technique, and a "certification condition" that sets the certification condition for the first certified technique. The "implementation state" is set to "single," which indicates that each technique is implemented alone, or "sequential," which indicates that each technique is implemented consecutively with another technique.

Specifically, in the first certification condition information 136 shown in FIG. 16, the information in the first line has "1" set as the "skill ID," "solo" set as the "implementation state," and "end pose: handstand position" set as the "certification condition."

In the first certification condition information 136 shown in FIG. 16, the information in the second line has the "Technique ID" set to "1," the "Performance state" set to "Continuous," and the "Certification condition" set to "End pose: Waist angle > 170 degrees and elbow angle > 170 degrees."

In addition, in the first certification condition information 136 shown in FIG. 16, the information in the third line has "2" set as the "skill ID," "solo" set as the "implementation state," and "end pose: handstand position" set as the "certification condition."

In addition, in the first certification condition information 136 shown in FIG. 16, the information in the fourth line has "2" set as the "skill ID," "continuous" set as the "implementation state," and "none" set as the "certification condition." Explanation of the other information included in FIG. 16 is omitted.

In other words, the first technique certification unit 116 identifies, for example, information included in the first certification condition information 136 shown in FIG. 16, information set in the "certification condition" that matches the posture of the player P in the technique candidate that was determined to meet the condition in the processing of S38.

Returning to FIG. 9, the first skill certification unit 116 stores the combination of the candidate skills and certification conditions identified in the process of S41 in a first certified skill list (not shown) as a combination of the first certified skills and certification conditions (S42).

Then, the second skill certification unit 117 determines whether or not all frames have been acquired in the processing of S22 (S43).

As a result, if it is determined that not all frames have been acquired in the processing of S22 (NO in S43), the first segment identification unit 112 performs the processing from S22 onwards again.

On the other hand, if it is determined in the processing of S22 that all frames have been acquired (YES in S43), the second certified technique 117 acquires one combination of the first certified technique and the certification conditions stored in the first certified technique list (S44).

Then, the second technique certification unit 117, for example, refers to the second certification condition information 137 stored in the information storage area 130, and determines whether the first certified technique and the certification conditions included in the combination obtained in the processing of S44 match the conditions of the second certified technique (S45).

Specifically, the second technique certification unit 117, for example, refers to the first certification condition information 136 described in FIG. 16 and determines whether the "implementation state" corresponding to the first certified technique included in the combination obtained in the processing of S44 is "continuous." Then, for example, when the "implementation state" corresponding to the first certified technique included in the combination obtained in the processing of S44 is "continuous," the second technique certification unit 117 determines whether the first certified technique and the certification conditions included in the combination obtained in the processing of S44 match the conditions of the second certified technique.

This allows the second technique certification unit 117 to improve the accuracy of technique recognition when techniques are performed consecutively. Specific examples of the second certification condition information 137 are described below.

[Specific example of second certification condition information]
FIG. 17 is a diagram illustrating a specific example of the second certification condition information 137. As shown in FIG.

The second certification condition information 137 shown in FIG. 17 includes a "skill ID" that identifies each technique, an "implementation state" that indicates the implementation state of each technique, "certification conditions (1)" in which the certification conditions for the first certified technique are set, and "certification conditions (2)" in which the conditions for the second certified technique are set.

Specifically, in the second certification condition information 137 shown in FIG. 17, the information in the first line has "1" set as the "skill ID," "solo" set as the "implementation state," "ending pose: handstand position" set as the "certification condition (1)," and "none" set as the "certification condition (2)."

In the second certification condition information 137 shown in FIG. 17, the information in the second line has "2" set as the "skill ID", "continuous" set as the "implementation state", "end pose: waist angle > 170 degrees, elbow angle > 170 degrees" set as the "certification condition (1)", and "next skill: 2" set as the "certification condition (2)", indicating that the "skill ID" of the next skill to be performed is "2". Explanation of the other information included in FIG. 17 will be omitted.

That is, the second technique certification unit 117 determines whether the first certified technique and certification conditions included in the combination obtained in the processing of S44 match any of the combinations of information set in the "Technique ID" and "Certification Condition (1)" in the second certification condition information 137 shown in FIG. 17. If it is determined that there is a match, the second technique certification unit 117 further determines whether the first certified techniques before and after the first certified technique included in the combination obtained in the processing of S44 (the first certified techniques stored before and after in the first certified technique list) match the information set in the "Certification Condition (2)".

Returning to FIG. 9, if it is determined that the first certified technique and the certification conditions included in the combination obtained in the processing of S44 match the conditions for the second certified technique (YES in S45), the second technique certification unit 117, for example, certifies the first certified technique included in the combination obtained in the processing of S44 as a second certified technique (S46).

On the other hand, if it is determined that the first certified technique and the certification conditions included in the combination obtained in the processing of S44 do not match the conditions of the second certified technique (YES in S45), the second technique certification unit 117 does not perform the processing of S45.

In addition, in the process of S42, the first skill certification unit 116 may include the implementation state of the candidate skills identified in the process of S41 in a combination and store it in the first certified skill list. Then, in the process of S45, the second skill certification unit 117 may determine whether the implementation state included in the combination obtained in the process of S44 is "continuous."

Then, it is determined whether all combinations have been obtained in the processing of S44 (S47).

As a result, if it is determined in the processing of S44 that not all combinations have been obtained (NO in S47), the second skill certification unit 117 performs the processing from S44 onwards again.

On the other hand, if it is determined in the process of S44 that all combinations have been acquired (YES in S47), the data output unit 118 outputs, for example, information indicating the type of the second certified technique identified in the process of S45 to the output device 5 as technique recognition result information 138 (S48). Specific examples of technique recognition result information 138 are described below.

[Example of technique recognition result information]
FIG. 18 is a diagram illustrating a specific example of the technique recognition result information 138. As shown in FIG.

The technique recognition result information 138 shown in FIG. 18 has an "order" that indicates the order in which each technique is performed, and a "recognition result" that sets the name of each technique.

Specifically, in the technique recognition result information 138 shown in FIG. 18, the information in the first line has "1" set as the "order" and "front wheel" set as the "recognition result."

In addition, in the technique recognition result information 138 shown in FIG. 18, the information in the second line has "2" set as the "order" and "forward floating support spinning handstand" set as the "recognition result."

That is, the data output unit 118 outputs, for example, the name of the technique recognized as the second certified technique in the processing of S46 (the name corresponding to the "technique ID" of the second certified technique recognized in the processing of S46).

In addition, the data output unit 118 may also output, for example, information about the first certified technique that was not certified as the second certified technique in the processing of S46.

[Specific example (1) of details in the first embodiment]
Next, a detailed example of the first embodiment will be described. Figures 19 to 21 are diagrams for explaining a specific example of the first embodiment.

First, a specific example will be described in which athlete P performs the front wheel alone. Figure 19 shows states ST11, ST12, ST13, ST14, ST15, ST16, and ST17, which are arranged in chronological order while athlete P is performing the routine.

In the example shown in FIG. 19, state ST11 is a state in which "head is down", "center of body is closest to perpendicular to the ground", and "angle between arms and center of body > 170 degrees". In the first row of the first segmentation condition information 132 described in FIG. 12, "head is down, center of body is closest to perpendicular to the ground, angle between arms and center of body > 170 degrees" is set as the "segmentation condition". Therefore, for example, in the process of S23, the first segment identification unit 112 determines that the information in the first row of the first segmentation condition information 132 described in FIG. 12 (information in which "segment ID" is "11") matches the frame corresponding to state ST11.

Similarly, in the process of S23, the first segment identification unit 112 determines that the information in the second row of the first segment condition information 132 described in FIG. 12 (information in which the "segment ID" is "12") matches the frame corresponding to state ST14.

In the first row of the basic exercise condition information 134 described in FIG. 14, "11" is set as the "start pose", "12" is set as the "end pose", and "0 degrees < forward rotation amount < 180 degrees, minimum waist angle > 170 degrees, minimum elbow angle > 170 degrees" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the first row of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "1") matches each frame from state ST11 to state ST14.

Similarly, in the process of S36, the exercise recognition unit 114 determines that the information in the second row of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "2") matches each frame from state ST14 to state ST17.

Next, in the first line of the technique action sequence information 135 described in FIG. 15, "1, 2" is set as the "action sequence condition." Therefore, in the process of S38, the technique recognition unit 115 determines that the information in the first line of the technique action sequence information 135 described in FIG. 15 (information in which the "technique ID" is "1") matches each frame from state ST11 to state ST17.

Furthermore, in the first line of the first certification condition information 136 described in FIG. 16, "Ending pose: handstand position" is set as the "certification condition". And the position of the player P in state ST17 corresponds to the handstand position. Also, in the first line of the second certification condition information 137 described in FIG. 17, "none" is set as the "certification condition (2)". Therefore, in the process of S46, the second technique certification unit 117 certifies that the "technique ID" corresponding to each frame from state ST11 to state ST17 is "1" and the name is "front wheel".

[Specific example (2) of details in the first embodiment]
Next, a specific example of a case where the athlete P is performing a forward floating support spinning handstand alone will be described. Fig. 20 is a diagram showing states ST21, ST22, ST23, ST24, ST25, and ST26 in which the states of the athlete P while performing the routine are arranged in chronological order.

In the example shown in FIG. 20, state ST21 is a state in which "head is down", "center of body is closest to perpendicular to the ground", and "angle between arms and center of body > 170 degrees". In the first row of the first segmentation condition information 132 described in FIG. 12, "head is down, center of body is closest to perpendicular to the ground, angle between arms and center of body > 170 degrees" is set as the "segmentation condition". Therefore, in the process of S23, the first segment identification unit 112 determines that the information in the first row of the first segmentation condition information 132 described in FIG. 12 (information in which "segment ID" is "11") matches the frame corresponding to state ST21.

Similarly, in the process of S23, the first segment identification unit 112 determines that the information in the third row of the first segment condition information 132 described in FIG. 12 (information in which the "segment ID" is "13") matches the frame corresponding to state ST24.

In the third line of the basic exercise condition information 134 described in FIG. 14, "11" is set as the "start pose", "13" is set as the "end pose", and "0 degrees < forward rotation amount" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the third line of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "3") matches each frame from state ST21 to state ST24.

Similarly, in the process of S36, the exercise recognition unit 114 determines that the information in the fourth row of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "4") matches each frame from state ST24 to state ST25. Also, in the process of S36, the exercise recognition unit 114 determines that the information in the fifth row of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "5") matches each frame from state ST25 to state ST26.

Here, "3, 4, 5" is set as the "movement order condition" in the second line of the technique action sequence information 135 described in FIG. 15. Therefore, in the process of S38, the technique recognition unit 115 determines that the information in the second line of the technique action sequence information 135 described in FIG. 15 (information in which the "technique ID" is "2") matches each frame from state ST21 to state ST26.

Furthermore, in the third line of the first certification condition information 136 described in FIG. 16, "Ending pose: handstand position" is set as the "certification condition". And the position of the player P in state ST26 corresponds to the handstand position. Also, in the third line of the second certification condition information 137 described in FIG. 17, "none" is set as the "certification condition (2)". Therefore, in the processing of S46, the second technique certification unit 117 certifies that the "technique ID" corresponding to each frame from state ST21 to state ST27 is "2" and the name is "forward floating supported spinning handstand".

[Specific example (3) of the details of the first embodiment]
Next, a specific example will be described in which the athlete P performs a front wheel and a front floating support spinning handstand in succession. Fig. 21 is a diagram showing states ST31, ST32, ST33, ST34, ST35, ST36, ST37, ST38, ST39, ST40, and ST41 in which the states of the athlete P while performing the routine are arranged in chronological order.

Each of states ST31 to ST36 shown in FIG. 21 corresponds to each of states ST11 to ST16 described in FIG. 19, but state ST37 shown in FIG. 21 does not correspond to state ST17 described in FIG. 19. Also, each of states ST38 to ST41 shown in FIG. 21 corresponds to each of states ST23 to ST26 described in FIG. 20, but state ST37 shown in FIG. 21 does not correspond to states ST11 and ST12 described in FIG. 11.

In other words, when athlete P performs a front wheel and a forward floating supported spinning handstand consecutively, he or she performs the routine without passing through the end pose (state ST17) when performing a front wheel alone, or the start pose (state ST21) when performing a front floating supported spinning handstand alone. Therefore, in the example shown in FIG. 21, the end pose of the front wheel corresponds to state ST37, and the start pose of the forward floating supported spinning handstand corresponds to state ST38.

21, state ST37 and state ST38 are respectively a "head down" state, a "center of body perpendicular to the ground at 45 degrees forward or backward", and an "angle between the arm and the center of body > 90 degrees". In the first line of the second segmentation condition information 133 described in FIG. 13, "head down, center of body perpendicular to the ground at 45 degrees forward or backward, angle between the arm and the center of body > 90 degrees" is set as the "segmentation condition". Therefore, in the process of S24, the second segment identification unit 113 determines that the information in the first line of the second segmentation condition information 133 described in FIG. 13 (information in which the "segment ID" is "21") matches the frames corresponding to state ST37 and state ST38.

Next, in the first line of the basic exercise condition information 134 described in FIG. 14, "11" is set as the "start pose", "12" is set as the "end pose", and "0 degrees < forward rotation amount < 180 degrees, minimum waist angle > 170 degrees, minimum elbow angle > 170 degrees" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the first line of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "1") matches each frame from state ST31 to state ST34.

Similarly, in the second row of the basic exercise condition information 134 described in FIG. 14, "12" is set as the "start pose", "11" or "21" is set as the "end pose", and "0 degrees < forward rotation amount < 180 degrees, minimum waist angle > 170 degrees, minimum elbow angle > 170 degrees" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the second row of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "2") matches each frame from state ST34 to state ST37.

Therefore, in processing S46, the second technique certification unit 117 certifies that the "Technique ID" corresponding to each frame from state ST31 to state ST37 is "1" and the name is "Front Wheel."

Next, in the third line of the basic exercise condition information 134 described in FIG. 14, "11" or "21" is set as the "start pose", "13" is set as the "end pose", and "0 degrees < forward rotation amount" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the third line of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "1") matches each frame from state ST38 to state ST39.

Similarly, in the fourth line of the basic exercise condition information 134 described in FIG. 14, "13" is set as the "start pose", "14" is set as the "end pose", and "0 degrees < forward rotation amount < 180 degrees" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the fourth line of the basic exercise condition information 134 described in FIG. 14 (information in which the "basic exercise ID" is "4") matches each frame from state ST39 to state ST40. Also, in the fifth line of the basic exercise condition information 134 described in FIG. 14, "14" is set as the "start pose", "11" is set as the "end pose", and "0 degrees < forward rotation amount < 180 degrees" is set as the "basic exercise condition". Therefore, in the process of S36, the exercise recognition unit 114 determines that the information in the fifth row of the basic exercise condition information 134 described in FIG. 14 (information in which "basic exercise ID" is "5") matches each frame from state ST40 to state ST41.

Therefore, in the process of S46, the second technique certification unit 117 certifies that the "Technique ID" corresponding to each frame from state ST38 to state ST41 is "2" and the name is "Forward Float Support Spinning Handstand."

In this manner, the information processing device 1 in this embodiment identifies segment frames that satisfy the first condition from among the multiple frames included in the time series data 131. In addition, the information processing device 1 identifies a group of segment frames that consecutively satisfy the second condition from among the multiple frames included in the time series data 131.

Then, the information processing device 1 segments the multiple frames into multiple groups based on the identified segment frames and segment frame groups. Specifically, the information processing device 1 segments the multiple frames into multiple groups, for example, by setting the multiple frames as segment frames and segment frame groups, respectively, as segmentation points. Then, the information processing device 1 identifies the basic exercise performed by the player P for each of the segmented groups.

Then, the information processing device 1 recognizes the techniques included in the performance performed by the player P based on the combination and order of the basic exercises for each of the identified groups.

In other words, the information processing device 1 in this embodiment identifies frames that can be determined to correspond to the start pose or end pose of each basic exercise that constitutes each technique as segment frames. Furthermore, when multiple techniques are being performed consecutively, the information processing device 1 identifies multiple frames that can be determined to correspond to the period between, for example, the end pose of the last basic exercise that constitutes the first technique and the start pose of the first basic exercise that constitutes the second technique as a segment frame group. The information processing device 1 then uses not only the segment frames but also the segment frame groups as segment points of the basic movements to identify each of the multiple basic movements included in the multiple frames.

As a result, the information processing device 1 in this embodiment is able to segment multiple frames for each frame corresponding to each basic movement, even when multiple techniques are performed consecutively, and is able to accurately recognize each of the basic movements performed by the player P. This allows the information processing device 1 to accurately recognize the type of technique performed by the player P.

In addition, in the process of S48, the information processing device 1 may output information including the start time and end time of each technique (hereinafter also referred to as technique time information 139) in addition to the technique recognition result information 138. Below, the process of S48 when technique time information 139 is output will be described.

[Details of the process of S48]
FIG. 10 is a flow chart illustrating the details of the process of S48.

As shown in FIG. 10, the time determination unit 119 acquires one of the second certified techniques identified in the processing of S46 (S51).

Then, the time determination unit 119 identifies the start time and end time of the second certified technique obtained in the processing of S51 (S52).

Specifically, the time determination unit 119, for example, identifies the recognition time by the sensor 2 of the first frame corresponding to the second certified technique obtained in the processing of S51 as the start time of the second certified technique obtained in the processing of S51. Also, the time determination unit 119, for example, identifies the recognition time by the sensor 2 of the last frame corresponding to the second certified technique obtained in the processing of S51 as the end time of the second certified technique obtained in the processing of S51.

The second skill certification unit 117 may, for example, in the process of S46, store a combination of the start time and end time of the second certified skill in a time list (not shown). The time determination unit 119 may, for example, in the processes of S51 and S52, obtain one combination of the start time and end time of the second certified skill from the time list.

Next, the time determination unit 119 determines whether or not the first frame corresponding to the second certified technique obtained in the processing of S51 is segmented by a segment frame group (hereinafter also referred to as the first segment frame group) (S53).

As a result, if it is determined that the first frame corresponding to the second certified technique obtained in the processing of S51 is segmented by the first segment frame group (YES in S53), the time determination unit 119 sets the start time of the second certified technique obtained in the processing of S51 as the start time of the first segment frame group (S54).

On the other hand, if it is determined that the first frame corresponding to the second certified technique obtained in the process of S51 is not segmented by the first segment frame group (NO in S53), the time determination unit 119 does not perform the process of S54.

Next, the time determination unit 119 determines whether or not the last frame corresponding to the second certified technique obtained in the processing of S51 is segmented by a segment frame group (hereinafter also referred to as the second segment frame group) (S55).

As a result, if it is determined that the last frame corresponding to the second certified technique obtained in the processing of S51 is segmented by the second segment frame group (YES in S55), the time determination unit 119 sets the end time of the second certified technique obtained in the processing of S51 as the end time of the second segment frame group (S56).

On the other hand, if it is determined that the first frame corresponding to the second certified technique obtained in the process of S51 is not segmented by the second segment frame group (NO in S55), the time determination unit 119 does not perform the process of S56.

Then, the data output unit 118 outputs, for example, technique time information 139 including the start time identified in the process of S52 (the start time changed in the process of S54) and the end time identified in the process of S52 (the end time changed in the process of S56) together with the technique recognition result information 138 to the output device 5 (S57). A specific example of technique time information 139 will be described below.

[Examples of skill time information]
Fig. 22 is a diagram for explaining a specific example of the technique time information 139. Specifically, Fig. 22 is a diagram for explaining a specific example of information including the technique recognition result information 138 and the technique time information 139.

The information shown in FIG. 22 includes "Order," which indicates the order in which each technique is performed, "Recognition Result," which sets the name of each technique, "Start Time," which sets the start time of each technique, and "End Time," which sets the end time of each technique.

Specifically, in the information shown in FIG. 22, the information in the first row has "1" set as the "sequence," "front wheels" set as the "recognition result," "AA:AA:AA.AAA" set as the "start time," and "BB:BB:BB.BBB" set as the "end time."

In addition, in the information shown in FIG. 22, the information in the second row has "2" set as the "order", "forward floating support spinning handstand" set as the "recognition result", "CC:CC:CC.CCC" set as the "start time", and "DD:DD:DD.DDD" set as the "end time".

This enables the information processing device 1 to accurately identify and output the start and end times of each technique, even when the player P performs techniques consecutively.

The technique recognition process in this embodiment may be used not only to score the performances of the player P, but also to recognize the types of movements made by humans, for example, in the field of video production such as animation.

The above embodiments can be summarized as follows:

(Appendix 1)
Acquire multiple frames in time series, each of which contains a human pose;
Identifying a segment frame that satisfies a first condition from among the plurality of frames;
Identifying a group of segmented frames that consecutively satisfy a second condition from among the plurality of frames;
The specified plurality of frames are segmented into a plurality of groups by using the segmented frames and the segmented frame group as segmentation points;
Identifying a basic movement performed by the human for each of the segmented groups;
identifying a type of technique performed by the human based on the identified combination and sequence of the basic movements for each of the plurality of groups;
A technique recognition method characterized in that processing is executed by a computer.

(Appendix 2)
In Appendix 1,
Each of the plurality of frames includes position information of a plurality of feature points corresponding to each of a plurality of parts of the human body.
A technique recognition method comprising:

(Appendix 3)
In Appendix 1,
In the process of identifying the segment frame, a frame corresponding to any one of a plurality of first orientations stored in a storage unit is identified as the segment frame among the plurality of frames.
A technique recognition method comprising:

(Appendix 4)
In Appendix 1,
In the process of identifying the segment frame group, a plurality of frames that consecutively correspond to any one of a plurality of second orientations stored in a storage unit are identified as the segment frame group among the plurality of frames.
A technique recognition method comprising:

(Appendix 5)
In Appendix 1,
In the process of identifying the basic movements, the basic movements corresponding to each of the plurality of groups are identified according to a posture of the person in one or more frames included in each of the plurality of groups.
A technique recognition method comprising:

(Appendix 6)
In Appendix 1,
In the process of recognizing the type of technique, one or more technique candidates corresponding to the plurality of groups are identified from the combination and sequence of the basic movements for each of the plurality of groups;
identifying a third condition that is satisfied by a posture of the human in the basic motion corresponding to each of the identified one or more technique candidates;
For each of the one or more identified technique candidates, if an order between each technique candidate and another technique candidate performed immediately before or after each technique candidate satisfies a fourth condition corresponding to the third condition, the type of each technique candidate is identified as the type of the technique performed by the human.
A technique recognition method comprising:

(Appendix 7)
In Appendix 1, further:
A technique recognition method, characterized in that a process is executed by a computer to output information indicating the technique to an output device.

(Appendix 8)
In Appendix 7:
In the output process, a start time and an end time of each of the techniques are output.
A technique recognition method comprising:

(Appendix 9)
In Appendix 8:
In the output process,
When the plurality of groups corresponding to the technique and the plurality of groups corresponding to a first technique executed immediately before the technique are segmented by the segment frame group, the start time of the technique is set to the start time of the segment frame group between the technique and the first technique,
When the group corresponding to the technique and the group corresponding to a second technique executed immediately after the technique are segmented by the segment frame group, the end time of the technique is set to the end time of the segment frame group between the technique and the second technique.
A technique recognition method comprising:

(Appendix 10)
Acquire multiple frames in time series, each of which contains a human pose;
Identifying a segment frame that satisfies a first condition from among the plurality of frames;
Identifying a group of segmented frames that consecutively satisfy a second condition from among the plurality of frames;
The specified plurality of frames are segmented into a plurality of groups by using the segmented frames and the segmented frame group as segmentation points;
Identifying a basic movement performed by the human for each of the segmented groups;
identifying a type of technique performed by the human based on the identified combination and sequence of the basic movements for each of the plurality of groups;
A technique recognition program for causing a computer to execute a process.

(Appendix 11)
In Supplementary Note 10,
In the process of recognizing the type of technique, one or more technique candidates corresponding to the plurality of groups are identified from the combination and sequence of the basic movements for each of the plurality of groups;
identifying a third condition that is satisfied by a posture of the human in the basic motion corresponding to each of the identified one or more technique candidates;
For each of the one or more identified technique candidates, if an order between each technique candidate and another technique candidate performed immediately before or after each technique candidate satisfies a fourth condition corresponding to the third condition, the type of each technique candidate is identified as the type of the technique performed by the human.
A technique recognition program.

(Appendix 12)
A gymnastics scoring support system including a sensor and an information processing device,
The sensor recognizes a human posture,
The information processing device includes:
acquiring a plurality of frames arranged in a time series, each frame including a posture of the person recognized by the sensor;
Identifying a segment frame that satisfies a first condition from among the plurality of frames;
Identifying a group of segmented frames that consecutively satisfy a second condition from among the plurality of frames;
The specified plurality of frames are segmented into a plurality of groups by using the segmented frames and the segmented frame group as segmentation points;
Identifying a basic movement performed by the human for each of the segmented groups;
identifying a type of technique performed by the human based on the identified combination and sequence of the basic movements for each of the plurality of groups;
A gymnastics scoring support system characterized by:

(Appendix 13)
In Appendix 12,
The information processing device includes:
Identifying one or more candidate techniques corresponding to the plurality of groups from the combination and sequence of the basic movements for each of the plurality of groups;
identifying a third condition that is satisfied by a posture of the human in the basic motion corresponding to each of the identified one or more technique candidates;
For each of the one or more identified technique candidates, if an order between each technique candidate and another technique candidate performed immediately before or after each technique candidate satisfies a fourth condition corresponding to the third condition, the type of each technique candidate is identified as the type of the technique performed by the human.
A gymnastics scoring support system characterized by:

1: Information processing device 2: Sensor 5: Operation terminal 10: Information system P: Player

Claims (11)

Acquire multiple frames in time series, each of which contains a human pose;
Identifying a segment frame that satisfies a first condition from among the plurality of frames;
Identifying a group of segmented frames that consecutively satisfy a second condition from among the plurality of frames;
The specified plurality of frames are segmented into a plurality of groups by using the segmented frames and the segmented frame group as segmentation points;
Identifying a basic movement performed by the human for each of the segmented groups;
identifying a type of technique performed by the human based on the identified combination and sequence of the basic movements for each of the plurality of groups;
A technique recognition method characterized in that processing is executed by a computer. In claim 1,
Each of the plurality of frames includes position information of a plurality of feature points corresponding to each of a plurality of parts of the human body.
A technique recognition method comprising: In claim 1,
In the process of identifying the segment frame, a frame corresponding to any one of a plurality of first orientations stored in a storage unit is identified as the segment frame among the plurality of frames.
A technique recognition method comprising: In claim 1,
In the process of identifying the segment frame group, a plurality of frames that consecutively correspond to any one of a plurality of second orientations stored in a storage unit are identified as the segment frame group among the plurality of frames.
A technique recognition method comprising: In claim 1,
In the process of identifying the basic movements, the basic movements corresponding to each of the plurality of groups are identified according to a posture of the person in one or more frames included in each of the plurality of groups.
A technique recognition method comprising: In claim 1,
In the process of recognizing the type of technique, one or more technique candidates corresponding to the plurality of groups are identified from the combination and sequence of the basic movements for each of the plurality of groups;
identifying a third condition that is satisfied by a posture of the human in the basic motion corresponding to each of the identified one or more technique candidates;
For each of the one or more identified technique candidates, if an order between each technique candidate and another technique candidate performed immediately before or after each technique candidate satisfies a fourth condition corresponding to the third condition, the type of each technique candidate is identified as the type of the technique performed by the human.
A technique recognition method comprising: In claim 1, further comprising:
A technique recognition method, characterized in that a process is executed by a computer to output information indicating the technique to an output device. In claim 7,
In the output process, a start time and an end time of each of the techniques are output.
A technique recognition method comprising: In claim 8,
In the output process,
When the group corresponding to the technique and the group corresponding to a first technique executed immediately before the technique are segmented by the segment frame group, the start time of the technique is set to the start time of the segment frame group between the technique and the first technique,
When the group corresponding to the technique and the group corresponding to a second technique executed immediately after the technique are segmented by the segment frame group, the end time of the technique is set to the end time of the segment frame group between the technique and the second technique.
A technique recognition method comprising: Acquire multiple frames in time series, each of which contains a human pose;
Identifying a segment frame that satisfies a first condition from among the plurality of frames;
Identifying a group of segmented frames that consecutively satisfy a second condition from among the plurality of frames;
The specified plurality of frames are segmented into a plurality of groups by using the segmented frames and the segmented frame group as segmentation points;
Identifying a basic movement performed by the human for each of the segmented groups;
identifying a type of technique performed by the human based on the identified combination and sequence of the basic movements for each of the plurality of groups;
A technique recognition program for causing a computer to execute a process. A gymnastics scoring support system including a sensor and an information processing device,
The sensor recognizes a human posture,
The information processing device includes:
acquiring a plurality of frames arranged in a time series, each frame including a posture of the person recognized by the sensor;
Identifying a segment frame that satisfies a first condition from among the plurality of frames;
Identifying a group of segmented frames that consecutively satisfy a second condition from among the plurality of frames;
The specified plurality of frames are segmented into a plurality of groups by using the segmented frames and the segmented frame group as segmentation points;
Identifying a basic movement performed by the human for each of the segmented groups;
identifying a type of technique performed by the human based on the identified combination and sequence of the basic movements for each of the plurality of groups;
A gymnastics scoring support system characterized by:

Images