JP7656920B2 - Exercise evaluation system - Google Patents

Description

Applicable under Article 30, Paragraph 2 of the Patent Act. Published on the website "https://youtu.be/lsrAgAKVxMA" on November 1, 2018.

The present invention relates to an exercise evaluation system.

Conventional analytical techniques in the field of physical movement include a technique for extracting repeated movement patterns in a video, comparing the movement patterns at multiple positions in the extracted video at different time positions from a specific viewpoint, and outputting the comparison results as a video or still image (Patent Document 1). In sports, for example, a technique for analyzing a golf wing using a motion sensor by attaching a device to the user's body (Patent Document 2), and a technique for capturing images of each stage of a pitch (placement of the pivot foot, horizontal rotation of the trunk) with two cameras and calculating the transition time (Patent Document 3).

JP 2017-229081 A JP 2010-068947 A JP 2018-114138 A

However, all of the technologies described in Patent Documents 1 to 3 require equipment and resources such as multiple video images, specific devices, and multiple cameras, making it difficult for a single user to use these devices to analyze physical movements and understand the analysis results.

The present invention was made in light of this background, and aims to provide a technology that can easily evaluate physical movement.

The main invention of the present invention for solving the above problems is a system for evaluating body movement, comprising a reference value storage unit that stores a reference value relating to the position of at least one of the body parts, a body part identification unit that analyzes an image of the body to identify the body part, and an evaluation unit that compares the position of the body part in the image with the reference value to determine an evaluation value of the movement.

Other problems and solutions disclosed in this application will be made clear in the description of the embodiments of the invention and the drawings.

The present invention makes it easy to evaluate physical exercise.

1 is a diagram illustrating an example of the overall configuration of an exercise evaluation system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal 10. FIG. 2 is a diagram illustrating an example of a software configuration of the user terminal 10. 4 is a diagram showing an example of the configuration of physical information stored in a physical information storage unit 130. FIG. 13 is a diagram showing an example of the configuration of a rating request transmitted by a rating request transmitting unit 112 to a server device 20. FIG. 13 is a diagram showing an example of the configuration of evaluation information received by an evaluation information receiving unit 113 from a server device 20. FIG. 13 is a diagram showing an example of the configuration of an improvement measure request transmitted by an improvement measure request transmitting unit 116 to a server device 20. FIG. FIG. 13 is a diagram illustrating an example of a configuration of improvement measure information. FIG. 2 illustrates an example of a hardware configuration of a server device 20. FIG. 2 is a diagram illustrating an example of a software configuration of the server device 20. 4 is a diagram showing an example of the configuration of image information stored in an image data storage unit 231. FIG. 13 is a diagram showing an example of the configuration of reference information stored in a reference information storage unit 232. FIG. 13 is a diagram showing an example of the configuration of evaluation condition information stored in an evaluation condition information storage unit 233. FIG. FIG. 13 is a diagram showing an example of the configuration of improvement condition information stored in the improvement condition storage unit 234. FIG. 2 is a diagram showing a flow of processes executed in the exercise evaluation system of the present embodiment. 13A and 13B are diagrams illustrating a case where a moving image is captured in a pitching mode. FIG. 13 is a diagram showing an example of displaying an image of a checkpoint. FIG. 13 is a diagram showing an example of a video evaluation screen in a weightlifting mode. FIG. 13 is a diagram showing an example of a video evaluation screen in a weightlifting mode. FIG. 13 is a diagram showing an example of a video evaluation screen in a weightlifting mode.

The details of the embodiment of the present invention will be listed and explained. The exercise evaluation system according to the embodiment of the present invention has the following configuration.

[Item 1]
A system for evaluating physical exercise, comprising:
a reference value storage unit that stores a reference value related to the position of at least one of the parts;
a part identification unit that identifies a part of the body by analyzing an image of the body;
an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value of the movement;
An exercise evaluation system comprising:
[Item 2]
Item 1, an exercise evaluation system comprising:
the evaluation unit determines the evaluation value according to a distance from the reference value to the position;
An exercise evaluation system comprising:
[Item 3]
Item 1, an exercise evaluation system comprising:
the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion;
the part identification unit identifies each of the first to third parts from the image;
the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value;
An exercise evaluation system comprising:
[Item 4]
Item 1, an exercise evaluation system comprising:
a comment storage unit that stores a comment about the part in association with a condition for the part and the position;
a comment output unit that outputs the comment corresponding to the condition satisfied by the position;
The exercise evaluation system further comprising:
[Item 5]
Item 1, an exercise evaluation system comprising:
an improvement information storage unit that stores advice on an improvement measure related to the exercise in association with a goal of improvement related to the exercise and a condition for the position;
an advice output unit that outputs the advice corresponding to the condition satisfied by the position;
The exercise evaluation system further comprising:

The exercise evaluation system according to one embodiment of the present invention is intended to evaluate the physical exercise of a user. Examples of the user's physical exercise include baseball pitching and weightlifting. The exercise evaluation system of this embodiment identifies body parts from images (which may be still images or moving images, but in this embodiment, moving images) captured of the user performing physical exercise, and evaluates the posture and movement of the body based on the absolute position of the part and the relative positional relationships of multiple parts.

Figure 1 is a diagram showing an example of the overall configuration of an exercise evaluation system according to this embodiment. As shown in the figure, the exercise evaluation system according to this embodiment includes a user terminal 10 and a server device 20. The user terminal 10 and the server device 20 are communicatively connected to each other via a communication network 30. The communication network 30 is, for example, the Internet or a LAN (Local Area Network), and is constructed using a public telephone line network, a dedicated telephone line network, a mobile phone line network, Ethernet (registered trademark), a wireless communication path, etc.

The user terminal 10 is a computer operated by a user performing physical exercise or by his/her supporter. The user terminal 10 is, for example, a smartphone, a tablet computer, a personal computer, etc. The user terminal 10 is equipped with an imaging device such as a camera, which can capture images of the user's body during exercise. In this embodiment, video images of the user's body during exercise are transmitted from the user terminal 10 to the server device 20.

The server device 20 is a computer that evaluates physical exercise. The server device 20 is, for example, a workstation, a personal computer, or a virtual computer that is logically realized by cloud computing. The server device 20 receives video images captured by the user terminal 10 and analyzes the received video images to evaluate the physical exercise. The server device 20 also makes suggestions regarding measures to improve the physical exercise. Details of the evaluation of physical exercise and the suggestions of measures to improve the physical exercise will be described later.

2 is a diagram showing an example of the hardware configuration of the user terminal 10. The user terminal 10 includes a CPU 101, a memory 102, a storage device 103, a communication interface 104, a touch panel display 105, and a camera 106. The storage device 103 is, for example, a hard disk drive, a solid state drive, or a flash memory that stores various data and programs. The communication interface 104 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The touch panel display 105 is a device for inputting and outputting data. The user terminal 10 may further include input devices such as a keyboard, a mouse, buttons, and a microphone, and output devices such as a speaker and a printer.

Figure 3 is a diagram showing an example of the software configuration of the user terminal 10. The user terminal 10 includes functional units, namely, an imaging unit 111, an evaluation request sending unit 112, an evaluation information receiving unit 113, an evaluation display unit 114, a checkpoint display unit 115, an improvement request sending unit 116, an improvement information receiving unit 117, and an improvement display unit 118, as well as storage units, namely, a physical information storage unit 130, an image storage unit 131, an evaluation information storage unit 132, and an improvement storage unit 133.

Each of the above functional units is realized by the CPU 101 of the user terminal 10 reading a program stored in the storage device 103 into the memory 102 and executing it, and each of the above storage units is realized as part of the storage area provided by the memory 102 and storage device 103 of the user terminal 10.

The imaging unit 111 captures images of the user's body while exercising. The imaging unit 111 can capture video images of the body by controlling the camera 106. The user or the user's supporter places the user terminal 10 on a flat surface, points the optical axis of the camera 106 toward the location where the user is exercising, and issues an instruction to start video recording. In response, the imaging unit 111 operates the camera 106 to capture the video. The imaging unit 111 stores the captured video in the image storage unit 131.

The image storage unit 131 stores images captured by the imaging unit 111. In this embodiment, the images are moving images. The image storage unit 131 can store the moving images, for example, as a file.

The physical information storage unit 130 stores information about the user's body (hereinafter referred to as physical information). FIG. 4 is a diagram showing an example of the configuration of physical information stored in the physical information storage unit 130. As shown in the figure, the physical information includes height, weight, flexibility, etc.

The evaluation request sending unit 112 sends a request to the server device 20 to evaluate the physical exercise based on the image captured by the imaging unit 111 (hereinafter referred to as an evaluation request).

Figure 5 is a diagram showing an example of the configuration of an evaluation request sent by the evaluation request sending unit 112 to the server device 20. As shown in the figure, the evaluation request includes a user ID, a mode, physical information, and image data. The user ID is information that identifies the user. The mode is information that indicates the exercise performed by the user. The mode can be, for example, "pitching" or "lifting." It should be noted that the mode is selected from a predetermined selection of options. The physical information is physical information stored in the physical information storage unit 130. The image data is data of a moving image acquired by the imaging unit 111.

The evaluation information receiving unit 113 receives information regarding the evaluation of physical exercise (hereinafter referred to as evaluation information) that is returned from the server device 20 in response to the evaluation request. The evaluation information receiving unit 113 registers the received evaluation information in the evaluation information storage unit 132.

Figure 6 is a diagram showing an example of the configuration of evaluation information that the evaluation information receiving unit 113 receives from the server device 20. As shown in the figure, the evaluation information includes a mode, a user ID, position information, posture information, movement information, and checkpoint information.

The user ID and mode are the user ID and mode included in the rating request. The captured image shows the user's body performing the exercise indicated by the mode.

The position information indicates the position of each part of the body (e.g., head, shoulders, elbows, waist, knees, ankles, etc.) in the image. The position information includes the part and the position of the part in association with the time on the time axis of the video. The state of the body skeleton (bones) can be displayed based on the position information. That is, for example, a figure indicating the part (e.g., a circle, etc.) can be displayed by superimposing it on the image at the position indicated by the position information. Note that the positions of multiple parts can be included for one time point. Note that position information does not need to be included for parts connecting two parts (e.g., the forearm connecting the wrist and elbow, or the thigh connecting the waist and knee, etc.). In this case, the part connecting these two parts can be expressed by connecting a pair of marks (e.g., a circle) indicating two specific parts with a line. The position information may be included for each frame constituting the video, may be included for each key frame, may be included for every arbitrary number of frames, or may be included for a random time point. If My Frame does not include position information, the bones can be displayed based on the position information of the closest past time point.

The posture information is information related to the posture of the user's body. The posture information includes the part to be evaluated, the posture value, the evaluation rank, and the evaluation comment, which are associated with a time point on the time axis of the video. The posture value is a value that represents the posture. The posture value is, for example, the distance from the ground to the part, the distance between two parts, the angle of the joint part (the angle formed by the straight line from the first end part to the joint part and the straight line from the second end part to the joint part), etc. The evaluation rank is a value that represents the evaluation value by rank. The evaluation rank is expressed, for example, from 1 to 5 on a five-point scale, or ABC. The evaluation comment is a comment related to the evaluation of the posture. For example, when the mode is "lifting" and bending is not sufficient, an evaluation comment such as "your hips are not lowered" may be included.

The movement information is information related to the movement of the user's body. The movement information includes the parts to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment, all of which are associated with a period on the time axis of the video. The list of posture values is a time series of posture values within a period. The evaluation comment is a comment related to the evaluation of the movement. For example, if the mode is "lifting" and the knee extension is not smooth, an evaluation comment such as "The knee movement is not smooth" may be included.

Checkpoint information is information that indicates points (hereinafter referred to as checkpoints) where posture, etc. should be checked during a series of movements of the user's body. For example, when the mode is "pitching", checkpoints include when the foot is raised, when the raised foot is lowered and weight is transferred, and when the ball is released. The checkpoint information stores information indicating the checkpoint (hereinafter referred to as checkpoint ID) in association with a time point on the time axis of the video. In other words, it is possible to identify the frame (still image) in the video in which the checkpoint indicated by the checkpoint ID is displayed.

The evaluation display unit 114 displays the evaluation information. For example, the evaluation display unit 114 can display bone movements superimposed on the video by displaying figures representing body parts (for example, circles representing end parts or joints and lines connecting them) superimposed on the video based on the position information included in the evaluation information. The evaluation display unit 114 can also display, for example, a graph showing the change in the position of the body part over time.

The evaluation display unit 114 can also display an evaluation rank and an evaluation comment in conjunction with the display of the video, based on the posture information and movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and evaluation comment included in the posture information when the playback time of the video reaches a point around the time point included in the posture information (for example, this can be any length, such as around 5 seconds). The evaluation display unit 114 can also display the evaluation rank and evaluation comment included in the movement information when the playback time of the video reaches a period included in the movement information. The evaluation display unit 114 can also display the posture values included in the posture information. The evaluation display unit 114 can also display a graph of the time-series conversion of the posture values, based on a list of the posture values included in the movement information.

The checkpoint display unit 115 can extract and display images of checkpoints from the video. The checkpoint display unit 115 can read frames corresponding to time points included in the checkpoint information from the image data of the video stored in the image storage unit 131 and display them as still images. Furthermore, the checkpoint display unit 115 may, for example, extract and display only body parts from the read frames.

The improvement measure request sending unit 116 sends a request for acquiring improvement measures related to physical exercise (hereinafter referred to as an improvement measure request) to the server device 20. FIG. 7 is a diagram showing an example of the configuration of an improvement measure request. As shown in the figure, the improvement measure request includes a user ID, a mode, and a purpose. The purpose is the purpose for which the user wishes to make an improvement. Examples of purposes may be "increase ball speed," "increase muscle strength," "stabilize the lower body," etc. The purpose is also selected from a predetermined list of options.

The improvement information receiving unit 117 receives information about improvement measures (hereinafter referred to as improvement information) transmitted from the server device 20 in response to an improvement request. The improvement information receiving unit 117 stores the received improvement information in the improvement condition storage unit 133. An example of the configuration of improvement information is shown in FIG. 8. As shown in the figure, the improvement information includes a purpose, advice, and reference information. In this embodiment, the advice is assumed to be a character string representing the improvement measure, but it may also be content that presents the improvement measure using an image, video, or the like. The reference information is an ideal body posture (the position and angle of each part).

The improvement measure display unit 118 displays the improvement measures. The improvement measure information display unit 118 displays advice included in the improvement measure information. In addition, if the improvement measure information includes a suitable position or angle for a part,

FIG. 9 is a diagram showing an example of the hardware configuration of the server device 20. The server device 20 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. for inputting data. The output device 206 is, for example, a display, a printer, a speaker, etc. for outputting data.

Fig. 10 is a diagram showing an example of the software configuration of the server device 20. As shown in the figure, the server device 20 includes functional units, namely, an evaluation request receiving unit 211, an image analysis unit 212, an evaluation unit 213, an evaluation information transmitting unit 214, an improvement request receiving unit 215, and an improvement information transmitting unit 216, as well as storage units, namely, an image data storage unit 231, a reference information storage unit 232, an evaluation condition information storage unit 233, and an improvement condition storage unit 234.

The above-mentioned functional units are realized by the CPU 201 of the server device 20 reading out a program stored in the storage device 203 into the memory 202 and executing it, and the above-mentioned storage units are realized as part of the storage area provided by the memory 202 and storage device 203 of the server device 20.

The evaluation request receiving unit 211 receives an evaluation request sent from the user terminal 10. The evaluation request receiving unit 211 registers information including image data included in the received evaluation request (hereinafter referred to as image information) in the image data storage unit 231. FIG. 11 is a diagram showing an example of the configuration of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data associated with a user ID indicating the user who captured the image. The image data was included in the evaluation request.

The reference information storage unit 232 stores information (hereinafter referred to as reference information) including reference values for postures (positions and angles of body parts) related to physical movement. FIG. 12 is a diagram showing an example of the configuration of reference information stored in the reference information storage unit 232. As shown in the figure, the reference information includes reference information regarding the absolute position of a part or its relative position with respect to other parts or other reference objects (hereinafter referred to as position reference information), and, for three parts including a joint part, reference information on the angle formed by the straight lines connecting each of two parts with the joint part (hereinafter referred to as angle reference information).

The position reference information includes a body part and a reference position for that body part, in association with a mode and a checkpoint ID. There may be multiple body parts. Regarding the position, the vertical position may be, for example, the height from the ground, or the distance from one of the toes. The horizontal position may be the distance from a specified reference object (for example, a mound plate or a mark on the floor) or the distance from a reference body part such as the shoulders, chest, or feet. The position reference information is assumed to be registered in advance.

The angle reference information corresponds to a mode and a checkpoint ID and includes two parts (part 1 and part 2), one joint part, and a reference value of the angle between a line connecting part 1 and the joint part, and a line connecting part 2 and the joint part.

The evaluation condition information storage unit 233 stores information for evaluation (hereinafter, referred to as evaluation condition information). FIG. 13 is a diagram showing an example of the configuration of evaluation condition information stored in the evaluation condition information storage unit 233. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. For example, the category can be "ball speed" or "control". The condition is a condition for the position or movement of each part of the body in the image (change in position in time series). For example, when analyzing a pitching form, conditions for the elbow angle, arm line speed, etc. can be set in the evaluation condition information for the checkpoint where the ball is released. The evaluation rank is an evaluation value when the above conditions are met. The comment is an explanation of the posture and movement of the body when the above conditions are met.

The image analysis unit 212 (body part identification unit) analyzes the image data. The image analysis unit 212 analyzes the image data to extract features of each body part and identify the position of each part in the image. Note that a general method of image analysis is adopted by the image analysis unit 212, and detailed explanation is omitted here. The image analysis unit 212 may analyze the image data for each frame or key frame, or may analyze at random timing.

The image analysis unit 212 also compares the position of each part extracted from the image data for each checkpoint ID with the position reference information stored in the reference information storage unit 232, and identifies the closest time as the checkpoint time.

The evaluation unit 213 evaluates the user's body movements based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition storage unit 233 for evaluation condition information including the conditions satisfied by the positions of each body part and the movements of each body part identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, obtains the evaluation rank and comments contained therein.

The evaluation information transmission unit 214 transmits the evaluation information to the user terminal 10. The evaluation information transmission unit 214 generates position information including the time on the time axis of the video identified by the image analysis unit 212 and the position of each part. For the evaluation rank and comment acquired by the evaluation unit 213, if the position of the part satisfies the condition, posture information including the time, part, and posture value, and the evaluation rank and comment is generated, and if the movement of the part (change in position in time series) satisfies the condition, movement information including a list of the time, part, and posture value, and the evaluation rank and comment is generated. In addition, the evaluation condition transmission unit 214 generates checkpoint information including the time corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmission unit 214 creates evaluation information including the generated position information, posture information, movement information, and checkpoint information and transmits it to the user terminal 10. Note that the evaluation unit 213 and the evaluation information transmission unit 214 may correspond to the comment output unit of the present invention.

The improvement condition storage unit 234 stores information related to improvement measures (hereinafter referred to as improvement condition information). FIG. 14 is a diagram showing an example of the configuration of improvement condition information stored in the improvement condition storage unit 234. As shown in the figure, the improvement condition information includes advice associated with the objective, category, and condition. The condition may be a condition for a physical condition (such as flexibility), or a condition for the position or movement of a body part.

The improvement request receiving unit 215 receives an improvement request sent from the user terminal 10.

The improvement information sending unit 216 searches for improvement condition information corresponding to the mode and purpose included in the improvement request, where the user's physical information included in the evaluation request and the positions and movements of each part identified by the image analysis unit 212 satisfy the conditions. The improvement information sending unit 216 acquires advice from the searched improvement information, creates improvement information in which the purpose and advice are set, and responds with the created improvement information to the user terminal 10. The improvement information sending unit 216 also transmits the positions and angles of each part included in the reference information, including them in the improvement information.

Figure 15 shows the flow of processing executed in the exercise evaluation system of this embodiment.

In the user terminal 10, the imaging unit 111 accepts the input of the mode, images the user's body while exercising, and obtains video data (S321). The evaluation request sending unit 112 sends an evaluation request including the user ID indicating the user, the accepted mode, physical information, and video data to the server device 20 (S322).

When the evaluation request receiving unit 211 in the server device 20 receives an evaluation request, the image analysis unit 212 analyzes the video data to extract features (S323) and identifies the position of each part (S324). Here, the image analysis unit 212 may identify the position on the image, or may identify the actual position (height from the ground, distance from a reference point such as the center of gravity of the body, etc.) using body information. The evaluation unit 213 obtains an evaluation rank and comments from evaluation condition information in which the position of each part and the movement of the part (changes in position over time) satisfy the conditions (S325). The evaluation information sending unit 214 creates evaluation information and sends it to the user terminal 10 (S326).

The evaluation display unit 114 in the user terminal 10 displays the positions (bones) of each body part indicating the posture of the body on the video data based on the received evaluation information, as well as the evaluation rank and comments (S327). Here, the evaluation display unit 114 may display a graph of the time-series changes in the positions and movements of the body parts. The checkpoint display unit 115 may also extract and display images of checkpoints from the video. The improvement measure request sending unit 116 sends an improvement measure request to the server device 20 in response to instructions from the user (S328).

In the server device 20, when the improvement request receiving unit 215 receives an improvement request sent from the user terminal 10, the improvement information sending unit 216 searches for improvement condition information that satisfies the conditions, acquires advice contained in the searched improvement condition information (S329), creates improvement information including the acquired advice, and sends it to the user terminal 10 (S330).

When the improvement information receiving unit 117 receives improvement information in the user terminal 10, the improvement display unit 118 displays the advice contained in the received improvement information and can display an appropriate body posture in the form of bones superimposed on the video data (S331).

In this way, the exercise evaluation system of this embodiment allows easy evaluation of physical exercise. In particular, for physical exercise related to sports, the positional relationships and movements of each part of the body can be evaluated, which is likely to lead to concrete efforts to improve and improve performance. In addition, the exercise evaluation system of this embodiment also provides comments and advice, allowing the user to easily understand the current situation and measures for improvement.

Figures 16 to 20 show examples of screens that display physical exercise evaluations or improvement measures.

Figure 16 explains the case where a video is captured in pitching mode. As shown in the figure, the body posture is displayed on a screen 41 using bones 411 (which may be defined as lines connecting 17 feature points such as eyes, ears, and joints, for example) that indicate the position of each part of the body. In addition, comments according to deviations from the reference value are displayed in a comment field 412. The user can use this as a reference to consider posture corrections, etc.

Figure 17 shows an example of displaying an image of a checkpoint. A checkpoint selection field 421 is displayed on screen 42, from which a checkpoint can be selected. In the example of Figure 17, checkpoint 422 for the release point is selected, and an image 423 from the video corresponding to that checkpoint is displayed. This makes it easy to understand one's own body posture and movement at each checkpoint. As a method of estimating the checkpoint, for example, it can be determined that the release point has been reached when certain conditions are met, such as when the pitcher's upper body faces forward (toward the pitch), when the coordinate of the pitcher's right hand (dominant hand) is at its highest, and when the distance between the pitcher's shoulders is the widest (when the distance is longest).

18 to 20 are diagrams showing an example of an evaluation screen for a video in weightlifting mode. In the example of screen 43, in addition to the predetermined position of the part identified from the image (the position where the user's buttocks reach the lowest point), the preferred posture set as reference information is also displayed on bone 431. Furthermore, on screen 43, information 432 related to the movement of the shaft is displayed. In this way, it is possible to identify the position and movement not only of the body part but also of the tool used by the user. This makes it possible to count, for example, the number of times the weight has been lifted from the image. Also, as shown in FIG. 19, comments or advice 433 related to evaluation or improvement measures can be displayed on screen 43 together with the video. Also, in the example of FIG. 20, a time series graph 434 of the knee bending angle is displayed. By further analyzing the changes in graph 434, an abnormal value 435 can also be detected. Also, although not shown, a time series graph of the knee bending speed can be displayed together with graph 434 and/or. These graphs can present the appropriate weight and number of repetitions for the user.

Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the interpretation of the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

For example, in this embodiment, the image analysis is performed in the server device 20, but this is not limited to the above. The image analysis may be performed in the user terminal 10 to identify the positional relationship of each part.

In addition, in this embodiment, the position of the part is assumed to be a position on a two-dimensional image, but this is not limited to the above, and may be a three-dimensional position. For example, if the user terminal 10 is equipped with a depth camera in addition to the camera 106, the three-dimensional position of the part can be identified based on the image from the camera 106 and the depth map from the depth camera. It is also possible to provide a depth camera instead of the camera 106 and identify the three-dimensional position only from the depth map from the depth camera. In this case, the user terminal 10 can transmit the depth map to the server device 20 together with or instead of the image data, and the image analysis unit 212 of the server device 20 can analyze the three-dimensional position.

In addition, in this embodiment, an image of the user's body during exercise is transmitted from the user terminal 10 to the server device 20, but this is not limited to the above. The user terminal 10 may extract features from the image and transmit the features to the server device 20. Alternatively, the user terminal 10 may estimate body parts based on the features, obtain the absolute position of the part (which may be the position on the XY coordinate system of the image, or the actual distance from a reference position (for example, the ground, the toes, the head, the center of gravity of the body, etc.), or the position in any other coordinate system) or the relative positional relationship between multiple parts, and transmit these absolute positions and relative positional relationships to the server device 20.

In addition, in this embodiment, the improvement information is provided as content prepared by the server device 20, but this is not limited thereto. For example, it is also possible to include reference values and display bones that result in the correct posture (position and angle of each part) based on the reference values superimposed on a video or a still image extracted from the video. This makes it easy to understand what posture the user should have.

In addition, in this embodiment, the position or movement (position over time) of a body part is evaluated, but this is not limiting, and the position of a tool held or worn by the user may be identified and evaluated. In this case, reference information and evaluation condition information regarding the tool position may be registered, and the feature values of the tool may be extracted to identify the position, and the evaluation may be performed in the same way as for the position of a body part.

In addition, in this embodiment, content such as advice is provided as a remedy, but for example, equipment recommendations may be made. In this case, the server device 20 stores the equipment and a reference value for the size (length, etc.) of the equipment in association with the user's physical information (height, weight, etc.), extracts features of the equipment used by the user from the image data to identify the shape of the equipment, estimates the size of the equipment based on the shape and the user's size (e.g. height) included in the physical information, and if the difference between the estimated equipment size and the reference value is equal to or greater than a predetermined threshold, recommends equipment of the reference size.

In addition, by estimating the position of the tool, the server device 20 can estimate the speed of the tool. In addition, by extracting a pattern of changes in the tool position over time, the server device 20 can estimate the number of times the pattern occurs as the number of actions using the tool.

In addition, in this embodiment, the exercise is evaluated, but the present invention is not limited to this. When a certain posture or movement is detected, a task for that movement may be suggested. In this case, the server device 20 may store a task in association with one or a series of postures or movements instead of an evaluation comment, and output the task.

In addition, in this embodiment, the exercise is evaluated, but the present invention is not limited to this, and when a certain posture or movement is detected, training to be performed may be presented. In this case, the server device 20 may store the training content in association with one or a series of postures or movements, instead of an evaluation comment, and output the content.

In addition, in this embodiment, the movement is evaluated, but it is not limited to this, and it is also possible to automatically detect the movements performed by the player. In this case, the server device 20 stores the posture (position of each part of the body) when performing a specific movement such as shooting or passing as reference information, and can compare the positions of the body parts analyzed from the image with the reference information to identify the movement performed by the player in the image.

In addition, in this embodiment, images captured in the past are analyzed to evaluate the movement, but this is not limited to the above. The analysis process may be performed in real time, and when a specific movement is detected, the next tactic to be taken may be recommended. In this case, the tactic may be stored in place of the evaluation comment in association with the posture or movement, and the tactic may be output in real time.

In addition, in this embodiment, the execution of a specific function and the storage of information are performed by the user terminal or the server device, but this is not limited thereto, and the execution of the function and the storage of information may be performed by either one of the devices. Alternatively, the functional unit and the storage unit may be provided in a distributed manner in a form different from this embodiment.

REFERENCE SIGNS LIST 10 User terminal 20 Server device 30 Communication network 111 Imaging unit 112 Evaluation request transmission unit 113 Evaluation information reception unit 114 Evaluation display unit 115 Checkpoint display unit 116 Improvement measure request transmission unit 117 Improvement measure information reception unit 118 Improvement measure display unit 130 Physical information storage unit 131 Image storage unit 132 Evaluation information storage unit 133 Improvement measure storage unit 211 Evaluation request reception unit 212 Image analysis unit 213 Evaluation unit 214 Evaluation information transmission unit 215 Improvement measure request reception unit 216 Improvement measure information transmission unit 231 Image data storage unit 232 Reference information storage unit 233 Evaluation condition information storage unit 234 Improvement condition information storage unit

Claims (5)

A system for evaluating a user's physical exercise, comprising:
a reference value storage unit that stores a reference value relating to a relative position of at least one body part with respect to another reference object at a checkpoint, which is a timing for checking posture, that exists in a plurality of times during the predetermined exercise when the exercise is performed appropriately;
a part identification unit that analyzes an image of the body captured by a camera of an information terminal used by the user, and identifies the part and the position of the part in the image of the body by extracting a feature amount of each part of the body;
an evaluation unit that determines an evaluation value of the movement by comparing a position of the part with respect to the other reference object in the image and the reference value;
Equipped with
the evaluation unit compares a position of the part relative to the other reference object with the reference value in the images of the plurality of consecutive checkpoints;
An exercise evaluation system comprising: The exercise evaluation system according to claim 1 ,
the evaluation unit determines the evaluation value according to a distance obtained when comparing the reference value with a position of the part relative to the other reference object in the image;
An exercise evaluation system comprising: The exercise evaluation system according to claim 1 ,
the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion;
the part identification unit identifies the first to third parts and positions of the parts from the image;
the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value;
An exercise evaluation system comprising: The exercise evaluation system according to claim 1 ,
a comment storage unit that stores a comment about the part in association with a condition for the part and the position;
a comment output unit that outputs the comment corresponding to the condition satisfied by the position;
The exercise evaluation system further comprising: The exercise evaluation system according to claim 1 ,
an improvement information storage unit that stores advice on an improvement measure related to the exercise in association with a goal of improvement related to the exercise and a condition for the position;
an advice output unit that outputs the advice corresponding to the condition satisfied by the position;
The exercise evaluation system further comprising:

Images