JP7620969B2 - Exercise Support System - Google Patents

Description

Applicable under Article 30, Paragraph 2 of the Patent Act. Published on the website "https://youtu.be/lsrAgAKVxMA" on November 1, 2018.

The present invention relates to an exercise support system.

Conventional analytical techniques in the field of physical movement include a technique for extracting repeated movement patterns in a video, comparing the movement patterns at multiple positions in the extracted video at different time positions from a specific viewpoint, and outputting the comparison results as a video or still image (Patent Document 1). In sports, for example, a technique for analyzing a golf wing using a motion sensor by attaching a device to the user's body (Patent Document 2), and a technique for capturing images of each stage of a pitch (placement of the pivot foot, horizontal rotation of the trunk) with two cameras and calculating the transition time (Patent Document 3).

JP 2017-229081 A JP 2010-068947 A JP 2018-114138 A

However, all of the technologies described in Patent Documents 1 to 3 require equipment and resources such as multiple video images, specific devices, and multiple cameras, making it difficult for a single user to use these devices to analyze physical movements and understand the analysis results.

The present invention was made in light of this background, and aims to provide technology that can efficiently support physical exercise.

The main invention of the present invention for solving the above problems is a system for assisting a user in exercising, comprising a camera for photographing the user while exercising, a display installed at the location where the exercise is performed, an explanation output unit for outputting an explanation of the exercise to the display, a body part identification unit for analyzing the image captured by the camera and identifying the positions of body parts of the user, and an output unit for superimposing the positions of the body parts on the image and outputting it to the display.

Other problems and solutions disclosed in this application will be made clear in the description of the embodiments of the invention and the drawings.

The present invention can efficiently support physical exercise.

1 is a diagram illustrating an example of the overall configuration of an exercise support system according to an embodiment of the present invention. FIG. 2 illustrates an example of a hardware configuration of a server device 20. FIG. 2 is a diagram illustrating an example of the software configuration of the server device 20. FIG. 2 is a diagram illustrating an example of the configuration of a training information storage unit 231. 2 is a diagram illustrating an example of the configuration of a user information storage unit 232. FIG. 1 is a diagram showing an example of an input screen 41 for inputting user information. FIG. 2 is a diagram illustrating an example of the configuration of an image data storage unit 233. FIG. 2 is a diagram illustrating an example of the configuration of a reference information storage unit 234. FIG. 2 is a diagram illustrating an example of the configuration of an evaluation condition information storage unit 235. FIG. 2 is a diagram illustrating an example of the configuration of a training plan storage unit 236. FIG. 2 is a diagram showing an example of the configuration of a training history storage unit 237. FIG. 2 is a diagram illustrating an example of the configuration of an improvement condition storage unit 238. FIG. 4 is a diagram showing an example of a training information output screen 42. 13 is a diagram showing an example of the configuration of evaluation information created by an evaluation unit 215. FIG. FIG. 4 is a diagram showing an example of a screen 42 output by an evaluation information output unit 216. FIG. 4 is a diagram showing an example of a screen 42 output by an evaluation information output unit 216. FIG. 4 is a diagram showing an example of a screen 42 output by an evaluation information output unit 216. FIG. 13 is a diagram showing an example of a training history displayed by the user terminal 10. FIG. 4 is a diagram showing an example of an output screen 44 for training volume information. FIG. 2 is a diagram showing a flow of processing executed in the exercise support system of the present embodiment.

The details of the embodiment of the present invention will be listed and explained. The exercise support system according to the embodiment of the present invention has the following configuration.

[Item 1]
A system for assisting a user in exercising, comprising:
a camera that photographs the user while exercising;
A display installed in a place where the exercise is performed;
an explanation output unit that outputs an explanation regarding the exercise to the display;
a body part identification unit that analyzes an image captured by the camera to identify a position of a body part of the user;
an output unit that superimposes the position of the part on the image and outputs the image to the display;
An exercise support system comprising:
[Item 2]
Item 1, the exercise support system comprising:
a reference value storage unit that stores a reference value related to the position of at least one of the parts;
an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value of the movement;
Further equipped with
The output unit further outputs the evaluation value to the display.
An exercise support system characterized by the above.
[Item 3]
Item 2: The exercise support system according to item 2,
the evaluation unit determines the evaluation value according to a distance from the reference value to the position;
An exercise support system characterized by the above.
[Item 4]
Item 2: The exercise support system according to item 2,
the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion;
the part identification unit identifies each of the first to third parts from the image;
the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value;
An exercise support system characterized by the above.
[Item 5]
Item 1, the exercise support system comprising:
A number counting unit that counts the number of times of the exercise based on the position of the part is further provided,
The output unit further outputs the number of times of the exercise to the display.
An exercise support system characterized by the above.
[Item 6]
The exercise support system according to claim 2,
The output unit further outputs the evaluation value by voice.
An exercise support system characterized by the above.
[Item 7]
Item 1, the exercise support system comprising:
The output unit further outputs remedial action information by voice based on the identified body position of the user;
An exercise support system characterized by the above.

An exercise support system according to one embodiment of the present invention is intended to support a user's training at a training gym. The exercise support system of this embodiment uses a camera to capture images of the user exercising using training equipment, and counts the number of training sessions from the captured images (which may be still images or video images, but in this embodiment, video images are used), and evaluates the user's physical posture and movements during training.

Figure 1 is a diagram showing an example of the overall configuration of an exercise support system according to this embodiment. As shown in the figure, in the exercise support system of this embodiment, a server device 20, a camera 21, and a display 22 are communicatively connected to each other via a communication network 23. The communication network 23 is, for example, the Internet or a LAN (Local Area Network), and is constructed using a public telephone line network, a dedicated telephone line network, a mobile phone line network, Ethernet (registered trademark), a wireless communication path, etc.

The camera 21 captures images of the user 1 during training. The camera 21 may be, for example, a surveillance camera, or may be installed at the training location, such as by attaching the camera 21 to training equipment. The camera 21 can transmit the captured images to the server device 20 using protocols such as TCP/IP or Bluetooth (registered trademark). The camera 21 may be set to capture images continuously, may start capturing images when it detects that the user 1 is nearby, or may start capturing images in response to an instruction from the user 1.

The display 22 is installed so as to be visible to the user 1 before and during training. The display 22 outputs information under the control of the server device 20 via the communication network 23. By connecting a computer to the display 22 and having the computer communicate with the server device 20, information can be output to the display 22 in response to instructions from the server 20. In addition, in addition to or instead of the display 22, the user 1 can be made to wear an audio output device such as headphones, earphones, or a neck speaker, so that information can be output as audio.

The user terminal 10 is a computer operated by the user 1 or his/her supporter. The user terminal 10 is, for example, a smartphone, a tablet computer, or a personal computer. The user terminal 10 may be equipped with an imaging device such as a camera, which may be used to capture images of the body of the user 1 during exercise. The information displayed on the display 22 is also provided to the user terminal 10 from the server device 20. The user 1 can view information about the training by checking the user terminal 10 later.

The server device 20 is a computer that supports training. The server device 20 is, for example, a workstation, a personal computer, or a virtual computer that is logically realized by cloud computing. The server device 20 receives video images captured by the user terminal 10 and analyzes the received video images to evaluate the physical exercise. The server device 20 also makes suggestions regarding measures to improve the physical exercise. Details of the evaluation of physical exercise and the suggestions of measures to improve will be described later.

The administrator terminal 30 is a computer operated by the administrator of the training gym. The administrator terminal 30 can be, for example, a smartphone, a tablet computer, or a personal computer.

2 is a diagram showing an example of the hardware configuration of the server device 20. The server device 20 includes a calculation device 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The calculation device 201 performs various calculation processes, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. for inputting data. The output device 206 is, for example, a display, a printer, a speaker, etc. for outputting data.

Fig. 3 is a diagram showing an example of the software configuration of the server device 20. As shown in the figure, the server device 20 includes functional units of a user authentication unit 210, a training information output unit 211, an image acquisition unit 212, an image analysis unit 213, a frequency count unit 214, an evaluation unit 215, an evaluation information output unit 216, a training history transmission unit 217, and an improvement information output unit 218, and storage units of a training information storage unit 231, a user information storage unit 232, an image data storage unit 233, a reference information storage unit 234, an evaluation condition information storage unit 235, a training plan storage unit 236, a training history storage unit 237, and an improvement condition storage unit 238.

The above-mentioned functional units are realized by the arithmetic unit 201 of the server device 20 reading out a program stored in the storage device 203 into the memory 202 and executing it, and the above-mentioned storage units are realized as part of the storage area provided by the memory 202 and storage device 203 of the server device 20.

The training information storage unit 231 stores information about training. The information about training is, for example, a description of the training. The description of the training includes how to use the training equipment, and how to effectively train using the training equipment. FIG. 4 is a diagram showing an example of the configuration of the training information storage unit 231. As shown in the figure, the training information storage unit 231 stores training information and display information. The training information is information about training. The training information includes information for identifying the training menu (menu ID) and video data representing the content of the training, the number of times, and the method of training, in association with the purpose of the training. The purpose of the training indicates the results desired from training, such as resolving lack of exercise and increasing muscle mass. The content of the training can be set to any information, such as how to use the training equipment and points to note about training. The video data is, for example, a video of a person demonstrating how to train, a tutorial video showing how to use the training machine, and the like. The content of the training and the video data are linked to the purpose of the training, and are prepared to suggest what kind of training should be done and how many times to do it in order to achieve the purpose using the training equipment. The display information is information that sets a training menu to be performed while watching the display 22. The display information includes a menu ID that corresponds to a display ID that identifies the display 22.

The user information storage unit 232 stores information about the user 1 (hereinafter, referred to as user information). FIG. 5 is a diagram showing an example of the configuration of the user information storage unit 232. As shown in the figure, the user information stored in the user information storage unit 232 includes authentication information, height, weight, muscle mass, flexibility, and the like, associated with a user ID that identifies the user 1. The authentication information is information that can be used to authenticate the user 1. For example, the authentication information may be a feature amount of the face of the user 1 for use in face authentication, a device ID of the user terminal 10, a combination of a username and password, or a public key of the user 1, but is not limited to these as long as it is information that can authenticate the user 1. In addition to the above items, the user information may include various information about the user's body that can be measured or evaluated.

Figure 6 is a diagram showing an example of an input screen 41 for inputting user information. For example, an input screen 41 as shown in the figure can be displayed on the user terminal 10, and the user 1 can input each item 411 of the user information. In this case, each item of the user information can be transmitted from the user terminal 10 to the server device 20 and registered in the user information storage unit 232.

The image data storage unit 233 stores images captured by the camera 21. FIG. 7 is a diagram showing an example of the configuration of the image data storage unit 233. As shown in the figure, the image data storage unit 233 stores image information, and the image information includes image data associated with a camera ID that identifies the camera 21 that captured the image. The image data is data for displaying the image captured by the camera 21. The image data can be data in any format, such as MP4, MPEG2, AVI, MOV, etc.

The reference information storage unit 234 stores information (hereinafter referred to as reference information) including reference values of posture (position and angle of part) related to the physical movement of the user 1. FIG. 8 is a diagram showing an example of the configuration of the reference information storage unit 234. As shown in the figure, the reference information storage unit 234 stores reference information (hereinafter referred to as position reference information) related to the absolute position of a part or its relative position with respect to other parts or other reference objects, and, for three parts including a joint part, reference information of the angle formed by the straight lines connecting each of two parts with the joint part (hereinafter referred to as angle reference information).

The position reference information includes a body part and a reference position for that body part, in association with a mode and a checkpoint ID. The checkpoint ID is information indicating a point (hereinafter referred to as a checkpoint) where posture, etc. should be checked in a series of movements of the user's body. There may be multiple body parts. Regarding the position, the vertical position may be, for example, the height from the ground or the distance from one of the toes. The horizontal position may be the distance from a specified reference object (for example, a mark on the floor or an installed instrument) or the distance from a reference body part such as the shoulders, chest, or feet. It is assumed that the position reference information is registered in advance.

The angle reference information corresponds to a mode and a checkpoint ID and includes two parts (part 1 and part 2), one joint part, and a reference value of the angle between a line connecting part 1 and the joint part, and a line connecting part 2 and the joint part.

The evaluation condition information storage unit 235 stores information for evaluation (hereinafter, referred to as evaluation condition information). FIG. 9 is a diagram showing an example of the configuration of the evaluation condition information storage unit 235. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. Examples of categories include "arm curl," "clean," and "press." The condition is a condition for the position or movement of each part of the body in the image (change in position over time). For example, when analyzing weightlifting movements, conditions for the elbow angle, waist angle, and the like can be set in the evaluation condition information for the checkpoint for cleaning the weights. The evaluation rank is an evaluation value when the above conditions are met. The comment is an explanation of the posture and movement of the body when the above conditions are met.

The training plan storage unit 236 stores training plans set by the user 1 or their supporter. FIG. 10 is a diagram showing an example of the configuration of the training plan storage unit 236. The training plan storage unit 236 stores training plans, and each training plan includes a menu ID indicating a training menu, settings for training equipment, etc. (e.g., weight, etc.), and the number of training sessions (e.g., repeated actions such as lifting weights or doing squats), in association with a user ID indicating the user 1 and a date. It is assumed that the training plan is set in advance by the user 1 or their supporter.

The training history storage unit 237 stores the training history of user 1. FIG. 11 is a diagram showing an example of the configuration of the training history storage unit 237. As shown in the figure, the training history storage unit 237 stores training history, and each training history includes a menu ID indicating the training menu performed by user 1, setting information indicating the settings of the training equipment, etc., the number of training sessions, the time the training was performed, etc., associated with a user ID indicating user 1 and the date and time when user 1 performed the training.

The improvement condition storage unit 238 stores information related to improvement measures for training (hereinafter referred to as improvement condition information). FIG. 12 is a diagram showing an example of the configuration of the improvement condition storage unit 238. As shown in the figure, the improvement condition information stored in the improvement condition storage unit 238 includes advice associated with a menu ID, a purpose, a category, and a condition. The condition may be a condition for a physical condition (such as flexibility), or a condition for the position or movement of a body part. In this embodiment, the advice is assumed to be a character string representing the improvement measure, but it may also be content that presents the improvement measure using an image, video, or the like.

The user authentication unit 210 authenticates the user 1 who is looking at the display 22. The user authentication unit 210 can authenticate the user 1 using the authentication information of the user information. In this embodiment, it is assumed that the authentication information of the user information is a feature of the face of the user 1, and the user authentication unit 210 authenticates the user 1 by extracting the feature from the image captured by the camera 21. Note that the user authentication unit 210 may, for example, communicate with the user terminal 10 to authenticate the user 1, or the display 22 may authenticate the user 1 by communicating with the user terminal 10, and the result may be transmitted from the display 22 to the user authentication unit 210.

The training information output unit 211 outputs training information to the display 22. The training information output unit 211 identifies a menu ID corresponding to a display ID indicating the display 22 from the display information, and outputs the training information content and video data corresponding to the identified menu ID to the display 22. The training information output unit 211 can output training information to the display 22 when the user 1 comes near the display 22 (for example, the display 22 has a human sensor and the human sensor detects the user 1, or the display 22 has a communication function for communicating with the user terminal 10 and detects that the display 22 is able to communicate with the user terminal 10, thereby detecting that the user 1 is in the vicinity). The user 1 can refer to the training content and video output to the display 22 before or during training by looking at the training content and video output to the display 22.

Figure 13 is a diagram showing an example of a training information output screen 42. The output screen 42 displays a training objective 421, training content 422, and video 423. Multiple pieces of video data may be included in the training information, and multiple videos may be displayed on the output screen 42.

The training information output unit 211 may correct the content of the training information based on the physical information. For example, the training information output unit 211 may change the height of the chair of the training equipment according to the height, or set the recommended weight according to the height and weight.

The training information output unit 211 may also correct the contents of the training information according to the past training history of the user 1. For example, the training information output unit 211 may read out the training history corresponding to the user ID indicating the user 1 from the training history storage unit 237, and depending on the setting information (weight, chair height, etc.) and number of times included in the read out training history, may include in the training information content, for example, information similar to the content of the previous setting information, or a weight that is a predetermined value heavier than the previous weight, or a number of times that is a predetermined value more than the previous number of times.

The image acquisition unit 212 acquires images captured by the camera 21. In this embodiment, the camera 21 transmits captured image data to the server device 20 via communication, and the image acquisition unit 212 registers the image data transmitted from the camera 21 in the image data storage unit 233 in association with a camera ID indicating the camera 21. Note that the image acquisition unit 212 may transmit an instruction to the camera 21 when the user authentication unit 210 has successfully authenticated the user 1, causing the camera 21 to start capturing images.

The image analysis unit 213 (body part identification unit) analyzes the image data. The image analysis unit 213 analyzes the image data to extract features of each body part and identify the position of each part in the image. Note that a general method of image analysis is adopted by the image analysis unit 213, and detailed explanation is omitted here. The image analysis unit 213 may analyze the image data for each frame or key frame, or may analyze at random timing.

The image analysis unit 213 also compares the position of each part extracted from the image data for each checkpoint ID with the position reference information stored in the reference information storage unit 232, and identifies the closest time as the checkpoint time.

The image analysis unit 213 can also recognize the body movements of the user 1. In this case, the image analysis unit 213 can create a model capable of recognizing body movements (e.g., squats, lift-ups, etc.) from video data by machine learning, for example, and recognize the movements using the model. In this case, the image analysis unit 213 can be realized by, for example, performing parallel processing using multiple GPUs.

The number of times counting unit 214 counts the number of training sessions. The number of times counting unit 214 can count the number of times the user 1 changes to a predetermined posture (from another posture) using the position of each part analyzed by the image analysis unit 213. For example, the number of times counting unit 214 can count up when the distance between the position of each part analyzed by the image analysis unit 213 and the position included in the position reference information is within a predetermined value. Also, the number of times counting unit 214 may be configured to count up when the angle between two parts with the joint part included in the angle reference information as the vertex is within a predetermined value. Also, the number of times counting unit 214 may prepare a model created by machine learning using the posture of the body as teacher data and the position (relative positional relationship) of each part as input data in advance, identify the posture of the user 1 using the model, and count up when the user 1 takes a predetermined posture.

The repetition counting unit 214 can also count the number of training sessions and the number of training sets, multiply these values by the weights, and calculate the training volume to calculate the workload of user 1. This training volume can be used as an index to measure the fatigue level of user 1. By storing this training volume in the training history storage unit 237, user 1 can display and check this training volume and its progress on the display 22 or the like at any time of their choice.

The evaluation unit 215 evaluates the body movements of the user 1 based on the image data. In this embodiment, the evaluation unit 215 searches the evaluation condition storage unit 235 for evaluation condition information including the conditions satisfied by the positions of each body part and the movements of each body part identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, it obtains the evaluation rank and comments contained therein.

Figure 14 is a diagram showing an example of the configuration of evaluation information created by the evaluation unit 215. The evaluation information includes a menu ID, a user ID, position information, posture information, movement information, and checkpoint information.

The position information indicates the position of each part of the body (e.g., head, shoulders, elbows, waist, knees, ankles, etc.) in the image. The position information includes the part and the position of the part in association with the time on the time axis of the video. The state of the body skeleton (bones) can be displayed based on the position information. That is, for example, a figure indicating the part (e.g., a circle, etc.) can be displayed by superimposing it on the image at the position indicated by the position information. Note that the positions of multiple parts can be included for one time point. Note that position information does not need to be included for parts connecting two parts (e.g., the forearm connecting the wrist and elbow, or the thigh connecting the waist and knee, etc.). In this case, the part connecting these two parts can be expressed by connecting a pair of marks (e.g., a circle) indicating two specific parts with a line. The position information may be included for each frame constituting the video, may be included for each key frame, may be included for every arbitrary number of frames, or may be included for a random time point. If My Frame does not include position information, the bones can be displayed based on the position information of the closest past time point.

The posture information is information related to the posture of the user's body. The posture information includes the part to be evaluated, the posture value, the evaluation rank, and the evaluation comment, which are associated with a time point on the time axis of the video. The posture value is a value that represents the posture. The posture value is, for example, the distance from the ground to the part, the distance between two parts, the angle of the joint part (the angle formed by the straight line from the first end part to the joint part and the straight line from the second end part to the joint part), etc. The evaluation rank is a value that represents the evaluation value by rank. The evaluation rank is expressed, for example, from 1 to 5 on a five-point scale, or ABC. The evaluation comment is a comment related to the evaluation of the posture. For example, when the mode is "lifting" and bending is not sufficient, an evaluation comment such as "your hips are not lowered" may be included.

The movement information is information related to the movement of the user's body. The movement information includes the parts to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment, all of which are associated with a period on the time axis of the video. The list of posture values is a time series of posture values within a period. The evaluation comment is a comment related to the evaluation of the movement. For example, if the mode is "lifting" and the knee extension is not smooth, an evaluation comment such as "The knee movement is not smooth" may be included.

Checkpoint information is information about a checkpoint. For example, in the case of a weightlifting training menu, checkpoints include gripping the weight bar, standing upright while holding the weights, and raising the weights to chest height. The checkpoint information stores the checkpoint in association with a time point on the timeline of the video. In other words, it is possible to identify the frame (still image) in the video in which the checkpoint indicated by the checkpoint ID is displayed.

The evaluation unit 215 can generate position information including the time on the time axis of the video identified by the image analysis unit 212 and the position of each part. If the position of the part satisfies the conditions for the acquired evaluation rank and comment, the evaluation unit 215 generates posture information including the time, part, and posture value, as well as the evaluation rank and comment, and if the movement of the part (change in position in time series) satisfies the conditions, the evaluation unit 215 generates movement information including a list of the time, part, and posture value, as well as the evaluation rank and comment. The evaluation unit 215 also generates checkpoint information including the time corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint.

The evaluation information output unit 216 outputs the evaluation information to the display 22. For example, the evaluation display unit 114 can display bone movements superimposed on the video by displaying figures representing body parts (for example, circles representing end parts or joints and lines connecting them) superimposed on the video based on the position information included in the evaluation information. The evaluation information output unit 216 can also display, for example, a graph showing the change in the position of the body part over time.

The evaluation display unit 114 can also display an evaluation rank and an evaluation comment in conjunction with the display of the video, based on the posture information and movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and evaluation comment included in the posture information when the playback time of the video reaches a point around the time point included in the posture information (for example, this can be any length, such as around 5 seconds). The evaluation display unit 114 can also display the evaluation rank and evaluation comment included in the movement information when the playback time of the video reaches a period included in the movement information. The evaluation display unit 114 can also display the posture values included in the posture information. The evaluation display unit 114 can also display a graph of the time-series conversion of the posture values, based on a list of the posture values included in the movement information.

15, 16, and 17 are diagrams showing an example of a screen 42 output by the evaluation information output unit 216. In FIG. 15, the screen 42 shows bones 422 superimposed on image data 421 of the user 1. In the example of FIG. 15, the screen 42 shows evaluation comments 423 for posture information and evaluation comments 424 for movement information. In addition, the example of the screen 42 shows the number of times 425 of training information and the number of times 426 counted by the number counting unit 214. In the example of FIG. 16, an example of a graph 427 showing the speed of the shoulders, wrists, ankles, etc. is shown. In the example of FIG. 17, video data 423 of the training information is displayed. By displaying such a screen 42 on the display 22, the user 1 can correct the posture and movement of the body during training and perform efficient and effective training.

The evaluation information output unit 216 can also transmit the evaluation information to the user terminal 10. The user 1 can use the user terminal 10 to review the training during and after the training.

The evaluation information output unit 216 registers the training history of the user 1 in the training history storage unit 237. The evaluation information output unit 216 can register the training history including the user ID indicating the user 1, the date and time when the user authentication unit 210 authenticated the user 1, the menu ID corresponding to the display 22, the setting information, the number of times counted by the number counting unit 214, and the time from the time when the user was authenticated to the end of the training in the training history storage unit 237. Regarding the setting information, the display 22 may receive the setting information input from the user 1 (for example, the display 22 may be a touch panel display, and options or the like may be displayed on the touch panel display to receive the input), or the user terminal 10 may receive the setting information from the user terminal 10, or the training equipment may be provided with a communication function to obtain the setting information from the training equipment.

The training history transmission unit 217 transmits the training history to the user terminal 10. The training history transmission unit 217 may transmit training information to the user terminal 10 each time training is completed, or may respond with the training history in response to a request from the user terminal 10. The training history can be displayed on the user terminal 10. FIG. 18 is a diagram showing an example of the training history displayed by the user terminal 10. On the screen 43 shown in FIG. 18, the content 431 of the training performed by the user 1 and a video 432 of the user 1 during the training are displayed. In this way, the user 1 can look back on the state during the training, enabling effective training.

The training history transmission unit 217 transmits the training volume to the user terminal 10. The screen 44 shown in FIG. 19 shows an example of the training volume information output screen 44. As shown in FIG. 19, the training volume can be displayed along with the user's daily physical information such as height, weight, etc., and the daily, weekly, and monthly trends. Also, as shown in FIG. 19, by simultaneously displaying the training volume information along with the user's fat percentage, the user can compare the trends in fat percentage against the amount of work.

The training history transmission unit 217 can also transmit the training history to the administrator terminal 30. The training history transmission unit 217 may transmit training information to the administrator terminal 10 each time training is completed, or may respond with the training history in response to a request from the administrator terminal 30.

The improvement information output unit 218 outputs training improvement measures to the display 22. The improvement information output unit 218 searches for improvement condition information corresponding to the menu ID corresponding to the display 22, in which the physical information of the user 1 and the positions and movements of each part identified by the image analysis unit 213 satisfy the conditions. The improvement information output unit 218 acquires advice from the searched improvement information, and outputs the acquired advice to the display 22. The improvement information output unit 218 can also output the positions and angles of each part included in the reference information to the display 22. For example, the improvement information output unit 218 can display the state of the body skeleton (bones) based on the positions and angles of each part included in the reference position. That is, for example, a figure indicating a part (such as a circle) can be displayed superimposed on the image at the position indicated by the position information.

Figure 20 shows the flow of processing executed in the exercise support system of this embodiment.

The user authentication unit 210 of the server device 20 detects that the user 1 has come near the training equipment (S501). The user 1 may be detected, for example, by installing a human sensor near the display 22, or by monitoring an image from the camera 21 and detecting a human from the image.

Next, the user authentication unit 210 authenticates the user 1 (S502). The authentication of the user 1 can be performed, for example, by extracting facial features from the image and comparing them with the features included in the authentication information of the user information. The user authentication unit 210 may also authenticate the user 1 by, for example, communication with the user terminal 10 (for example, by accepting input of authentication information such as a user ID and password or a public key from the user terminal 10). An IC card reader may also be installed near the display 22 to perform authentication using a membership card.

When the user 1 is authenticated, the image acquisition unit 212 starts acquiring images from the camera 21 and registering them in the image data storage unit 233 (S503). The training information output unit 211 identifies a menu ID corresponding to the display 22 from the training information storage unit 231, reads out training information corresponding to the identified menu ID from the training information storage unit 231 (S504), and modifies the read out training information using user information corresponding to the user 1 (S505). The training information output unit 211 can modify the weight and number of repetitions, for example, according to the height, weight, muscle mass, and the like. The training information output unit 211 may also modify the training information according to the training history of the user 1. The training information output unit 211 outputs the modified training information to the display 22 (S506). The training information output unit 211 may accept an input of a purpose from the user 1 here, and read out training information corresponding to the purpose and the menu ID. An example of the output of training information is, for example, the screen 42 shown in FIG. 13.

The image analysis unit 213 analyzes the image data acquired from the camera 21 to identify the position of each part of the user 1's body (S507). The number of times counting unit 214 counts the number of times the user 1 has performed training based on the positions of the parts (S508). For example, the number of times counting unit 214 can count one leg flexion and extension when the knee angle is within a predetermined angle, or count one squat when the position of the buttocks is within a predetermined distance above the floor.

The evaluation information output unit 216 superimposes the position of each part and its connection relationship on the captured image of the user 1, thus displaying what is called bones (S509). The evaluation unit 215 also evaluates the training based on the positions and movements of the parts (time-series changes in position or angle), and the evaluation information output unit 216 outputs the evaluation results to the display 22 (S510). Here, if the user 1 is wearing an audio output device such as earphones, headphones, or a neck speaker, the evaluation results can also be output as audio.

The improvement measure information output unit 218 searches the improvement condition storage unit 238 for improvement condition information that satisfies the conditions of the physical information of the user 1 and the position and movement of each part identified by the image analysis unit 213, and outputs advice for the searched improvement condition information to the display 22 (S511). Note that the improvement measure information output unit 218 may also output other items included in the improvement condition information to the display 22. Furthermore, if the user 1 is wearing an audio output device such as earphones, headphones, or a neck speaker, information such as advice included in the improvement condition information may be output as audio.

Finally, the number counting unit 214 can register the training history in the training history storage unit 237 (S512). The number counting unit 214 may register the history each time it counts the number of training sessions, or when it detects the end of training because a predetermined time has passed since it became impossible to count the training sessions, etc.

As described above, according to the exercise support system of this embodiment, information such as training instructions is displayed on the display 22 installed on each training device, and the body of the user 1 photographed by the camera 21 is analyzed to provide an evaluation of posture and movement and suggestions for improvement. Therefore, since the system can explain the equipment and evaluate and give advice on body movement, which were previously done manually by personal trainers, it is possible to improve the efficiency of instruction for the user 1.

In addition, the exercise support system of this embodiment allows easy evaluation of physical exercise. Since the positional relationship and movement of each part of the body can be evaluated for physical exercise related to training, this is likely to lead to concrete efforts for improvement, and is expected to improve the effectiveness of training. In addition, the exercise support system of this embodiment also provides comments and advice, allowing the user to easily understand the current situation and improvement measures.

Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the interpretation of the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

For example, in this embodiment, the image analysis is performed in the server device 20, but this is not limited to the above. The image analysis may be performed in the camera 21 or the user terminal 10 to determine the positional relationship of each part.

In addition, in this embodiment, it is assumed that the position of the part is a position on a two-dimensional image, but this is not limited to this, and it may be a three-dimensional position. For example, in addition to the camera 21, a depth camera can be placed at the location where the training equipment is installed, and the three-dimensional position of the part can be identified based on the image from the camera 21 and the depth map from the depth camera. It is also possible to provide a depth camera instead of the camera 21, and identify the three-dimensional position from only the depth map from the depth camera.

In addition, in this embodiment, it is assumed that the parts of the user's body captured by the camera are parts seen from the front or side of the user, but this is not limited to the above. Taking advantage of the characteristics of a camera installed above the user, it is also possible to capture parts seen from above the user's head. In this case, the two-dimensional positions of the center of the user's head and both shoulders as viewed from above can be identified and analyzed.

In addition, in this embodiment, an image of the user's body during exercise is transmitted from the camera 21 to the server device 20, but this is not limited to the above. A computer may be placed at the location where the training equipment is installed, and the computer may extract features from the image and transmit the features to the server device 20. Alternatively, the computer may estimate body parts based on the features, obtain the absolute position of the part (which may be the position on the XY coordinate system of the image, or the actual distance from a reference position (for example, the ground, the toes, the head, the center of gravity of the body, etc.), or the position in any other coordinate system) or the relative positional relationship between multiple parts, and transmit these absolute positions and relative positional relationships to the server device 20.

In addition, in this embodiment, the improvement information is provided as content prepared by the server device 20, but this is not limited thereto. For example, it is also possible to include reference values and display bones that result in the correct posture (position and angle of each part) based on the reference values superimposed on a video or a still image extracted from the video. This makes it easy to understand what posture the user should have.

In addition, in this embodiment, the position or movement (position over time) of a body part is evaluated, but this is not limiting, and the position of a tool held or worn by the user may be identified and evaluated. In this case, reference information and evaluation condition information regarding the tool position may be registered, and the feature values of the tool may be extracted to identify the position, and the evaluation may be performed in the same way as for the position of a body part.

In addition, in this embodiment, content such as advice is provided as a remedy, but for example, equipment recommendations may be made. In this case, the server device 20 stores the equipment and a reference value for the size (length, etc.) of the equipment in association with the user's physical information (height, weight, etc.), extracts features of the equipment used by the user from the image data to identify the shape of the equipment, estimates the size of the equipment based on the shape and the user's size (e.g. height) included in the physical information, and if the difference between the estimated equipment size and the reference value is equal to or greater than a predetermined threshold, recommends equipment of the reference size.

In addition, by estimating the position of the tool, the server device 20 can estimate the speed of the tool. In addition, by extracting a pattern of changes in the tool position over time, the server device 20 can estimate the number of times the pattern occurs as the number of actions using the tool.

In addition, when a certain posture or movement is detected, a task for that movement may be suggested. In this case, the server device 20 may store a task in association with one or a series of postures or movements, instead of an evaluation comment, and output the task.

In addition, when a certain posture or movement is detected, the training to be performed may be presented. In this case, the server device 20 may store the training content in association with one or a series of postures or movements, instead of an evaluation comment, and output the content.

It is also possible to automatically detect the actions performed by the player. In this case, the server device 20 stores the posture (position of each body part) in which a specific action such as shooting or passing is performed as reference information, and can identify the action performed by the player in the image by comparing the positions of the body parts analyzed from the image with the reference information.

In addition, in this embodiment, the execution of a specific function and the storage of information are performed by the user terminal or the server device, but this is not limited thereto, and the execution of the function and the storage of information may be performed by either one of the devices. Alternatively, the functional unit and the storage unit may be provided in a distributed manner in a form different from this embodiment.

In addition, in this embodiment, the reference information (position reference information and angle reference information) is set in advance, but this is not limiting, and the reference information may be created from image data of multiple users 1. In this case, for example, input as to whether the posture or movement of user 1 is correct or not may be accepted, and the reference information may be created based only on those that are evaluated as correct.

In addition, in this embodiment, information about the training, evaluation results, etc. are displayed on the display 22, or output as audio from an audio output device worn by the user 1, but a message encouraging the training may be displayed on the display 22, or a cheer encouraging the training may be output as audio from an audio output device worn by the user 1. This can improve the motivation of the user 1 for training.

REFERENCE SIGNS LIST 10 User terminal 20 Server device 21 Camera 22 Display 23 Communication network 30 Administrator terminal 210 User authentication unit 211 Training information output unit 212 Image acquisition unit 213 Image analysis unit 214 Number count unit 215 Evaluation unit 216 Evaluation information transmission unit 231 Training information storage unit 232 User information storage unit 233 Image data storage unit 234 Reference information storage unit 235 Evaluation condition information storage unit 236 Training plan storage unit 237 Training history storage unit 238 Improvement condition storage unit

Claims (9)

A system for assisting a user in exercising, comprising:
a camera that photographs the user while exercising;
A display installed in a place where the exercise is performed;
an explanation output unit that outputs an explanation regarding exercise to the display;
a body part identification unit that analyzes an image captured by the camera to identify a position of a body part of the user;
an output unit that superimposes the position of the part on the image and outputs the image to the display;
an improvement information storage unit that stores improvement information including advice consisting of character strings excluding character strings consisting of numerical values in association with a condition of a position or movement of the body part of the user ;
Equipped with
the output unit acquires, from the remedy information storage unit, the remedy information including advice that the position or movement of the part identified by the identification unit satisfies a condition, and presents the remedy information ;
An exercise support system characterized by the above. The exercise support system according to claim 1 ,
the explanation output unit outputs, to the display, an explanation including at least video data showing a method of training;
An exercise support system characterized by the above. The exercise support system according to claim 2,
a reference value storage unit that stores a reference value related to the position of at least one of the parts;
an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value of the movement;
Equipped with
the output unit displays, as improvement measure information, a state of the body skeleton based on the reference value by superimposing it on the image;
An exercise support system characterized by the above. The exercise support system according to claim 3,
the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion;
the part identification unit identifies each of the first to third parts from the image;
the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value;
An exercise support system characterized by the above. 5. The exercise support system according to claim 1,
A number counting unit that counts the number of times of the exercise based on the position of the part is further provided,
The output unit further outputs the number of times of the exercise to the display.
An exercise support system characterized by the above. The exercise support system according to claim 3 or 4,
The output unit further outputs the evaluation value by voice.
An exercise support system characterized by the above. 5. The exercise support system according to claim 1,
the output unit further outputs the remedy information by voice based on the position of the user's body identified by the part identification unit;
An exercise support system characterized by the above. A camera that photographs a user while the user is exercising;
A display installed in the exercise area;
A processor;
A program for supporting the user's exercise by an exercise support system comprising:
The processor,
an explanation output step of outputting an explanation regarding the exercise to the display;
a part identification step of analyzing an image captured by the camera to identify a position of a part of the user's body;
an output step of superimposing the position of the part on the image and outputting the image to the display;
an improvement information storage step of storing improvement information including advice consisting of character strings excluding character strings consisting of numerical values, in association with the condition of the position or movement of the body part;
Run the command,
the output step includes acquiring, from the remedy information storage step, the remedy information including advice that the position or movement of the part identified in the identification step satisfies a condition, and presenting the remedy information;
An exercise support program that features: A camera that photographs a user while the user is exercising;
A display installed in the exercise area;
A processor;
A method for supporting the exercise of the user by an exercise support system comprising:
The processor,
an explanation output step of outputting an explanation regarding the exercise to the display;
a part identification step of analyzing an image captured by the camera to identify a position of a part of the user's body;
an output step of superimposing the position of the part on the image and outputting the image to the display;
an improvement information storage step of storing improvement information including advice consisting of character strings excluding character strings consisting of numerical values, in association with the condition of the position or movement of the body part;
Run the command,
the output step includes acquiring, from the remedy information storage step, the remedy information including advice that the position or movement of the part identified in the identification step satisfies a condition, and presenting the remedy information;
An exercise support method comprising: