JP7415567B2 - Climbing support device, computer program, climbing support method, learning model generation method and display system - Google Patents

Description

The present invention relates to a climbing support device, a computer program, a climbing support method, a learning model generation method, and a display system.

In recent years, bouldering, which is one of the ``climbing sports'' among various sports, has been gaining popularity. Bouldering is attracting attention as a highly effective sport for fitness because it is easy to get started and is a fun way to exercise.

Patent Document 1 discloses information that guides an inexperienced user such as a beginner to a route by guiding a user wearing a wearable device as to where to move the left and right limbs during climbing (target hold). A processing device is disclosed.

JP2016-75498A

However, since the device disclosed in Patent Document 1 only guides the user's limbs in order, it is not possible to take the entire course from the start to the goal (understand the overall image of the course or how to climb the entire route). In addition, in order to climb successfully, it is necessary to understand the movement of the center of gravity of the entire body based on the course taken, but the device of Patent Document 1 does not provide such guidance. Furthermore, in the device of Patent Document 1, it is necessary to attach a beacon to the hold in order to confirm the position of the user, which increases the cost of introducing the equipment.

The present invention has been made in view of the above circumstances, and aims to provide a climbing support device, a computer program, a climbing support method, a learning model generation method, and a display system that can improve climbing skills. .

A climbing support device according to an embodiment of the present invention includes: a course information acquisition unit that acquires course information of a climbing wall; a skeletal information acquisition unit that acquires skeletal information of a climber at a first time point on the climbing wall; and a learning model that uses the course information and skeletal information obtained as input data and estimates time-series skeletal information of the climber after a second time point after the first time point.

A computer program according to an embodiment of the present invention causes a computer to perform a process of acquiring course information of a climbing wall, a process of acquiring skeletal information of a climber at a first point in time on the climbing wall, and a process of acquiring course information of a climber on the climbing wall. A process of inputting skeletal information into a learning model and estimating time-series skeletal information of the climber after a second time point after the first time point is executed.

A climbing support method according to an embodiment of the present invention acquires course information of a climbing wall, acquires skeletal information of a climber at a first point in time on the climbing wall, and uses the acquired course information and skeletal information as a learning model. input data to estimate time-series skeletal information of the climber from a second point in time after the first point in time.

A learning model generation method according to an embodiment of the present invention acquires course information of a climbing wall, acquires time-series skeleton information at each input point of a climber on the climbing wall, and acquires the acquired course information and time-series skeleton information. A learning model that has learned a climbing method is generated based on learning data that uses skeletal information as input data and uses time-series skeletal information of the climber after the input time as output data.

A display system according to an embodiment of the present invention includes a generation unit that generates a virtual image of a climber based on time-series skeletal information estimated by a learning model that has learned a climbing method of the climber, and a virtual image generated by the generation unit. and a display section that displays images.

According to the present invention, climbing skills can be improved.

FIG. 1 is a schematic diagram showing an example of the configuration of a climbing support system according to the present embodiment. FIG. 2 is a block diagram showing an example of the configuration of a user terminal device. It is a schematic diagram showing an example of a climbing wall. FIG. 2 is a block diagram showing an example of the configuration of a server. It is a schematic diagram showing an example of course information. FIG. 2 is a schematic diagram showing an example of a predetermined part of a human body. FIG. 3 is a schematic diagram showing an example of skeleton information. FIG. 2 is a schematic diagram showing a first example of a learning model generation method. FIG. 7 is a schematic diagram showing a second example of a learning model generation method. FIG. 7 is a schematic diagram showing a third example of a learning model generation method. FIG. 2 is a schematic diagram showing a first example of an estimation method using a learning model. FIG. 7 is a schematic diagram showing a second example of an estimation method using a learning model. FIG. 7 is a schematic diagram showing a third example of an estimation method using a learning model. FIG. 2 is a schematic diagram showing an example of a virtual image. FIG. 2 is a schematic diagram showing an example of a climbing course selection method. FIG. 2 is a schematic diagram showing an example of a method of displaying a virtual image. 3 is a flowchart illustrating an example of a processing procedure of a learning model generation method. 12 is a flowchart illustrating an example of a processing procedure of a method for estimating an optimal climbing method using a learning model.

Embodiments of the present invention will be described below based on the drawings. FIG. 1 is a schematic diagram showing an example of the configuration of a climbing support system according to the present embodiment. The climbing support system is also a display system and includes a server 50 as a climbing support device. The climbing support system is connected to a user terminal device 10 used by a user (climber), a climbing wall 25 installed in a sports facility such as a climbing gym, a facility terminal device 20 installed in the sports facility, and a facility terminal device 20. It is equipped with a camera 21, a projector 22, and the like. The server 50, the user terminal device 10, and the facility terminal device 20 are connected via the communication network 1. The climbing support system supports users (climbers) to learn how to climb the climbing wall 25 correctly, even in the absence of an instructor or trainer, depending on their level of proficiency (for example, beginner, expert, etc.). It is something. Note that although bouldering will be described as an example of climbing in this specification, the present embodiment is also applicable to "climbing sports" other than bouldering.

FIG. 2 is a block diagram showing an example of the configuration of the user terminal device 10. As shown in FIG. The user terminal device 10 may be, for example, a smartphone, but may also be a tablet terminal, a notebook personal computer, or the like. The user terminal device 10 includes a control section 11 that controls the entire device, a camera section 12, a communication section 13, a storage section 14, an operation section 15, a display panel 16, and a processing section 17. The processing section 17 includes a learning section 171, a behavior prediction section 172, and an output processing section 173.

The control unit 11 can be configured with a CPU, ROM, RAM, and the like. The control unit 11 can execute a climbing support application.

The camera unit 12 can shoot moving images and still images.

The communication unit 13 provides a communication function with the server 50 via the communication network 1. The communication unit 13 can, for example, transmit images (videos and still images) captured by the camera unit 12 to the server 50.

The storage unit 14 can be configured with a semiconductor memory or the like, and stores necessary information such as images taken by the camera unit 12, information received via the communication unit 13, processing results by the control unit 11, and climbing support applications. can do.

The display panel 16 can be configured with a liquid crystal panel, an organic EL (Electro Luminescence) display, or the like. The operation unit 15 can be configured with, for example, a touch panel built into the display panel 16, and allows a user to perform predetermined operations on the display panel 16. Further, the operation unit 15 can perform operations on a keyboard displayed on the display panel 16. Note that the operation unit 15 may be a hardware keyboard, a mouse, or the like.

Although the processing unit 17 is not an essential component, details will be described later.

The facility terminal device 20 can be, for example, a smartphone, a tablet terminal, a notebook personal computer, a desktop personal computer, or the like. The facility terminal device 20 can transmit moving images and still images captured by the camera 21 to the server 50. Furthermore, the facility terminal device 20 can receive the information transmitted by the server 50, output the received information as an image, and project the image onto the climbing wall 25 via the projector 22.

FIG. 3 is a schematic diagram showing an example of the climbing wall 25. The climbing wall 25 is, for example, a wall with a height of about 5 m, and a large number of colorful protrusions called holds are provided on the wall surface. A wall surface may be composed of a plurality of sloped surfaces. Bouldering is a sport in which a user climbs a climbing wall 25 along a set course using holds. Multiple holds are set for each course from the start to the finish. Courses can be distinguished, for example, by the color of the tape pasted near the holds (including the shape of the tape, the numbers written on the tape, etc.), the color of the holds themselves, etc.

FIG. 4 is a block diagram showing an example of the configuration of the server 50. The server 50 includes a control section 51 that controls the entire server 50, a communication section 52, a storage section 53, a user DB 54, and a processing section 55. The processing section 55 includes a learning section 551, a behavior prediction section 552, and an output processing section 553. The control unit 51 can be configured with a CPU, ROM, RAM, and the like. The control unit 51 can execute climbing support software.

The communication unit 52 provides a communication function between the user terminal device 10 and the facility terminal device 20 via the communication network 1. The communication unit 52 can transmit and receive required information between the user terminal device 10 and the facility terminal device 20.

The storage unit 53 can be configured with an HDD, a semiconductor memory, or the like, and can store necessary information such as information received via the communication unit 52, processing results by the control unit 51, and climbing support software.

The user DB 54 records the user name, e-mail address, telephone number, user's bouldering skill level, facility usage history, user's height, weight, age, gender, etc. for each user. Information recorded in the user DB 54 can be input from the user terminal device 10 or the facility terminal device 20.

The processing unit 55 has a function as an image acquisition unit, and can acquire an image of the climbing wall 25 taken by the camera 21 or the camera unit 12 of the user terminal device 10 via the communication unit 52.

The processing unit 55 has a function as a selection unit and can select a climbing course based on the acquired image. Images of the climbing wall also include images of various holds. The holds are distinguished by color or by letters or symbols depending on the difficulty of the course. By recognizing these colors, characters, or symbols through image processing, for example, holds of the same color can be recognized separately from other holds, and a desired course can be selected. Note that the course selection may be determined by the user using the user terminal device 10. For example, when a user selects a desired level of difficulty, holds are determined based on the selected level of difficulty, and a climbing course can be selected.

The processing unit 55 has functions as a course information generation unit and a course information acquisition unit, and can generate course information based on a required or selected climbing course. Specifically, the processing unit 55 generates course information based on the position information, type, and wall condition information of each hold on the climbing course.

FIG. 5 is a schematic diagram showing an example of course information. In the example of FIG. 5, it is assumed that there are holds H1, H2, . . . , Hm from the start of the course to the finish. The course information is composed of items such as the X coordinate, Y coordinate, Z coordinate, slope of the wall (which may include irregularities), and the type of hold (for example, shape and size) for each hold. For example, the X coordinate, Y coordinate, and Z coordinate can be determined using a predetermined position of the climbing wall 25 as a reference. As shown in FIG. 5, for hold H1, the X coordinate is Hx1, the Y coordinate is Hy1, the Z coordinate is Hz1, the wall inclination is Hα1, and the type of hold is ty1. The same applies to other holds. Note that the number of course information for one climbing wall 25 is not limited to one, and for example, there may be a plurality of course information as shown in FIG. 5 depending on the difficulty level of the course.

In addition, when generating or acquiring course information, in addition to images taken with a camera, etc., sensors such as distance sensors, lasers, and ultrasonic sensors may be used to acquire various information on holds on the course. .

The processing unit 55 can acquire an image of the user on the climbing wall 25 taken by the camera 21 or the camera unit 12 of the user terminal device 10 via the communication unit 52. The processing unit 55 has a function as a skeletal information acquisition unit, and can acquire skeletal information based on the user's image. Note that the skeletal information is not limited to the method of acquiring it based on images, but can also be acquired from a 3D laser, an inertial sensor attached to the user, or the like.

More specifically, the processing unit 55 uses deep learning to detect feature points of a human body's posture based on a 2D image (moving image) (for example, OpenPose, VisionPose). The positions of a plurality of predetermined parts (such as joints) can be estimated, and the user's posture can be determined based on the estimated positions of the predetermined parts.

FIG. 6 is a schematic diagram showing an example of a predetermined part of the human body. As shown in FIG. 6, the predetermined parts include, for example, the right eye, left eye, nose, right ear, left ear, neck, right shoulder, left shoulder, right wrist, left wrist, right elbow, left elbow, right hip, left hip, It can be the right knee, left knee, right ankle, or left ankle. Note that the predetermined portion may be a part of these portions.

FIG. 7 is a schematic diagram showing an example of skeleton information. In the example of FIG. 7, NO. 1 to NO. IDs up to 18 are attached. The skeletal information is composed of items such as X coordinate, Y coordinate, Z coordinate, and reliability for each part. For example, the X coordinate, Y coordinate, and Z coordinate can be determined using a predetermined position of the climbing wall 25 as a reference. As shown in FIG. 1, the X coordinate is determined as x1(t), the Y coordinate as Y1(t), the Z coordinate as z1(t), and the reliability as R1(t). The same applies to other parts. Here, t can be the time of any frame of the video. That is, skeleton information as shown in FIG. 7 can be acquired for each frame.

Next, a method of generating a learning model by the learning unit 551 will be described.

The learning unit 551 uses hardware such as a CPU (for example, a multi-processor equipped with a plurality of processor cores), a GPU (Graphics Processing Units), a DSP (Digital Signal Processor), and an FPGA (Field-Programmable Gate Array). It can be configured by combining. Note that the learning unit 551 may use a neural network model or other machine learning.

The learning unit 551 uses the acquired course information and time-series skeleton information as input data, and generates a learning model that has learned a climbing method based on learning data that uses time-series skeleton information after the time of the input data as output data. can do. The learning data may be obtained by collecting a large amount of data from users who have successfully climbed from the start to the goal, or data from experts may be used. The learning method will be explained below.

FIG. 8 is a schematic diagram showing a first example of a method for generating the learning model 551a. The learning model 551a is a neural network model including deep learning, and for example, RNN (Recurrent Neural Network), LSTM (Long short-term memory) which is an extension of RNN, Seq2Seq, etc. can be used. , other models may also be used. The learning model 551a is composed of an input layer, an output layer, and a plurality of intermediate layers. Although two intermediate layers are illustrated in FIG. 8 for convenience, the number of intermediate layers is not limited to two, and may be three or more.

A plurality of nodes (neurons) exist in the input layer, output layer, and intermediate layer, and the nodes in each layer are connected in one direction with the nodes in the previous and subsequent layers with a desired weight. A vector having the same number of components as the number of nodes in the input layer is given as input data for learning of the learning model 551a, and a vector having the same number of components as the number of nodes in the output layer is given as input data for learning of the learning model 551a. Given as output data.

The input data for learning can be vectorized by combining course information and skeleton information. As shown in FIG. 8, the input data for learning can be time series data of times t1, t2, ..., tn. Note that although the skeleton information changes from time to time, the course information does not change. The output data for learning can be vectorized skeleton information. As shown in FIG. 8, the output data for learning can be time series data of times t2, t3, . . . , t(n+1), with the first time being time t2 after time t1. The learning model 551a can be generated by determining parameters such as weighting coefficients between nodes using the learning data. Thereby, the learning model 551a can use the course information and the user's skeletal information at a certain point in time as input data to estimate the user's time-series skeletal information after that point in time.

FIG. 9 is a schematic diagram showing a second example of the method for generating the learning model 551a. The input data for learning can be vectorized by combining course information and skeleton information. As shown in FIG. 9, the input data for learning can be time series data at time ti. Note that although the skeleton information changes from time to time, the course information may not change. The output data for learning can be vectorized skeleton information. As shown in FIG. 9, the output data for learning is time series data of times t(i+1), t(i+2), ..., t(i+k) after time t(i+1) after time ti. be able to. The learning model 551a can be generated by determining parameters such as weighting coefficients between nodes using the learning data. Thereby, the learning model 551a can use the course information and the user's skeletal information at a certain point in time as input data to estimate the user's time-series skeletal information after that point in time. Note that the course information may be the entire course from the start to the goal, but may also be a part of the course, such as from the user's current location during climbing to the goal.

FIG. 10 is a schematic diagram showing a third example of the method for generating the learning model 551a. The difference from the second example is that accompanying information is added to the learning input data. The accompanying information may be, for example, body shape information including the user's height and weight. This makes it possible to estimate skeletal information that takes into account the user's body shape. Further, the accompanying information may be an index representing at least one of the difficulty level of the climbing course and the user's skill level. This makes it possible to estimate skeletal information that takes into account the difficulty level of the climbing course and the user's skill level.

Note that the learning model may be generated for each climbing wall 25, or may be generated for each climbing course even for the same climbing wall 25, or may be generated for each user's skill level. Further, the course information may be a feature amount of an image taken of the course. For example, an image taken of a course may be input to a convolutional neural network, and the feature quantities output by the convolutional neural network may be used as course information.

Next, a method of estimating the optimal way to climb using the generated learning model 551a will be described.

The behavior prediction unit 552 can input the course information of the climbing course and the user's skeletal information into the learning model 551a, and estimate time-series skeletal information indicating the optimal way to climb on the climbing course. For example, the behavior prediction unit 552 can estimate time-series skeletal information after the start time based on the user's skeletal information at the start time. The user's skeletal information at the start point may be the skeletal information of the state in which the user grasps the hold at the start point, or if the user does not know how to grasp the hold at the start point, the skeletal information may be , initial values (for example, 0, 0, . . . ) may be given as skeleton information at the start point. Furthermore, the behavior prediction unit 552 can estimate time-series skeletal information from a time point after the user's climbing point, based on the skeletal information at a point in time during the user's climbing.

As described above, since the learning model 551a is generated based on learning data including course information and time-series skeleton information, the learning model 551a is generated based on learning data that includes course information and time-series skeleton information, so the learning model 551a is generated based on learning data that includes course information and time-series skeleton information. Information can be estimated. This makes it possible to grasp the overall picture of the course, and also understand how to climb the course as a whole and how the center of gravity of the entire body shifts based on the course taken, thereby improving climbing techniques.

FIG. 11 is a schematic diagram showing a first example of the estimation method using the learning model 551a. The estimation method shown in FIG. 11 corresponds to the generation method shown in FIG. 8. The input data can be vectorized by combining course information and skeleton information. As shown in FIG. 11, when time series data at time ti is input to the learning model 551a as input data, the learning model 551a outputs the estimated value of the skeleton information at time t(i+1) after time ti. do. By inputting time-series input data (for example, i=1, 2, ...) to the learning model 551a, the learning model 551a acquires skeleton information at time t(i+1) (time-series data of i=1, 2, ...). skeletal information) can be output.

FIG. 12 is a schematic diagram showing a second example of the estimation method using the learning model 551a. The estimation method shown in FIG. 12 corresponds to the generation method shown in FIG. As shown in FIG. 12, when time-series data at time ti is input to the learning model 551a as input data, the learning model 551a calculates the time t(i+1), after time t(i+1) after time ti, Estimates of skeletal information at t(i+2), . . . , t(i+k) can be output.

FIG. 13 is a schematic diagram showing a third example of the estimation method using the learning model 551a. The estimation method shown in FIG. 13 corresponds to the generation method shown in FIG. As shown in FIG. 13, accompanying information can be added to the time series data at time ti as input data. The accompanying information may be, for example, body shape information including the user's height and weight. This makes it possible to estimate skeletal information that takes into account the user's body shape. Further, the accompanying information may be an index representing at least one of the difficulty level of the climbing course and the user's skill level. This makes it possible to estimate skeletal information that takes into account the difficulty level of the climbing course and the user's skill level.

The output processing unit 553 has a function as an output unit, and can output a virtual image of the user (for example, visual information such as an avatar) corresponding to the time-series skeleton information estimated by the learning model 551a.

FIG. 14 is a schematic diagram showing an example of a virtual image. A virtual image based on time-series skeletal information can be a video that shows the movement of the entire body, including the movement of the center of gravity. Since you can see the movement of the climber in the video, you can improve your climbing technique, for example, even if an instructor or trainer is not present. Note that the virtual image may be an image of a character whose display mode differs depending on the user's gender, body shape, skill level, and the like. Further, virtual images such as character images may be displayed in advance in a selectable manner on the user terminal device 10 so that the user can select a desired character image.

FIG. 15 is a schematic diagram showing an example of a climbing course selection method. In the following, a method for selecting a climbing course by taking an image of the climbing wall 25 with the user terminal device 10 will be explained, but the method is not limited to this, and a similar method can be used using the facility terminal device 20 and the camera 21. can be processed.

As shown in FIG. 15, the photographed image of the climbing wall 25 is displayed on the display screen 101 of the display panel 16. On the selection screen 102, the difficulty level of each course is displayed using, for example, a plurality of selectable icons. In the example of FIG. 15, icons for 4th to 5th grade, 6th to 9th grade, 1st dan...2nd dan are displayed. Icons for other grades or levels can be displayed by scrolling the screen. For example, when the user selects an icon for grades 6 to 9, a plurality of holds corresponding to the selected difficulty level are displayed on the image of the climbing wall 25 in a display mode 103 (in the illustrated example, indicated by circles). (enclosed). When the user operates the determination icon 104, the selected climbing course is determined, and course information of the determined climbing course is generated.

FIG. 16 is a schematic diagram showing an example of a method for displaying a virtual image. In FIG. 16, the image indicated by a solid line is an actual user or an image of the user, and the image indicated by a broken line is a virtual image (an image indicating the optimal way to climb). First, assume that the image indicated by the solid line is an actual user climbing the climbing wall 25. In this case, the output processing unit 553 can project the virtual image onto the climbing wall 25 via the projector 22. The output processing unit 553 can project the virtual image onto the climbing wall 25 via the projector 22 in synchronization with the movement of the user climbing the climbing wall 25. Synchronization can mean, for example, the same time and the same display rate (frame rate). This allows the user to understand the difference between his own climbing method and the correct climbing method for each part of his body while climbing, and allows him to improve his climbing technique, for example, even in the absence of an instructor or trainer.

Next, assume that the image indicated by the solid line is the user's image. In this case, the output processing unit 553 can synchronize and display the user's image displayed on the display panel 16 and the virtual image. The display panel 16 has a display mode that allows the user's image and the virtual image to be distinguished, and can display them in a superimposed manner. This allows the user to understand the difference between his own climbing method and the correct climbing method for each part of his body while climbing, and allows him to improve his climbing technique, for example, even in the absence of an instructor or trainer.

The processing unit 55 can acquire skeletal information of a user who climbs the climbing wall 25, and update the virtual image based on the skeletal information estimated by the learning model 551a using the acquired skeletal information and course information as input data. The display panel 16 can display the updated virtual image. As a result, even if the user starts climbing the climbing wall 25 in the wrong way, deviating from the correct way, the virtual image is updated according to the user's current situation, so the user can reach the goal. You can figure out the correct way to climb up to the top. Note that U-Net (Auto Encoder) or the like may be used as a means for directly outputting a virtual image without using skeleton information.

FIG. 17 is a flowchart illustrating an example of the processing procedure of the method for generating the learning model 551a. Hereinafter, for convenience, the main body of processing will be described as the control unit 51. The control unit 51 acquires course information (S11), and acquires time-series skeletal data (time-series skeletal information) of an expert or a successful climber (S12).

The control unit 51 generates or updates the learning model 551a using the acquired information as learning data (S13), and determines the presence or absence of learning data (S14). If there is learning data (YES in S14), the control unit 51 repeats the processing from step S11 onwards, and if there is no learning data (NO in S14), the control unit 51 records the learning model 551a in the storage unit 53 (S15). , ends the process.

FIG. 18 is a flowchart illustrating an example of a processing procedure of a method for estimating the optimal climbing method using the learning model 551a. The control unit 51 acquires course information (S21) and acquires skeletal data of the user at the first time point (S22). The control unit 51 inputs the course information and skeletal data to the learning model 551a, and estimates time-series skeletal data at or after the second time point (S23). The second time point is a time point after the first time point, and may be, for example, one frame later or several frames later. The processes shown in FIGS. 17 and 18 may be performed separately depending on the user's body shape and skill level. Further, in steps S11 and S21, in addition to the course information, additional information such as the user's body shape and skill level may be acquired.

The control unit 51 generates a virtual image based on the estimated time-series skeleton data (S24), synchronizes and outputs the user's image and the virtual image (S25), and determines whether the user has reached the goal or not. is determined (S26). If the user has not reached the goal (NO in S26), the control unit 51 repeats the process from step S22 onwards, and if the user has reached the goal (YES in S26), the control unit 51 ends the process.

The server 50 can also be implemented using a computer equipped with a CPU (processor), RAM, and the like. A computer program (recordable on a recording medium) that defines the processing procedure shown in FIGS. 17 and 18 is read by a recording medium reading unit installed in the computer, the read computer program is loaded into RAM, and the computer program is read. By executing on a CPU (processor), the server 50 can be realized on a computer.

In the above-described embodiment, the server 50 includes the processing section 55, but the user terminal device 10 may also include the processing section 17 and perform the same processing as the server 50. In this case, the learning section 171, the behavior prediction section 172, and the output processing section 173 have the same functions as the learning section 551, the behavior prediction section 552, and the output processing section 553, respectively.

As described above, according to the present embodiment, in the climbing sport, it is possible to teach techniques and skills of experts and provide beginners and inexperienced users with a navigation function for how to climb. It can help you find the best way to climb and the best way to climb.

In the embodiment described above, the virtual image is projected onto the climbing wall 25 using the projector 22, but the virtual image may be provided to a user wearing a wearable device using AR or VR technology. Alternatively, instructions may be issued using a laser point.

The climbing support device of the present embodiment includes a course information acquisition unit that acquires course information of a climbing wall, a skeletal information acquisition unit that acquires skeletal information of a climber at a first point in time on the climbing wall, and the acquired course information. and a learning model that uses the skeleton information as input data and estimates time-series skeleton information of the climber after a second time point after the first time point.

The computer program of this embodiment causes a computer to perform a process of acquiring course information of a climbing wall, a process of acquiring skeletal information of a climber at a first point in time on the climbing wall, and a process of acquiring course information and skeletal information of the climber on the climbing wall. A process of inputting the information into a learning model and estimating time-series skeletal information of the climber from a second time point onward after the first time point is executed.

The climbing support method of the present embodiment acquires course information of a climbing wall, acquires skeletal information of a climber at a first point in time on the climbing wall, and inputs the acquired course information and skeletal information into a learning model. Then, time-series skeletal information of the climber is estimated from a second point in time after the first point in time.

The course information acquisition unit acquires course information of the climbing wall. The course information includes the position information of each of the multiple holds from the start of the climbing wall to the goal (e.g., x coordinate, y coordinate, z coordinate from the reference coordinates, etc.), the type of each of the multiple holds (e.g., size, shape, etc.) etc.), and includes state information (for example, the inclination angle and unevenness of the wall) of the wall on which each of the plurality of holds is provided. Note that the number of course information for one climbing wall is not limited to one, and there may be a plurality of pieces of course information depending on the difficulty level, for example.

The skeletal information acquisition unit acquires skeletal information of the climber on the climbing wall at a first point in time. The skeleton information includes position information of a predetermined part of the climber. The predetermined parts are parts of the human body, for example, right eye, left eye, nose, right ear, left ear, neck, right shoulder, left shoulder, right wrist, left wrist, right elbow, left elbow, right hip, left hip, right It can be the knee, left knee, right ankle, or left ankle. Note that the predetermined portion may be a part of these portions. The position coordinates can be, for example, the x, y, and z coordinates from the reference coordinates. The first point in time may be, for example, the start point of the climb or any point during the climb.

The learning model is generated in advance based on learning data including course information and time-series skeleton information. The learning model uses the acquired course information and skeletal information as input data, and estimates time-series skeletal information of the climber from a second time point onward after the first time point. For example, the learning model can estimate time-series skeletal information after the start time based on the skeletal information at the start time of the climber. The skeletal information at the start may be the skeletal information of the climber holding the hold at the start, or if the climber does not know how to hold the hold at the start and is not holding the hold. , an initial value (for example, 0, 0, . . . ) may be given as the skeleton information to serve as the skeleton information at the start point. Furthermore, the learning model can estimate time-series skeletal information from a point after the climber's point in time based on the skeletal information at a point in time during the climb of the climber.

As mentioned above, the learning model is generated based on learning data that includes course information and time-series skeleton information, so the time-series skeleton information is generated in a manner that reflects the overall state of the course from the start to the goal. can be estimated. This makes it possible to grasp the overall picture of the course, and also understand how to climb the course as a whole and how the center of gravity of the entire body shifts based on the course taken, thereby improving climbing techniques.

The climbing support device of this embodiment includes an output unit that outputs a virtual image of the climber corresponding to the time-series skeleton information estimated by the learning model.

The output unit outputs a virtual image of the climber corresponding to the time-series skeleton information estimated by the learning model. A virtual image of a climber based on time-series skeletal information can be a video that shows the movement of the entire body, including the movement of the center of gravity, and not only the next movement position of the left and right limbs, but also the course taking, body posture, Since the movement of the entire body can be seen in video, it is possible to improve climbing techniques, for example, even when an instructor or trainer is not present.

In the climbing support device of this embodiment, the output unit includes a projector that projects the virtual image onto the climbing wall.

The output section includes a projector that projects the virtual image onto the climbing wall. Since a virtual image is projected onto the climbing wall using a projector, climbers themselves can understand the correct way to climb from the start to the goal at the beginning of the climb, or they can check their own climbing method and the correct way to climb during the climb. It is possible to understand the differences in each part of the body, and for example, it is possible to improve climbing techniques even if an instructor or trainer is not present.

In the climbing support device of this embodiment, the first time point is the start time of climbing or the middle time of climbing.

The first point in time is the start point of climbing or a point in the middle of climbing. The skeletal information at the start of climbing may be the skeletal information in the state where the user is holding the hold at the start, or if the user does not know how to hold the hold at the start, the skeletal information may be An initial value (for example, 0, 0, . . . ) may be given as skeleton information at the starting point. This allows the user to know the correct way to climb from the start to the goal at the beginning of the climb. Also, even in the middle of climbing, you can figure out the correct way to climb from that point to the goal.

The climbing support device according to the present embodiment includes an image acquisition unit that acquires an image of the climbing wall, a selection unit that selects a climbing course based on the image acquired by the image acquisition unit, and a climbing wall selected by the selection unit. and a course information generation section that generates the course information based on a course, and the course information acquisition section acquires the course information generated by the course information generation section.

The image acquisition unit acquires an image of the climbing wall. Images of the climbing wall also include images of various holds. The holds are distinguished by color or by letters or symbols depending on the difficulty of the course.

The selection unit selects a climbing course based on the acquired image. For example, by selecting a desired difficulty level for a climber, holds are determined based on the selected difficulty level, and a climbing course can be selected.

The course information generation unit generates course information based on the selected climbing course. Specifically, course information is generated based on the position information, type, and wall condition information of each hold on the selected climbing course.

The course information acquisition unit can acquire the generated course information. Thereby, it is possible to select a desired climbing course according to the climber's skill level and obtain course information.

In the climbing support device according to the present embodiment, the course information generation unit includes position information of each of the plurality of holds on the climbing course, the type of each of the plurality of holds, and the state of the wall on which each of the plurality of holds is provided. Generate course information containing information.

The course information generation unit can generate course information including position information of each of the plurality of holds on the climbing course, types of each of the plurality of holds, and state information of the wall on which each of the plurality of holds is provided. Thereby, it is possible to select a desired climbing course according to the climber's own skill level and generate course information.

In the climbing support device of this embodiment, the skeleton information includes position information of a predetermined part of the climber.

The skeleton information includes position information of a predetermined part of the climber. The predetermined parts are parts of the human body, for example, right eye, left eye, nose, right ear, left ear, neck, right shoulder, left shoulder, right wrist, left wrist, right elbow, left elbow, right hip, left hip, right It can be the knee, left knee, right ankle, or left ankle. This allows you to understand your body posture and overall body movements.

In the climbing support device of this embodiment, the learning model estimates the skeletal information of the climber using body shape information including height and weight of the climber as input data.

The learning model can estimate the skeletal information of the climber using body shape information including the climber's height and weight as input data. This allows you to understand the correct way to climb, taking into account the climber's body shape.

In the climbing support device of the present embodiment, the learning model estimates the skeletal information of the climber using as input data an index representing at least one of the difficulty level of the climbing course and the climber's level of improvement.

The learning model can estimate the skeletal information of the climber using as input data an index representing at least one of the difficulty level of the climbing course and the climber's skill level. This allows you to understand the correct way to climb, taking into account the difficulty of the climbing course and the climber's proficiency level.

The learning model generation method of the present embodiment acquires course information of a climbing wall, acquires time-series skeletal information at each input point of the climber on the climbing wall, and uses the acquired course information and time-series skeletal information. A learning model that has learned a climbing method is generated based on learning data in which the input data is time-series skeletal information of the climber after the input time as output data.

The learning model generation method acquires course information of a climbing wall. The course information includes the position information of each of the multiple holds from the start of the climbing wall to the goal (e.g., x coordinate, y coordinate, z coordinate from the reference coordinates, etc.), the type of each of the multiple holds (e.g., size, shape, etc.) etc.), and includes state information (for example, the inclination angle and unevenness of the wall) of the wall on which each of the plurality of holds is provided. Note that the number of course information for one climbing wall is not limited to one, and there may be a plurality of pieces of course information depending on the difficulty level, for example.

The learning model generation method acquires time-series skeletal information at each input point of the climber on the climbing wall. The skeleton information includes position information of a predetermined part of the climber. The predetermined parts are parts of the human body, for example, right eye, left eye, nose, right ear, left ear, neck, right shoulder, left shoulder, right wrist, left wrist, right elbow, left elbow, right hip, left hip, right It can be the knee, left knee, right ankle, or left ankle. Note that the predetermined portion may be a part of these portions. The position coordinates can be, for example, the x, y, and z coordinates from the reference coordinates.

The learning model generation method uses acquired course information and time-series skeletal information as input data, and creates a learning model that has learned climbing methods based on learning data that uses time-series skeletal information of the climber after the input time as output data. generate. The learning data may be obtained by collecting a large amount of data on users who have successfully climbed from the start to the goal, or may be learning data categorized by user body shape information, skill level, difficulty level of the climbing course, etc.

Thereby, the learning model uses the course information and the skeletal information of the climber at a certain point as input data, and can estimate the time-series skeletal information of the climber after that point.

The learning model generation method of this embodiment generates the learning model based on learning data whose input data is body shape information including height and weight of the climber.

The learning model generation method can generate a learning model based on learning data that uses body shape information including height and weight of a climber as input data. This makes it possible to estimate skeletal information that takes into account the climber's body shape.

The learning model generation method of this embodiment generates the learning model based on learning data whose input data is an index representing at least one of the difficulty level of the climbing course and the climber's skill level.

The learning model generation method can generate a learning model based on learning data whose input data is an index representing at least one of the difficulty level of a climbing course and the climber's skill level. This makes it possible to estimate skeletal information that takes into account the difficulty of the climbing course and the climber's skill level.

The display system of this embodiment includes a generation unit that generates a virtual image of a climber based on time-series skeletal information estimated by a learning model that has learned the climbing method of the climber, and a display system that displays the virtual image generated by the generation unit. and a display section.

The generation unit generates a virtual image of the climber based on time-series skeletal information estimated by a learning model that has learned the climber's climbing method. A virtual image of a climber based on time-series skeletal information can be a video that shows the movement of the entire body, including the movement of the center of gravity, and not only the next movement position of the left and right limbs, but also the course taking, body posture, You can see the movements of your entire body in the video.

The display unit displays the generated virtual image. This makes it possible to understand course planning, body posture, and overall body movement through video, so it is possible to improve climbing techniques, for example, even in the absence of an instructor or trainer.

The display system of the present embodiment includes an image acquisition section that acquires an image of a climber on a climbing wall, and the display section synchronizes and displays the image of the climber and the virtual image.

The image acquisition unit acquires an image of the climber on the climbing wall. The image of the climber can be, for example, an image of the climber in the middle of climbing. The display unit displays the image of the climber and the virtual image in synchronization. Synchronization can mean, for example, the same time and the same display rate (frame rate). The display unit has a display mode in which the image of the climber and the virtual image can be distinguished, and can be displayed in a superimposed manner on the display screen. This allows the user to understand the difference between his own climbing method and the correct climbing method for each part of his body while climbing, and allows him to improve his climbing technique, for example, even in the absence of an instructor or trainer.

In the display system of this embodiment, the display unit includes a projector, and the projector projects the virtual image onto the climbing wall in synchronization with the movement of a climber climbing the climbing wall.

The display section includes a projector. The projector projects a virtual image onto the climbing wall in synchronization with the movement of the climber climbing the wall. This allows the user to understand the difference between his own climbing method and the correct climbing method for each part of his body while climbing, and allows him to improve his climbing technique, for example, even in the absence of an instructor or trainer.

The display system of this embodiment includes a skeletal information acquisition unit that acquires skeletal information of a climber climbing a climbing wall, a course information acquisition unit that acquires course information of the climbing wall, and inputs the acquired skeletal information and course information. and an updating section that updates the virtual image generated by the generating section based on skeleton information estimated by the learning model as data, and the display section displays the virtual image updated by the updating section.

The skeletal information acquisition unit acquires skeletal information of a climber climbing a climbing wall. The course information acquisition unit acquires course information of a climbing wall that a climber climbs. The updating unit updates the virtual image generated by the generation unit based on the skeletal information estimated by the learning model using the acquired skeletal information and course information as input data. The display unit displays the virtual image updated by the update unit. As a result, even if a climber starts climbing the wrong way from the correct way while climbing the climbing wall, the virtual image will be updated according to the climber's situation, so they will know the correct way to climb to the goal. can do.

1 Communication network 10 User terminal device 11 Control unit 12 Camera unit 13 Communication unit 14 Storage unit 15 Operation unit 16 Display panel 17 Processing unit 171 Learning unit 172 Behavior prediction unit 173 Output processing unit 20 Facility terminal device 21 Camera 22 Projector 50 Server 51 Control unit 52 Communication unit 53 Storage unit 54 User DB
55 Processing unit 551 Learning unit 551a Learning model 552 Behavior prediction unit 553 Output processing unit

Claims (17)

a course information acquisition unit that acquires course information for a climbing wall;
a skeletal information acquisition unit that acquires skeletal information at a first point in time of the climber on the climbing wall;
A climbing support device comprising: a learning model that uses the acquired course information and skeletal information as input data and estimates time-series skeletal information of the climber after a second time point after the first time point. The climbing support device according to claim 1, further comprising an output unit that outputs a virtual image of the climber corresponding to the time-series skeleton information estimated by the learning model. The output section is
The climbing support device according to claim 2, further comprising a projector that projects the virtual image onto the climbing wall. The climbing support device according to any one of claims 1 to 3, wherein the first time point is a starting point of climbing or a point in the middle of climbing. an image acquisition unit that acquires an image of the climbing wall;
a selection unit that selects a climbing course based on the image acquired by the image acquisition unit;
a course information generation unit that generates the course information based on the climbing course selected by the selection unit;
The course information acquisition unit
The climbing support device according to any one of claims 1 to 4, wherein the climbing support device acquires course information generated by the course information generation unit. The course information generation unit includes:
The climbing support device according to claim 5, wherein the climbing support device generates course information including position information of each of the plurality of holds on the climbing course, types of each of the plurality of holds, and state information of a wall on which each of the plurality of holds is provided. . The skeleton information is
The climbing support device according to any one of claims 1 to 6, including position information of a predetermined part of the climber. The learning model is
The climbing support device according to any one of claims 1 to 7, wherein skeletal information of the climber is estimated using body shape information including the climber's height and weight as input data. The learning model is
The climbing support device according to any one of claims 1 to 8, wherein the skeletal information of the climber is estimated by using as input data an index representing at least one of the difficulty level of the climbing course and the climber's skill level. to the computer,
Processing to obtain climbing wall course information;
a process of acquiring skeletal information at a first point in time of the climber on the climbing wall;
A computer program that executes a process of inputting acquired course information and skeletal information into a learning model to estimate time-series skeletal information of the climber from a second time point onward after the first time point. Get climbing wall course information,
obtaining skeletal information of the climber on the climbing wall at a first point in time;
A climbing support method that inputs acquired course information and skeletal information into a learning model to estimate time-series skeletal information of the climber after a second time point after the first time point. Get climbing wall course information,
Obtaining time-series skeleton information at each input point of the climber on the climbing wall,
A learning model that generates a learning model that has learned a climbing method based on learning data in which the acquired course information and time-series skeleton information are used as input data, and the time-series skeleton information of the climber after the input time is used as output data. Generation method. 13. The learning model generation method according to claim 12, wherein the learning model is generated based on learning data whose input data is body shape information including height and weight of the climber. 14. The learning model generation method according to claim 12 or 13, wherein the learning model is generated based on learning data whose input data is an index representing at least one of a difficulty level of a climbing course and a climber's proficiency level. a generation unit that generates a virtual image of the climber based on time-series skeletal information estimated by a learning model that has learned how to climb the climber;
a display unit that displays the virtual image generated by the generation unit ;
an image acquisition unit that acquires an image of a climber on a climbing wall;
Equipped with
The display section is
A display system that synchronizes and displays the image of the climber and the virtual image . The display unit includes a projector,
The projector includes:
The display system according to claim 15, wherein the virtual image is projected onto the climbing wall in synchronization with the movement of a climber climbing the climbing wall. a skeletal information acquisition unit that acquires skeletal information of a climber climbing a climbing wall;
a course information acquisition unit that acquires course information of the climbing wall;
an updating unit that updates the virtual image generated by the generating unit based on the skeletal information estimated by the learning model using the acquired skeletal information and course information as input data,
The display section is
The display system according to any one of claims 15 to 16 , which displays a virtual image updated by the update unit.

Images