JP7840230B2 - Online lesson coaching support device - Google Patents

Description

This disclosure relates, for example, to a coaching support device for online lessons.

For health, yoga, dance, and freil prevention, it is common for users to go to a gym or studio and receive lessons in person from a coach. However, this can be a significant burden for users. Therefore, in recent years, online lessons have been proposed, where users receive lessons online, for example, via video call (see, for example, Patent Document 1).

Japanese Patent Publication No. 2022-023867

However, online lessons can present challenges, such as difficulty for coaches to assess a user's situation or inability to effectively communicate appropriate instruction to the user.
Considering these circumstances, the objective of the present invention is to support coaches so that they can provide appropriate guidance to users in online lessons.

To solve the above problems, an online lesson coach support device according to one aspect of the present disclosure is an online lesson coach support device that receives a video signal and an audio signal of a user during a lesson from a terminal device via a network, and comprises: an estimation unit that estimates the user's emotions based on the video signal and the audio signal; an evaluation unit that evaluates the user's exercise form based on the video signal; and an instruction scenario generation unit that generates an instruction scenario for the user based on at least the emotion estimation result by the estimation unit and the exercise form evaluation result by the evaluation unit.
According to the coach support device for online lessons in the above-described embodiment, it is possible to support coaches so that they can provide appropriate guidance to users during online lessons.

This diagram shows the schematic configuration of the online lesson system. This diagram shows the configuration of the coach support device in the system. This block shows the functions built into the coach support device. This diagram shows the configuration of terminal devices in the system. This diagram shows the processing and data flow in the estimation and evaluation units. This diagram shows the processing and data flow in the instruction scenario generation unit. This is a flowchart showing the operation of the coaching support device. This figure shows an example of a screen displayed by a coaching support device. This figure shows an example of a screen displayed on a terminal device. This figure shows another example of a screen displayed on a coaching support device. This figure shows an example of a screen displayed on a terminal device.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
Figure 1 is a schematic diagram of an online lesson system 1 including a coach support device 10 according to an embodiment. In the online lesson system 1, the coach support device 10 and a plurality of terminal devices 20-1, 20-2, ..., 20-n are connected via a network 30.
In this embodiment, online lessons are possible not only between a user and a coach one-on-one, but also between multiple users and coaches in a multi-to-one format.

The coach support device 10 is, for example, a personal computer and is operated by a coach who instructs the user in online lessons. The coach support device 10 has a program installed for providing online lessons.

Terminal devices 20-1, 20-2, 20-3, ..., 20-n are, for example, smartphones or personal computers, and are operated by users taking online lessons.
In the following, to describe a typical terminal device, terminal devices 20-1, 20-2, 20-3, ..., 20-n will be simply referred to as "20," omitting the hyphen for identification. Each terminal device 20 has a program (browser) installed for taking online lessons.

Figure 2 shows the hardware configuration of the coach support device 10. The coach support device 10 includes a processing device 110, a storage device 120, a display device 130, a communication device 140, a camera 150, and an audio input/output device 160.
Although not specifically shown in the diagram, the coach support device 10 has an operation input device such as a keyboard that accepts operations entered by the coach.

The processing unit 110 is composed of, for example, one or more processors and controls each element of the coach support device 10. Specifically, the processing unit 110 is composed of one or more types of processors, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

The storage device 120 stores the program executed by the processing device 110 and various data used by the processing device 110. As the storage device 120, known recording media such as semiconductor recording media and magnetic recording media, or a combination of multiple types of recording media, may be used.
The display device 130 is composed of, for example, a liquid crystal display panel or an organic EL panel, and displays video including various images under the control of the processing device 110. Specifically, in online lessons, the display device 130 displays video of one or more users, video of the coach, advice to the user, etc.

The communication device 140 communicates with each of the terminal devices 20 under the control of the processing device 110.
The camera 150 is a camera that captures the coach's entire body during online lessons.
The audio input/output device 160 consists of a microphone for inputting the coach's voice and a speaker for outputting the user's voice.

Figure 3 is a block diagram showing the functional configuration of the coach support device 10, which is constructed using the processing unit 110 and the storage device 120.
The processing unit 110 constructs multiple elements (control management unit 111, estimation unit 112, evaluation unit 113, guidance scenario generation unit 114, and presentation unit 115) by executing a program stored in the storage device 120.

The control and management unit 111 manages the control and progress of each part of the online lesson.
The estimation unit 112 analyzes the user's video and audio transmitted from the terminal device to estimate the user's emotions.
The evaluation unit 113 analyzes the video signal transmitted from the terminal device to estimate the user's posture and evaluates the user's movement form based on the estimated user posture.

The instruction scenario generation unit 114 generates an instruction scenario, including advice for the user, based on the user's emotion estimation results from the estimation unit 112, the user's exercise form evaluation results from the evaluation unit 113, as well as exercise instruction data, exercise menu data, instruction judgment rule data, basic user information, dialogue data, and coach information.
The various data and information necessary for generating the instruction scenarios will be described later. Furthermore, the instruction scenarios will include not only advice for users, but also information to support coaches in online lessons, such as what coaches should do during online lessons.

The presentation unit 115 controls the display content on the display device 230 to present various information to the coach. Specifically, the presentation unit 115 presents advice based on the coaching scenario to the coach, and, as described later, if there are multiple users, it displays which user's coaching scenario has been generated. The presentation unit 115 also causes the communication device 140 to transmit the coach's audio and video signals to the terminal device 20. As a result, the coach's audio and video are displayed on the display device 230 in the terminal device 20.

The control management unit 111 manages exercise instruction content, user basic information, and coach information using the database DB created in the storage device 120. The exercise instruction content includes exercise form data, exercise instruction data, exercise menu data, and instruction judgment rule data. The database DB also includes a learning database for analyzing user statements.

Figure 4 shows the hardware configuration of terminal device 12. Terminal device 20 includes a processing unit 210, a storage device 220, a display device 230, a communication device 240, a camera 250, and an audio input/output device 260.

The processing unit 210 controls each element of the terminal device 20.
The storage device 220 stores programs executed by the processing device 210 and various data used by the processing device 210. The programs executed by the processing device 210 include the OS as well as programs for receiving online lessons.
The display device 230 displays various images under the control of the processing device 210. Specifically, in online lessons, the display device 230 displays images from video calls with the coach, etc.

The communication device 240 communicates with the coach support device 10 under the control of the processing device 210.
The imaging device 250 is a camera that captures the user's entire body during online lessons.
The audio input/output device 260 consists of a microphone for inputting the user's voice and a speaker for outputting the coach's voice.
Although not specifically shown in the diagram, the terminal device 20 is equipped with a touch panel that is superimposed on the image display surface of the display device 230, and accepts user operations.
The functional configuration of the processing unit 210 and the storage device 220 in the terminal device 20 is not important in this case, so its explanation will be omitted.

As described above, this embodiment makes it possible to provide online lessons to multiple users simultaneously. However, for the sake of simplicity, Figures 5 to 8 illustrate the case where there is only one user of the online lesson.

Figure 5 is a diagram showing the flow of various data and various processes in the estimation unit 112 and the evaluation unit 113.
In online lessons, users perform various exercise forms (poses/movements) while video conferring with their coach. Specifically, the voice spoken by the user is input to the audio input/output device 260 of the terminal device 20, and the user's entire body is captured by the camera device 250.
The audio signal output from the audio input/output device and the video signal output from the camera 250 are transmitted to the coach support device 10 via the communication device 140 and the network 30.

In the coach support device 10, the received audio and video signals are transferred as follows: The audio signal is transferred to the estimation unit 112 as the user's voice (Vo), and the video signal is transferred to both the estimation unit 112 and the evaluation unit 113 as the user's video (Vi).

The estimation unit 112 estimates the user's emotions based on the results of analyzing the user's voice Vo in four different ways and the results of analyzing the user's video Vi.
The four types of analysis processes are language analysis process 112a, tone analysis process 112b, speed analysis process 112c, and timing analysis process 112d.
The language analysis process 112a analyzes the user's voice Vo and identifies the meaning of the voice.
The tone analysis process 112b analyzes the tone (pitch) of the user's voice Vo. For example, the tone analysis process 112b determines whether the voice tone is high/normal/low. The speed analysis process 112c analyzes the speed of the user's voice Vo. For example, the speed analysis process 112c determines whether the voice speed is fast/normal/slow.

Even if the semantic content identified by language analysis process 112a is the same, the actual meaning may differ depending on the tone and speed of the user's speech. Therefore, the semantic content identified by language analysis process 112a may be modified based on the results of tone analysis process 112b and speed analysis process 112c.

The interval analysis process 112d analyzes the user's voice (Vo) to determine the continuity of the speech. Specifically, if the user's voice is temporally discrete, the interval analysis process 112d determines whether the discrete voice segments constitute a continuous speech.

The analysis process of the user's video (Vi) in the estimation unit 112 is the facial expression analysis process 112e. The facial expression analysis process 112e focuses on the user's face in the user's video (Vi) and analyzes the user's facial expressions.

The emotion estimation process 112f in the estimation unit 112 estimates the user's emotion based on at least one of the user's facial expression analyzed by the facial expression analysis process 112e and the user's speech indicated by the user's voice Vo. Specifically, the emotion estimation process 112f estimates the user's emotion by appropriately combining five results: the result from the language analysis process 112a, the result from the tone analysis process 112b, the result from the speed analysis process 112c, the result from the timing analysis process 112d, and the result from the facial expression analysis process 112e. Examples of the user's emotion to be estimated include joy/anger/sadness/happiness, pleasure/displeasure, arousal/de-arousal, positive/negative, etc.
In addition, the facial expression analysis process 112e estimates the user's emotions in real time.

The evaluation unit 113 evaluates the user's exercise form based on the results of the analysis of the user's video Vi and the exercise instruction content Eg stored in the database DB.
The analysis process of the user's video Vi in the evaluation unit 113 is the posture estimation process 113a. The posture estimation process 113a focuses on the user's whole body in the user's video Vi and estimates the user's posture. For example, statically, the posture estimation process 113a focuses on the user's joints (shoulders, arms, wrists, ankles, etc.) and specific body parts (navel, etc.) and estimates the opening angle and coordinates of the joints and body parts. Dynamically, the posture estimation 113a estimates the amount of movement and direction of movement per unit time at the joints and body parts.

The exercise instruction content Eg includes exercise form data, exercise support commentary data, exercise menu data, and instruction policy data.
Of these, the exercise form data is data that shows the posture and movements that the user should adopt during online lessons. In other words, the exercise form data is data that shows exemplary posture and movements. The exercise form data is supplied to the evaluation unit 113 as 113b.

The exercise form evaluation process 113c compares the user's exercise form estimated by the posture estimation process 113a with the exemplary exercise form shown in the exercise form data 113b to evaluate the user's exercise form in online lessons in real time.
In evaluating exercise form, the evaluation results may include data indicating how closely the user's exercise form in the online lesson resembles the model exercise form. For example, in a given exercise, the model form might be one in which the straight line calculated by connecting the user's hand to the shoulder joint position remains horizontal for several seconds. The evaluation results could then be a binary value indicating whether the straight line calculated from the actual user's video is good if it is within 15 degrees of horizontal, or bad if it is not. Alternatively, the evaluation results could include an index (similarity) indicating how closely the opening angles and coordinates of the user's joints and body parts resemble the model.

Figure 6 is a diagram showing the flow of various data and various processes in the instruction scenario generation unit 114.
The instruction scenario generation unit 114 generates an instruction scenario based on the following data, information, and results. Specifically, the instruction scenario generation unit 114 uses emotion estimation results 114a, exercise form evaluation results 114b, exercise instruction data 114c, exercise menu data 114d, instruction judgment rule data 114e, user basic information 114f, dialogue data 114g, and coach information 114h when generating an instruction scenario.

The emotion estimation result 114a is the result of the estimation unit 112 estimating the emotions of the user taking the online lesson.
The exercise form evaluation result 114b is the result of the evaluation unit 113 evaluating the exercise form of a user taking an online lesson.

Exercise instruction data 114c is a collection of data showing the coach's facial expressions, movements, and statements during online lessons.
Although not specifically illustrated, data indicating the coach's facial expression can be obtained by focusing on the coach's face in the video Ci of the coach's entire body captured by the camera 150 and processing the coach's facial expression in the same manner as the facial expression analysis process 112e. The coach's movements can be obtained by processing the video Ci of the coach's entire body captured by the camera 150 in the same manner as the posture estimation process 113a. Data indicating the content of the coach's speech can be obtained by processing the coach's voice Co input by the voice input/output device 160 in the same manner as the language analysis process 112a.

The exercise menu data 114d is data that shows the content of the lessons provided by the online lesson system 1, and is data that combines the type of lesson, the body part that is expected to be affected (loaded) by the lesson, and the intensity and difficulty level. Specifically, the exercise menu data 114d is data that combines the type of lesson, such as frailty prevention A, B, C, ..., yoga A, B, C, ..., dance A, B, C, ..., the body part, such as upper arms, ankles, whole body, and the intensity and difficulty level, such as soft/medium/hard.
Coaching judgment rule data 114e contains data that shows the criteria a coach uses when giving instructions, such as data indicating gentle/normal/strict.
Furthermore, the exercise menu data 114d and the instruction judgment rule data 114e are included in the exercise instruction content Eg as described above.

User basic information 114f is information about the user receiving instruction from a coach in online lessons. For example, it includes information such as the user's name, age, gender, skill level, medical history, and exercise frequency. User basic information 114f is stored for each user in the user database of the storage device 120's database DB. In online lessons, the user basic information 114f corresponding to the user receiving instruction is read and used.

Dialogue data 114g is a collection of structured data, including the flow and content of the dialogue. For example, dialogue data 114g is stored in a database, specifically as a learning database.

Coach information 114h is information about the coach who instructs the user in the online lesson. For example, it includes information such as the coach's age, gender, teaching experience, qualifications, areas of expertise, teaching philosophy, and life satisfaction. Coach information 114h is stored for each coach in the coach database of the storage device 120, and in online lessons, the coach information 114h corresponding to the court where the user is being instructed is read and used.

The instruction scenario generation unit 114 generates an instruction scenario based on eight elements: emotion estimation result 114a, exercise form evaluation result 114b, exercise instruction data 114c, exercise menu data 114d, instruction judgment rule data 114e, user basic information 114f, dialogue data 114g, and coach information 114h.
Furthermore, as will be described later, when generating instruction scenarios using machine learning, the instruction scenario generation unit 114 may estimate and output instruction scenarios using information on other coaches and users, as well as information on past instruction. Specifically, while the past satisfaction levels for similar users and methods for calculating instruction scenarios with high effectiveness are learned for each coach, the instruction scenario generation unit 114 generates instruction scenarios for users based on the results of this learning.

The instruction scenarios may be generated using only rule-based methods with eight input elements, or solely using machine learning with eight input elements. Alternatively, some scenarios may be generated using machine learning, while the remaining scenarios are generated using rule-based methods. It is preferable to use pre-trained training data based on correct answers from excellent coaches when using machine learning.

Next, we will explain how online lesson system 1 works.
Online lessons are based on the premise that at least one user participates and that one coach instructs that user.
Users who wish to participate in online lessons operate the terminal device 20 to access the coach support device 10 in advance and input the information listed in the user basic information 114f. The input information is stored in the user database of the database DB, linked to the user's identifier.
Furthermore, the information listed in Coach Information 114h is assumed to already be stored in the coach database, linked to the identifier of the coach in question.

Users who wish to participate in an online lesson must connect to the coach support device 10 by operating their terminal device 20 before the predetermined lesson start date and time, and notify the device of their intention to participate in the online lesson. This lesson will follow the content indicated in the exercise menu data 114d.
When the lesson start time arrives, the control management unit 111 of the coach support device 10 identifies the users who have contacted them to participate before the start of the lesson.

Figure 7 is a flowchart showing the operation of the online lesson system 1 from the point in time when the users participating in the lesson are identified.
The control management unit 111, having identified the users participating in the lesson, reads the basic user information corresponding to the identified users and transfers it to the instruction scenario generation unit 114. The control management unit 111 also transfers the coach information corresponding to the coach who will be instructing the lesson to the instruction scenario generation unit 114 (step Sa1).

Next, the control management unit 111 transfers the exercise form data 113b corresponding to the lesson from the exercise instruction content Eg stored in the database DB to the evaluation unit 113. The control management unit 111 also transfers the exercise menu data 114d and instruction judgment rule data 114e corresponding to the lesson from the exercise instruction content Eg to the instruction scenario generation unit 114 (step Sa2).

When an online lesson begins, the terminal device 20 captures the entire body of the user taking the lesson using the camera device 250, and the resulting video signal is transmitted to the coach support device 10.
The user's voice is input to the audio input/output device, and the audio signal is transmitted to the coach support device 10. The control management unit 111 forwards the audio signal transmitted from the terminal device 20 as the user's voice Vo to the estimation unit 112, and forwards the video signal transmitted from the terminal device 20 as the user's video Vi to the estimation unit 112 and the evaluation unit 113.

The control management unit 111 determines whether or not it is time to generate a teaching scenario in the online lesson (step Sa3).
The timing for generating a coaching scenario is the moment when the coach is expected to provide instruction to the user. Examples of timing for generating a coaching scenario include when the user speaks, when a certain amount of time has elapsed since the start of the lesson, when the user is estimated to be experiencing a specific emotion, when the user's exercise form evaluation result falls below a threshold, when the exercise form evaluation result rises above a threshold, or combinations of these timings.
Furthermore, a user's exercise form evaluation result below the threshold indicates that the user's exercise form is inappropriate and should be corrected. Conversely, a user's exercise form evaluation result above the threshold indicates that the user's exercise form is appropriate and should be praised.
Furthermore, the timing for generating the instruction scenario could also be at the beginning of the lesson, when the coach informs the user of the proposed exercise menu, if the menu is decided before the lesson. Alternatively, the scenario could be generated at the end of the lesson, when the coach reviews the lesson's results and provides feedback to the user, if there is a process of reviewing the lesson's outcomes and assigning homework.

If it is determined that it is not time to generate a guidance scenario (i.e., the result of step Sa3 is "No"), the control management unit 111 returns the processing procedure to step Sa3. Therefore, the processing procedure cycles through step Sa3 and enters a waiting state until it is determined that it is time to generate a guidance scenario.

If the system determines that it is time to generate a coaching scenario (i.e., the result of step Sa3 is "Yes"), the control management unit 111 instructs the coaching scenario generation unit 114 to generate a coaching scenario based on the eight elements. Subsequently, the control management unit 111 instructs the display unit 115 to display information based on the generated coaching scenario (step Sa4). These instructions then cause the display device 130 to display coach advice and other information based on the coaching scenario in a predetermined area.

After step Sa4, the control management unit 111 returns the processing procedure to step Sa3. Therefore, each time it is determined that it is time to generate a guidance scenario, a command is issued to generate the guidance scenario, and advice and other information based on the generated guidance scenario are displayed in a predetermined area of the display device 130.

This process continues until the online lesson ends.
Note that steps Sa3 and Sa4 will be executed for each user if there are multiple users taking the online lesson.

In this lesson, the coach will adopt exercise forms appropriate to the type and difficulty level of the lesson. Meanwhile, the user will adopt various exercise forms under the coach's guidance. Both the coach and the user will speak as appropriate during the lesson.

Figure 8 shows an example of a screen displayed on the display device 130 of the coach support device 10 during an online lesson. This screen is an example where there are four users taking the lesson: A, B, C, and D. Figure 9 shows an example of a screen displayed on the display device 230 of the terminal device 20 operated by, for example, user A, one of the four users.

In the example shown in Figure 8, the display area of the display device 130 is divided into a total of six squares: two vertically and three horizontally. For convenience, these six squares are distinguished as upper left, upper center, upper right, lower left, lower center, and lower right. The upper left square displays User A's full-body image Ua, User A's profile picture, and User A's name "AA". Similarly, the upper center, lower left, and lower center squares display User B's full-body image Ub, User C's full-body image Uc, and User D's full-body image Ud, in that order, along with their profile pictures and user names.
The top right box displays the coach's full-body video (Ch) and profile picture. The bottom right box displays advice (Ad1) and (Ad2) based on the coaching scenario.

The black circles Bm superimposed on the whole-body images Ua, Ub, Uc, and Ud indicate the joints and specific body parts of users A, B, C, and D. These black circles are obtained through estimation by the posture estimation process 113a, which receives the user's image Vi as input. The black circles Bm serve as a guideline for coaches to evaluate the user's movement form.

In the example shown in Figure 9, the display device 230 shows scaled-down images of the coach's full-body video Ch and user A's own full-body video Ua.
The reason for displaying User A's full-body image Ua on User A's terminal device 20 is to allow User A to confirm how much their exercise form, as shown in their own full-body image Ua, differs from the exercise form shown in the coach's full-body image Ch.

Figure 9 shows an example of the display screen on user A's terminal device 20. Similarly, users B, C, and D will also see a reduced-size image of the coach's full body (Ch) and their own full body image.

Figure 8 shows an example where User B's exercise form evaluation result is below the threshold, and it is estimated that their emotional state is poor. Specifically, this example shows User B with joint angles in their arms and legs that are slightly wider than those of the coach, and with a distressed expression. In this case, it is time to generate a coaching scenario for User B (the judgment result in Step Sa3 is "Yes"), and a coaching scenario for User B in this situation is generated.

Advice Ad1 is a message based on the instruction scenario generated at this time. To inform the coach that an instruction scenario for user B has been generated on the display device 130 screen, the presentation unit 115 displays a marker Bkb with a thick dashed border in the upper middle square corresponding to user B, and also makes the marker Bkm blink.
Advice Ad1 is an example message based on the generated coaching scenario. By following the displayed Advice Ad1, even an inexperienced coach can provide appropriate guidance to User B at the right time and with the right content.

Furthermore, if the coach provides voice instructions to user B, the voice signal passes through the voice input/output device 160, communication device 140, network 30, communication device 240, and voice input/output device 260 in that order, and is converted into voice by the voice input/output device 260 and output.
Furthermore, the coach may transmit the audio spoken during instruction only to user B's terminal device 20, or to include the terminal devices 20 of the other users A, C, and D as well.

Figure 8 also shows an example where the motor form evaluation results for users A and C are above the threshold and their emotions are estimated to be positive. Specifically, this is an example where the joint angles of the limbs of users A and C are not significantly different from those of the coach, and their facial expressions indicate a state of joy. In this case as well, a coaching scenario for users A and C is generated.
Advice Ad2 is a message based on the instruction scenario generated at this time. To inform the coach that instruction scenarios for users A and C have been generated on the display device 130 screen, the presentation unit 115 displays and flashes a marker Bka with a thick solid border in the upper left square corresponding to user A, and a marker Bkc with a thick solid border in the lower left square corresponding to user C.

Advice Ad2 is an example message based on the generated coaching scenario. Following Advice Ad2, even an inexperienced coach can provide appropriate guidance, such as praising users A and C at the right time.

The reason for using a dashed line for marker Bkb and solid lines for markers Bka and Bkc is that the coaching scenario for user B is based on negative results, while the coaching scenarios for users A and C are based on positive results. This difference is intended to intuitively inform the coach of the difference between the two scenarios. If this difference can be distinguished, it may be represented by line color rather than dashed or solid lines.

When User D speaks during a lesson, their statement may be transcribed and displayed in text Cv1, as shown in the lower-middle square in Figure 8. In online lessons, because it's necessary to film the user's entire body, the user and the terminal device 20 are separated, which can make it difficult to hear the user's voice. Displaying the statement as text in this way prevents the coach from missing the user's comments.

The coach can evaluate the full-body video (Ua, Ub, Uc) and provide advice that differs from Ad1 and Ad2, or they can choose not to provide any guidance if they deem it unnecessary. In other words, the decision of whether or not to instruct the user according to the instruction scenario is left to the coach's own judgment.
In online lessons, the video and audio of both the user and the coach are recorded. This recording can be analyzed to provide coaches with feedback on what went well, what could have been improved, and whether the timing of their instruction was appropriate.
Additionally, if a change in the coach's skill level is recognized after the online lesson has ended, the coach's information may be updated.

Thus, according to the coach support device 10 of this embodiment, it is possible to support coaches so that they can provide appropriate instruction to users in online lessons at the appropriate time.

<Different example>
In the embodiment described above, the full-body video of the coach Ch was displayed on the display devices 130 and 230, but instead of the coach, the presentation unit 115 may display an avatar representing the coach. Figure 10 shows an example of a screen in which the presentation unit 115 displays an avatar At representing the coach in place of the full-body video of the coach Ch in the upper right square of the display device 130. Figure 11 shows an example of a screen in which the avatar At is displayed on the display device 230 of the terminal device 20 operated by user A.

Avatar A may be given movements similar to those of the coach's full-body video Ch. More specifically, the coach's posture may be estimated from the coach's full-body video Ch, the opening angles and coordinates of joints and body parts may be determined, and the parts corresponding to the joints and body parts in Avatar A may be given movements corresponding to the opening angles and coordinates.
Furthermore, avatar At may be given movements that follow the exemplary exercise form data 113b.

Even if an avatar (At) is displayed instead of the coach's full-body video (Ch), from the perspective of coach support, it is preferable that the coach is real, a coaching scenario is generated from eight elements, advice based on that coaching scenario is displayed, and the coach provides instruction based on that advice.
In this configuration, data characterizing the coach's voice quality may be pre-stored in the database DB of the storage device 120, and the presentation unit 115 may synthesize a message based on the generated coaching scenario and have the avatar At speak in place of the coach.
Furthermore, when one coach is giving online lessons to multiple users, if one user is having a high-priority conversation with the coach, and another user has a question, the presentation unit 115 may have a function to assist the coach, such as having the avatar At speak to the other user on behalf of the coach.

<Other>
The block diagram used in the description of the above embodiment shows functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may be realized by combining the one or more devices with software. Functions include, but are not limited to, judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communication, forwarding, configuration, reconfiguration, allocation (mapping), and assigning. For example, a functional block (configuration unit) that enables transmission is called a transmission unit or transmitter. As mentioned above, the method of implementation is not particularly limited.

Information notification is not limited to the embodiments described herein and may be carried out by other means. For example, information notification may be carried out by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block))), other signals, or combinations thereof. Furthermore, RRC signaling may also be called RRC messages, such as RRC Connection Setup messages and RRC Connection Reconfiguration messages.

Each aspect/embodiment described in this disclosure is LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (where x is, for example, an integer or decimal)), FRA (Future Radio) This may apply to systems utilizing Access, NR (new Radio), New Radio Access (NX), Future Generation Radio Access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), and other appropriate systems, as well as at least one of next-generation systems that are extended, modified, created, or defined based thereon. Furthermore, multiple systems may be combined and applied (for example, a combination of at least one of LTE and LTE-A with 5G).

The processing procedures, sequences, flowcharts, etc., of each aspect/embodiment described in this disclosure may be reordered, provided they are consistent. For example, the methods described in this disclosure present various step elements using exemplary order and are not limited to the specific order presented.

In this disclosure, certain operations performed by a base station may, in some cases, be performed by its upper node. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal can be performed by the base station and at least one other network node (e.g., an MME or S-GW, but not limited to these). While the above example illustrates a case where there is one other network node besides the base station, it may also be a combination of multiple other network nodes (e.g., an MME and an S-GW).

Information (described later) can be output from a higher layer (or lower layer) to a lower layer (or higher layer). Input and output may also occur via multiple network nodes.

Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.

The determination may be made using a single-bit value (0 or 1), a Boolean value (true or false), or a numerical comparison (for example, a comparison with a predetermined value).

Each aspect/embodiment described in this disclosure may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of specific information (e.g., notification of "being X") is not limited to explicit notification; it may also be implicit (e.g., by not providing such notification).

While the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in this disclosure are illustrative and not intended to be restrictive in any way.

Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.
Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technologies (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

The information, signals, etc., described in this disclosure may be represented using any of the various different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Terms used in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channels and symbols may be a signal (signaling). Furthermore, a signal may be a message. Also, a component carrier (CC) may be called a carrier frequency, cell, frequency carrier, etc.

The terms “System” and “Network” as used in this disclosure are interchangeable.

Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values from a given value, or corresponding other information. For example, radio resources may be indicated by an index. The names used for the parameters described above are not limiting in any way. Moreover, the formulas, etc., that use these parameters may differ from those expressly disclosed in this disclosure. Since various channels (e.g., PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, the various names assigned to these various channels and information elements are not limiting in any way.

In this disclosure, terms such as “Base Station (BS),” “Radio Base Station,” “Fixed Station,” “NodeB,” “eNodeB (eNB),” “gNodeB (gNB),” “Access Point,” “Transmission Point,” “Reception Point,” “Transmission/Reception Point,” “Cell,” “Sector,” “Cell Group,” “Carrier,” and “Component Carrier” may be used interchangeably. A base station may also be referred to by terms such as macrocell, small cell, femtocell, and picocell. A base station may house one or more (e.g., three) cells. When a base station accommodates multiple cells, the entire coverage area of the base station can be divided into several smaller areas, each of which may also be provided with communication services by a base station subsystem (e.g., a Remote Radio Head (RRH)). The terms "cell" or "sector" refer to a portion or all of the coverage area of at least one of the base station and/or base station subsystems providing communication services within that coverage. In this disclosure, the transmission of information by a base station to a terminal may be interpreted as the base station instructing the terminal to perform information-based control or operation.

In this disclosure, terms such as “Mobile Station (MS),” “user terminal,” “user equipment (UE),” and “terminal” may be used interchangeably. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms.

.
At least one of the base station and the mobile station may be called a transmitting device, a receiving device, a communication device, etc. At least one of the base station and the mobile station may also be a device mounted on a mobile body, the mobile body itself, etc. The mobile body refers to a movable object, and its speed of movement is arbitrary. This also includes the case when the mobile body is stationary. The mobile body includes, but is not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships (ship and other watercraft), airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and items mounted on them. The mobile body may also be a mobile body that moves autonomously based on operation commands. The base station may be a vehicle (e.g., a car, an airplane), an unmanned mobile device (e.g., a drone, an autonomous vehicle), or a robot (manned or unmanned). At least one of the base station and the mobile station may be a device that does not necessarily move during communication. For example, at least one of the base station and the mobile station may be an IoT (Internet of Things) device such as a sensor. Furthermore, the term "base station" in this disclosure may be replaced with "user terminal." For example, the various aspects/embodiments of this disclosure may be applied to a configuration in which communication between a base station and a user terminal is replaced with communication between multiple user terminals (for example, which may be called D2D (Device-to-Device), V2X (Vehicle-to-Everything), etc.). In this case, the user terminal may have the functions of the base station described above. Furthermore, terms such as "uplink" and "downlink" may be replaced with terms corresponding to terminal-to-terminal communication (for example, "side"). For example, uplink channel and downlink channel may be replaced with side channel. Similarly, a user terminal in this disclosure may be replaced with a base station. In this case, the base station may be configured to have the functions that the user terminal has.

The terms “determinating” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determinating” and “determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database or another data structure), and confirming, and considering these actions as “determining” or “determining.” Furthermore, "judgment" and "decision" can include considering something as a "judgment" or "decision" to have occurred when receiving (e.g., receiving information), transmitting (e.g., sending information), inputting, outputting, or accessing (e.g., accessing data in memory). "Judgment" and "decision" can also include considering something as a "judgment" or "decision" to have occurred when resolving, selecting, choosing, establishing, or comparing. In short, "judgment" and "decision" can include considering something as a "judgment" or "decision" to have occurred. Additionally, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

The terms “connected,” “coupled,” and any variations thereof mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be reinterpreted as “access.” As used in this disclosure, two elements may be considered “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical domain (both visible and invisible).

The reference signal can also be abbreviated as RS (Reference Signal), and may be called a Pilot depending on the applicable standard.

In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

Any reference to elements using designations such as “first,” “second,” etc., as used in this disclosure, does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Therefore, references to the first and second elements do not imply that only two elements may be employed, or that the first element must precede the second element in any way.

In the configuration of each of the above devices, "means" may be replaced with "part," "circuit," "device," etc.

Where the terms “include,” “including,” and their variations are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." Furthermore, the term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Therefore, the description herein is for illustrative purposes only and is not intended to be restrictive in any way to the present invention. Furthermore, multiple embodiments selected from those illustrated herein may be combined.

1…Online lesson system, 10…Coach support device, 20…Terminal device, 30…Network, 110…Processing device, 111…Control management unit, 112…Estimation unit, 113…Evaluation unit, 114…Instruction scenario generation unit, 114a…Emotion estimation result, 114b…Exercise form evaluation result, 114c…Exercise instruction data, 114d…Exercise menu data, 114e…Instruction judgment rule data, 114f…User basic information, 114g…Dialogue data, 114h…Coach basic information, 115…Presentation unit.

Claims (5)

A coach support device that assists a coach in online lessons, receiving video signals and audio signals from a user receiving lessons from a coach via a network from a terminal device,
An estimation unit that estimates the user's emotions based on the video signal and the audio signal,
An evaluation unit that evaluates the user's exercise form based on the aforementioned video signal,
At a minimum, a generation unit generates a guidance scenario for the user based on the emotion estimation result by the estimation unit and the exercise form evaluation result by the evaluation unit,
A presentation unit that presents a message based on the aforementioned coaching scenario to the coach,
It has,
When multiple users take a lesson at the same time,
The aforementioned display unit is,
Provide information indicating which users the proposed instructional scenario is intended for.
A coaching support device for online lessons.
The estimation unit,
Based on at least one of the user's facial expression shown in the video signal and the user's statement shown in the audio signal, the user's emotions are estimated.
The evaluation unit,
The online lesson coach support device according to claim 1, which evaluates the user's exercise form based on a comparison of the user's exercise form shown in the video signal with a model exercise form.
The aforementioned display unit is,
The coach is presented with a message based on the aforementioned coaching scenario.
The coach support device for online lessons according to claim 1.
The aforementioned display unit is,
The system presents the coach with a message based on the aforementioned instructional scenario and causes the coach to output an audio signal to the terminal device.
The coach support device for online lessons according to claim 1.
The aforementioned instructional scenario generation unit,
Furthermore, the exercise instruction data showing the content of the aforementioned coach's instruction,
Exercise menu data showing the content of the lesson,
Instructional decision-making rule data that shows the teaching policy for the lesson,
Basic user information identifying the user,
Dialogue data showing the content of the conversation the coach should have,
Information about the coach who instructs the user,
The online lesson coaching support device according to claim 1, which generates a coaching scenario for the user based on the above.