JP7842948B2 - Information processing methods, apparatus, and programs - Google Patents

Description

This invention relates to an information processing method, apparatus, and program.

Technologies that support rehabilitation, a form of treatment for individuals with injuries or disabilities, are known. Patent Document 1 discloses a technology that automatically determines a rehabilitation menu based on the individual's movements.

Japanese Patent Publication No. 2019-24580

For individuals with injuries or disabilities, treatment, including rehabilitation, is often painful, making it difficult to maintain motivation. The technology described in Patent Document 1 has room for improvement in terms of maintaining the motivation of the individuals involved.

This invention aims to suppress the decline in motivation of patients undergoing treatment to improve their physical function.

An information processing method according to one embodiment is an information processing method executed by an information processing device, and includes: an acquisition process for acquiring a subject video, which is a video of a subject performing a first action; an extraction process for extracting a skeletal video of the subject from the subject video; a learning process for training a generation model to generate the subject video from the skeletal video; and a generation process for inputting a skeletal video corresponding to a second action according to the level of injury or illness into the generation model, and generating an injury/illness level video, which is a video of the subject performing the second action according to the level of injury or illness.

According to one embodiment, it is possible to suppress the decline in motivation of patients undergoing treatment to improve their physical function.

This figure shows an example of the configuration of the information processing system according to this embodiment. This figure shows an example of the hardware configuration of an information processing device. This figure shows an example of the functional configuration of a video generation device. This flowchart shows an example of the process performed by a video generation device. This diagram schematically illustrates an example of the processing performed by a video generation device. This figure shows an example of a video display screen shown on a user's terminal.

The embodiments of the present invention will be described below with reference to the accompanying drawings. In the description and drawings of each embodiment, components having substantially the same functional configuration will be denoted by the same reference numerals, thus omitting redundant descriptions.

<System Configuration>
First, an overview of the information processing system according to this embodiment will be described. The information processing system according to this embodiment is a system for suppressing a decline in the motivation of a subject with an injury or illness by generating a video of the subject after treatment and showing the video to the subject. Injuries and illnesses include diseases and traumas. Treatment includes surgery, medication, and rehabilitation. The information processing system is used, for example, in medical facilities, nursing care facilities, or rehabilitation facilities.

Figure 1 shows an example of the configuration of the information processing system according to this embodiment. As shown in Figure 1, the information processing system according to this embodiment comprises a video generation device 1 and a user terminal 2, which are interconnected and able to communicate with each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public network, a mobile data communication network, or a combination thereof. In the example in Figure 1, the information processing system comprises one video generation device 1 and one user terminal 2, but it may comprise multiple instances of each.

The video generation device 1 is an information processing device that generates injury/illness level videos 127 based on the subject video 121. The video generation device 1 is, for example, a PC (Personal Computer), smartphone, tablet terminal, server device, or microcomputer, but is not limited to these. The video generation device 1, subject video 121, and injury/illness level videos 127 will be described in detail later.

User terminal 2 is an information processing device used by the user. The user may be, for example, a doctor, caregiver, occupational therapist, physical therapist, or nurse, but is not limited to these. User terminal 2 may be, for example, a PC, smartphone, tablet, server, or microcomputer, but is not limited to these. User terminal 2 transmits the subject video 121 to the video generation device 1 and receives and displays the injury/illness level video 127 from the video generation device 1. User terminal 2 may also acquire the subject video 121 from an external device and transmit the acquired subject video 121 to the video generation device 1. Furthermore, user terminal 2 may be equipped with a camera and transmit the subject video 121 captured by the camera to the video generation device 1. Note that the user terminal 2 transmitting the subject video 121 and the user terminal 2 displaying the injury/illness level video 127 may be different information processing devices.

<Hardware Configuration>
Next, the hardware configuration of the information processing device 100 will be described. Figure 2 is a diagram showing an example of the hardware configuration of the information processing device 100. As shown in Figure 2, the information processing device 100 comprises a processor 101, a memory 102, a storage 103, a communication I/F 104, an input/output I/F 105, and a drive device 106, all interconnected via a bus B.

The processor 101 controls the various components of the information processing device 100 and realizes the functions of the information processing device 100 by loading various programs, including the OS (Operating System), stored in the storage 103, into the memory 102 and executing them. The processor 101 is, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), or a combination thereof.

Memory 102 is, for example, ROM (Read Only Memory), RAM (Random Access Memory), or a combination thereof. ROM is, for example, PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), or a combination thereof. RAM is, for example, DRAM (Dynamic RAM), SRAM (Static RAM), or a combination thereof.

The storage device 103 stores various programs and data, including the operating system. The storage device 103 may be, for example, flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an SCM (Storage Class Memories), or a combination thereof.

The communication interface 104 is an interface for connecting the information processing device 100 to an external device via the network N and controlling communication. The communication interface 104 is, for example, an adapter compliant with Bluetooth®, Wi-Fi®, ZigBee®, Ethernet®, or optical communication.

The input/output interface 105 is an interface for connecting the input device 107 and the output device 108 to the video generation device 1. The input device 107 is, for example, a mouse, keyboard, touch panel, microphone, scanner, camera, various sensors, operation buttons, or a combination thereof. The output device 108 is, for example, a display, projector, printer, speaker, vibrator, or a combination thereof.

The drive device 106 reads and writes data to the disk media 109. The drive device 106 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or a combination thereof. The disk media 109 is, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), an FD (Floppy Disk), an MO (Magneto-Optical disk), a BD (Blu-ray® Disc), or a combination thereof.

In this embodiment, the program may be written to the memory 102 or storage 103 during the manufacturing stage of the information processing device 100, or it may be provided to the information processing device 100 via the network N, or it may be provided to the information processing device 100 via a non-temporary, computer-readable recording medium such as disk media 109.

<Functional Configuration>
Next, the functional configuration of the video generation device 1 will be described. Figure 3 is a diagram showing an example of the functional configuration of the video generation device 1. As shown in Figure 3, the video generation device 1 comprises a communication unit 11, a storage unit 12, and a control unit 13.

The communication unit 11 is implemented by the communication interface 104. The communication unit 11 transmits and receives information with the user terminal 2 via the network N. The communication unit 11 receives the subject video 121 from the user terminal 2 and transmits the injury/illness level video 127 to the user terminal 2.

The memory unit 12 is implemented by memory 102 and storage 103. The memory unit 12 stores the subject video 121, the subject skeletal video 122, the skeletal extraction model 123, the generation model 124, the identification model 125, the injury/illness level skeletal video 126, and the injury/illness level video 127.

Subject video 121 is a video of a subject with an injury or illness performing a first action. The first action may be, but is not limited to, walking, standing up, sitting down, or raising an arm.

The subject skeletal video 122 is a skeletal video extracted from subject video 121 by the skeletal extraction model 123. A skeletal video is a video consisting of frames containing skeletal information. Skeletal information is information indicating the coordinates of multiple pre-defined skeletons. Each frame of subject skeletal video 122 corresponds to each frame of subject video 121. That is, subject skeletal video 122 contains the chronological skeletal information of the subject extracted from subject video 121.

The skeletal extraction model 123 is a machine learning model that extracts human skeletal information (detects posture) from video. Any model, such as an open-pose model, can be used as the skeletal extraction model 123.

The generative model 124 is a machine learning model that generates videos containing people from skeletal videos. In other words, the generative model 124 performs the reverse operation of the skeletal extraction model 123. Any model can be used as the generative model 124. The generative model 124 is used as a generator for a GAN (Generative Adversarial Network).

The discrimination model 125 is a machine learning model that identifies whether an input video was filmed by a person or generated by the generation model 124. Any model can be used as the discrimination model 125. The discrimination model 125 is used as a classifier in a GAN.

The injury/illness level skeletal video 126 is a skeletal video extracted by the skeletal extraction model 123 from a video of a person with an injury/illness (injured person) performing a second movement. The second movement may be the same as or different from the first movement. The second movement may be, for example, walking, standing up, sitting down, or raising an arm, but is not limited to these. An injury/illness level skeletal video 126 is prepared for each injury/illness. Furthermore, an injury/illness level skeletal video 126 is prepared for each injury/illness level. The injury/illness level is an index indicating the severity of the injury/illness, prepared for each injury/illness, and can be set arbitrarily. For example, if the injury/illness level for a certain injury/illness is set to five levels from 1 to 5, then the injury/illness level 1 skeletal video 126 for injury/illness level 1 is extracted from a video of an injured person with injury/illness level 1 performing a second movement and stored in the memory unit 12. The same applies to injury/illness level skeletal videos 126 for injury/illness levels 2 to 5.

The injury/illness level video 127 is a video in which the subject performs a second action corresponding to their injury/illness level. The injury/illness level video 127 can be generated for any injury/illness level, regardless of the subject's first action in the subject video 121 or the subject's actual injury/illness level.

The control unit 13 is realized by the processor 101 reading and executing a program from the memory 102 and cooperating with other hardware components. The control unit 13 controls the overall operation of the video generation device 1. The control unit 13 comprises an acquisition unit 131, an extraction unit 132, a learning unit 133, a generation unit 134, and a display control unit 135.

The acquisition unit 131 acquires the target person's video 121 received by the communication unit 11 from the user terminal 2 and stores it in the storage unit 12.

The extraction unit 132 uses the skeletal extraction model 123 to extract the subject's skeletal video 122 from the subject's video 121 and saves it to the storage unit 12.

The learning unit 133 uses the subject video 121 and the subject skeleton video 122 as training data to train the generation model 124 to generate the subject video 121 from the subject skeleton video 122, and stores the trained parameters in the storage unit 12. As a result, the generation model 124 is trained to output a video of the subject corresponding to the skeleton video when a skeleton video is input. The learning method will be described in more detail later.

The generation unit 134 inputs the injury/illness level skeletal video 126 into the trained generation model 124, generates an injury/illness level video 127 corresponding to the injury/illness level skeletal video 126, and stores it in the storage unit 12.

The display control unit 135 transmits the subject video 121 and the injury/illness level video 127 to the user terminal 2 via the communication unit 11, causing the subject video 121 and the injury/illness level video 127 to be displayed on the user terminal 2's display.

Furthermore, the functional configurations of the video generation device 1 may be implemented by software as described above, or by hardware such as IC chips, SoCs (System on Chip), LSIs (Large Scale Integration), and microcomputers.

<Processing performed by the video generation device>
Next, the processes performed by the video generation device 1 according to this embodiment will be described. Figure 4 is a flowchart showing an example of the processes performed by the video generation device 1. Figure 5 is a diagram illustrating the processes performed by the video generation device 1 in a schematic manner.

(Step S101)
The acquisition unit 131 of the video generation device 1 acquires the target person video 121 from the user terminal 2 and stores it in the storage unit 12.

(Step S102)
The extraction unit 132 uses the skeletal extraction model 123 to extract the subject's skeletal video 122 from the subject's video 121 acquired in step S101 and stores it in the storage unit 12. Specifically, as shown in Figure 5, the extraction unit 132 inputs the subject's video 121 into the skeletal extraction model 123 and acquires the skeletal information for each frame output by the skeletal extraction model 123 as the subject's skeletal video 122.

(Step S103)
The learning unit 133 uses the subject video 121 acquired in step S101 and the subject skeleton video 122 extracted in step S102 as training data to train the generation model 124 so that when the subject skeleton video 122 is input, it outputs the subject video 121. Specifically, as shown in Figure 5, the learning unit 133 inputs the subject skeleton video 122 into the generation model 124 and acquires the video 121' output by the generation model 124. Next, the learning unit 133 inputs the subject skeleton video 122 and the subject video 121 or video 121' into the identification model 125, causing the identification model 125 to identify whether the input video is the subject video 121 or video 121'. The learning unit 133 trains the generation model 124 so that video 121' is identified as the subject video 121 by the identification model 125, and trains the identification model 125 so that video 121' is identified as video 121' by the identification model 125. In other words, the learning unit 133 trains the generation model 124 using a GAN with the generation model 124 as the generator and the identification model 125 as the classifier. As a result, the generation model 124 becomes a model that, when inputting the subject skeleton video 122, outputs a video 121' that approximates the subject video 121. Consequently, the generation model 124 is trained to output a video of the subject corresponding to the skeleton video when inputting the skeleton video. The learning unit 133 stores the parameters of the generation model 124 obtained through training in the storage unit 12 as parameters for the generation model 124 for the subject.

(Step S104)
The generation unit 134 receives a selection from the user terminal 2 for the injury/illness level for which to generate the injury/illness level video 127.

(Step S105)
The generation unit 134 generates a subject's injury/illness level video 127 using the subject's generation model 124, which was learned in step S103, and the injury/illness level skeletal video 126, which corresponds to the subject's injury/illness and the injury/illness level received in step S104. Specifically, as shown in Figure 5, the generation unit 134 inputs the injury/illness level skeletal video 126, which corresponds to the subject's injury/illness and the injury/illness level received in step S104, into the generation model 124, and acquires the video output by the generation model 124 as the subject's injury/illness level video 127.

For example, if the subject's injury is osteoarthritis of injury level 3, and injury level 1 is selected in step S104, the generation unit 134 inputs the injury level skeletal video 126 corresponding to injury level 1 osteoarthritis into the generation model 124. This generates an injury level video 127 showing the subject with injury level 1 osteoarthritis performing the second movement.

(Step S106)
The display control unit 135 transmits the subject video 121 acquired in step S101 and the subject injury/illness level video 127 generated in step S105 to the user terminal 2 via the communication unit 11, and displays them on the user terminal 2's display.

Figure 6 shows an example of the video display screen D displayed on the user terminal 2. The video display screen D displays the subject video 121 and the injury/illness level video 127 side by side. It also displays a dropdown list d for the user to select the injury/illness level for which the injury/illness level video 127 will be generated, and a generate button b for requesting the video generation device 1 to generate the injury/illness level video 127 for the selected level. When the user selects injury/illness level 1 from the dropdown list d and presses the generate button b, the video generation device 1 generates the injury/illness level video 127 for injury/illness level 1, and the generated injury/illness level video 127 is displayed on the video display screen D. Note that the video display screen D is not limited to the example in Figure 6.

<Summary>
As explained above, the video generation device 1 extracts the subject's skeletal video 122 from the subject's video 121, trains the generation model 124 using the subject's video 121 and the subject's skeletal video 122, and inputs the injury/illness level skeletal video 126 into the trained generation model 124 to generate the subject's injury/illness level video 127. In other words, the video generation device 1 can generate an injury/illness level video 127 with an injury/illness level different from the subject's actual injury/illness level based on the subject's video 121. Furthermore, the video generation device 1 can generate an injury/illness level video 127 with a second action different from the subject's first video based on the subject's video 121.

Let's consider a scenario where the user is a physician and the subject is a patient with osteoarthritis at disease level 3. Generally, even if a physician verbally explains to a patient that treatment will restore them to disease level 1, the patient may not be able to visualize what disease level 1 looks like. Therefore, the longer the treatment lasts, or the greater the burden on the patient, the more likely their motivation for treatment is to decline.

In contrast, according to this embodiment, the doctor can simply transmit the subject video 121 to the video generation device 1 and select injury/illness level 1 to display the injury/illness level video 127 of a patient at injury/illness level 1 on the user terminal 2. By showing the patient the injury/illness level video 127 of a patient at injury/illness level 1 while explaining the treatment, the patient can clearly visualize the state of recovery to injury/illness level 1 through treatment. This helps to suppress a decline in the patient's motivation for treatment.

<Note>
This embodiment includes the following disclosures.

(Note 1)
An information processing method performed by an information processing device,
The acquisition process involves obtaining a video of the subject, which is a video of the subject performing their first action, and
An extraction process to extract a video of the subject's skeleton from the subject's video,
A learning process to train a generation model to generate the subject video from the skeletal video,
A generation process that inputs a skeletal video corresponding to a second movement according to the level of injury or illness into the generation model and generates an injury/illness level video, which is a video of the subject performing the second movement according to the level of injury or illness;
Information processing methods including

(Note 2)
The aforementioned skeletal video is processed using the information processing method described in Appendix 1, which includes chronological skeletal information.

(Note 3)
The skeletal video corresponding to the second movement according to the injury level is generated based on a video of a person with the injury level performing the second movement, according to the information processing method described in Appendix 1 or Appendix 2.

(Note 4)
The second action is walking, as described in any of the information processing methods described in Appendix 1 to Appendix 3.

(Note 5)
An acquisition unit that acquires a subject video, which is a video of the subject performing the first action,
An extraction unit that extracts a video of the subject's skeleton from the subject's video,
A learning unit that trains a generation model to generate the subject video from the skeletal video,
A generation unit inputs a skeletal video corresponding to a second movement according to the level of injury or illness into the generation model and generates an injury/illness level video, which is a video of the subject performing the second movement according to the level of injury or illness.
An information processing device equipped with the following features.

(Note 6)
An information processing method performed by an information processing device,
The acquisition process involves obtaining a video of the subject, which is a video of the subject performing their first action, and
An extraction process to extract a video of the subject's skeleton from the subject's video,
A learning process to train a generation model to generate the subject video from the skeletal video,
A generation process that inputs a skeletal video corresponding to a second movement according to the level of injury or illness into the generation model and generates an injury/illness level video, which is a video of the subject performing the second movement according to the level of injury or illness;
A program that causes a computer to execute an information processing method that includes such methods.

The embodiments disclosed herein should be considered illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims, not in the sense described above, and is intended to include all modifications in the sense and scope equivalent to the claims. Furthermore, the present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

1: Evaluation device 2: User terminal 11: Communication unit 12: Storage unit 13: Control unit 101: Processor 102: Memory 103: Storage 104: Communication interface
105: Input/Output Interface
106: Drive device 107: Input device 108: Output device 109: Disk media 121: Subject video 122: Subject skeleton video 123: Skeleton extraction model 124: Generation model 125: Identification model 126: Injury/illness level skeleton video 127: Injury/illness level video 131: Acquisition unit 132: Extraction unit 133: Learning unit 134: Generation unit 135: Display control unit

Claims (6)

An information processing method performed by an information processing device,
The acquisition process involves obtaining a video of the subject, which is a video of the subject performing their first action, and
An extraction process to extract a video of the subject's skeleton from the subject's video,
A learning process that trains a generation model to generate the subject video from the skeletal video , and trains an identification model to identify whether the video generated by the generation model is the subject video or the video generated by the generation model when the skeletal video and the subject video or the video generated by the generation model are input, so that the video generated by the generation model is identified as the subject video .
A generation process that, corresponding to the injury or illness of the subject and for each of the multiple injury/illness levels set, inputs a pre-prepared skeletal video corresponding to the second action, which represents the recovered state of the injury/illness and corresponds to the arbitrarily set injury/illness level, into the generation model, and generates an injury/illness level video which is a video of the subject performing the second action according to the injury/illness level,
Display process for displaying the aforementioned injury/illness level video on the video display screen,
Information processing methods including The aforementioned skeletal video includes time-series skeletal information, as described in claim 1. The information processing method according to claim 1 or 2, wherein the skeletal video corresponding to the second movement according to the injury level is generated based on a video of a person with the injury level performing the second movement. The information processing method according to any one of claims 1 to 3, wherein the second action is walking. An acquisition unit that acquires a subject video, which is a video of the subject performing the first action,
An extraction unit that extracts a video of the subject's skeleton from the subject's video,
A learning unit trains a generation model to generate the subject video from the skeletal video , and a recognition model that, when inputting the skeletal video and the subject video or a video generated by the generation model, identifies whether it is the subject video or a video generated by the generation model, to train the generation model so that the video generated by the generation model is identified as the subject video .
A generation unit that, corresponding to the injury or illness of the subject and for each of the multiple injury/illness levels set, inputs a pre-prepared skeletal video corresponding to the second action, which represents the recovered state of the injury/illness and is arbitrarily set to the injury/illness level, into the generation model, and generates an injury/illness level video which is a video of the subject performing the second action according to the injury/illness level,
A display control unit that displays the aforementioned injury/illness level video on a video display screen,
An information processing device equipped with the following features. The acquisition process involves obtaining a video of the subject, which is a video of the subject performing their first action, and
An extraction process to extract a video of the subject's skeleton from the subject's video,
A learning process that trains a generation model to generate the subject video from the skeletal video , and trains an identification model to identify whether the video generated by the generation model is the subject video or the video generated by the generation model when the skeletal video and the subject video or the video generated by the generation model are input, so that the video generated by the generation model is identified as the subject video .
A generation process that, corresponding to the injury or illness of the subject and for each of the multiple injury/illness levels set, inputs a pre-prepared skeletal video corresponding to the second action, which represents the recovered state of the injury/illness and corresponds to the arbitrarily set injury/illness level, into the generation model, and generates an injury/illness level video which is a video of the subject performing the second action according to the injury/illness level,
Display process for displaying the aforementioned injury/illness level video on the video display screen,
A program that causes a computer to execute an information processing method that includes such methods.