JP7670062B2 - Information processing device, information processing system, program, and recording medium - Google Patents

Description

The present disclosure relates to an information processing device, and in particular to an information processing device that outputs ADL (activities of daily living) indicators for nursing care level certification of a subject of a sensor that acquires detection output.

Conventionally, information processing technology has also been used in nursing care settings. For example, JP 2005-237479 A (Patent Document 1) discloses a technology for detecting a patient's turning over in bed using biological signals such as respiratory signals. JP 2018-507008 A (Patent Document 2) discloses a technology for monitoring the ADL of elderly people, such as medication, using a PIR (passive infrared) sensor. JP 2019-204419 A (Patent Document 3) discloses a technology for deriving a user's level of care requirement based on the analysis results of information from a group of sensors that detect the user's behavior and the analysis results of electronic medical records.

JP 2005-237479 A Special table 2018-507008 publication JP 2019-204419 A

In recent years, ADL-related indices (hereinafter also referred to as "ADL indices") have been used to certify the level of care required. The level of care required should be certified based on the actual condition of the care recipient. However, when deriving the ADL indices, the care recipient may behave in a way that makes the condition of the care recipient appear worse than it actually is, with the intention of being certified as having a higher level of care required. Also, when deriving the ADL indices, the care recipient may behave in a way that makes the condition of the care recipient appear better than it actually is, with the intention of being certified as having a lower level of care required. Taking these cases into consideration, there is a need for a technology that can derive an ADL indices based on the actual condition of the care recipient, regardless of the intention of the care recipient.

The present disclosure has been devised in light of the current situation, and its purpose is to provide technology for deriving and outputting ADL indices for individuals who are to be certified as requiring nursing care or assistance based on the actual condition of the person receiving care.

According to one aspect of the present disclosure, an information processing device is provided that includes an interface for acquiring detection output from a sensor that represents the movements of a subject for nursing care level or assistance level certification, and an analysis unit that outputs information relating to the nursing care level or assistance level certification by analyzing at least a portion of the detection output that corresponds to the subject's unconscious movements.

According to another aspect of the present disclosure, an information processing system is provided that includes a sensor that acquires the motion state of a subject, the information processing device, and an output device that displays information relating to the level of care required or the level of assistance required output from the information processing device.

In accordance with another aspect of the present disclosure, a program is provided that, when executed by a processor of a computer, causes the computer to perform the steps of acquiring detection output from a sensor that represents the movements of a subject for nursing care level or assistance level certification, and outputting information relating to the nursing care level or assistance level certification by analyzing at least a portion of the detection output that corresponds to the subject's unconscious movements.

According to another aspect of the present disclosure, a computer-readable recording medium is provided that non-temporarily stores a program. The program, when executed by a processor of a computer, causes the computer to perform the steps of acquiring detection output from a sensor that represents the movement of a subject for certification of a level of nursing care or a level of support required, and outputting information related to the certification of a level of nursing care or a level of support required by analyzing at least a portion of the detection output that corresponds to the subject's unconscious movement.

According to the present disclosure, the information processing device outputs information related to the certification of the level of care required or the level of support required by analyzing a portion of the detection output from the sensor that corresponds to an unconscious movement of the subject of the certification of the level of care required or the level of support required. In this way, information related to the certification of the level of care required or the level of support required is derived and output based on the actual state of the subject of the certification of the level of care required or the level of support required.

FIG. 1 is a diagram illustrating an example of the configuration of a monitoring system. 13 is a diagram for explaining an example of the detection range of a sensor in a room 900. FIG. 1 is a diagram showing the hardware configuration of a sensor box 100, a management server 200, and a mobile terminal 300 of a monitoring system. FIG. 4 is a diagram illustrating a hardware configuration of a cloud server 400. FIG. 13 is a diagram showing an example of the installation mode of pressure sensors in each room 900. 13 is a flowchart of an example of processing in the control device 101 for deriving an index for the item "turning over movement." FIG. 11 is a diagram showing an example of skeletal information used to detect a turning motion. 10 is a flowchart of an example of a process executed by the control device 101 to derive an ADL index. FIG. 1 is a diagram showing the procedure for transferring from a bed to a wheelchair. FIG. 10 shows the trunk rotation angles identified in each of the four images of FIG. 11A and 11B are diagrams illustrating an example of a manner in which the rotation angle of the trunk changes. 10 is a flowchart of another example of a process executed by the control device 101 for deriving an ADL index. 13 is a diagram showing an example of changes in the detection values of a measurement sheet 560A and a measurement sheet 560B. FIG. 11 is a diagram showing an example of skeletal information used to detect a standing up motion; FIG. 11 is a diagram showing an example of skeletal information used to detect a walking motion; FIG. FIG. 1 illustrates an example of a machine learning model. FIG. 15 is a diagram for explaining machine learning by a learning device 1502.

Below, one embodiment of an information processing device will be described with reference to the drawings. In the following description, identical parts and components are given the same symbols. Their names and functions are also the same. Therefore, their description will not be repeated.

[Configuration of monitoring system]
Fig. 1 is a diagram showing an example of the configuration of a monitoring system. As an example of a monitoring target, the residents in each room provided in a room area RM of a facility are adopted. In the monitoring system of Fig. 1, rooms 900A and 900B are provided in the room area RM. The room 900A is assigned to a resident 800A. The room 900B is assigned to a resident 800B. In the example of Fig. 1, the number of rooms included in the monitoring system is two, but the number is not limited to this.

In the monitoring system, a sensor box 100A installed in room 900A, a sensor box 100B installed in room 900B, a management server 200 installed in a management center ST, and an access point AP are connected via a network NT.

In the monitoring system, the mobile terminal 300A carried by the staff member NA and the mobile terminal 300B carried by the staff member NB are connected to the network NT via the access point AP. Furthermore, the sensor boxes 100A and 100B, the management server 200, and the access point AP can communicate with the cloud server 400 via the network NT.

The living room 900A includes, as equipment, a bed 901A, a toilet 902A, and furniture 903A. A door sensor 510A is installed on the door of the living room 900A to detect whether the door is open or closed. A toilet sensor 520A is installed on the door of the toilet 902A to detect whether the toilet 902A is open or closed. An odor sensor 530A is installed on the bed 901A to obtain excretion information of the resident 800A. The resident 800A wears a vital sensor 540A that detects vital information of the resident 800A. An example of the detected vital information is the resident's body temperature. Another example is the resident's breathing. Yet another example is the resident's heart rate. Yet another example is information that includes two or more types of these types of information. In the living room 900A, the resident 800A can operate the care call handset 500A.

In this specification, when referring to matters common to multiple sensor boxes 100A and 100B, the term "sensor box 100" is used to refer to sensor boxes 100A and 100B. Similarly, the terms "resident 800," "room 900," "toilet 902," "furniture 903," "care call handset 500," "door sensor 510," "toilet sensor 520," "odor sensor 530," and "vital sensor 540" are used for resident 800A and 800B, room 900A and 900B, bed 901A and 901B, toilet 902A and 902B, furniture 903A and 903B, care call handset 500A and 500B, door sensor 510A and 510B, toilet sensor 520A and 520B, odor sensor 530A and 530B, and vital sensor 540A and 540B, respectively.

Furthermore, each room may be equipped with a microphone (microphone 550 in Figure 3) for outputting audio information from each room to an external device such as the management server 200.

The sensor box 100A has built-in sensors for detecting the behavior of objects in the living room 900A. One example of a sensor is a Doppler sensor for detecting the movement of objects. Another example is a camera. Further examples are a care call handset 500, a door sensor 510, a toilet sensor 520, an odor sensor 530, and a vital sign sensor 540. The sensor box 100A includes at least one of these sensors as a sensor.

Figure 2 is a diagram illustrating an example of the detection range of a sensor in a living room 900. In the example of Figure 2, the sensor box 100 is installed on the ceiling CL of the living room 900.

The range AR generally represents the detection range of the sensor. If the sensor is a Doppler sensor, the Doppler sensor detects behavior that occurs within the range AR. If the sensor is a camera, the camera captures images within the range AR.

Returning to FIG. 1, the management server 200 installed in the management center ST is connected to a display 206 and an input device 209. The input device 209 is, for example, a keyboard.

In the monitoring system of FIG. 1, the number of mobile terminals connected to the network NT via the access point AP is two (mobile terminals 300A and 300B), but the number is not limited to this. In this specification, when referring to matters common to mobile terminals 300A and 300B, the term "mobile terminal 300" is used to collectively refer to mobile terminals 300A and 300B.

Communication between each element in the monitoring system of Figure 1 may be wired or wireless.

The sensor box 100 includes a camera 105 and an interface 105A for inputting data from the camera 105 to the control device 101. The sensor box 100 also includes a Doppler sensor 106 and an interface 106A for inputting data from the Doppler sensor 106 to the control device 101. Each of the interfaces 105A and 106A is configured, for example, by a circuit for controlling the input and output of data.

It should be noted that the sensor box 100 does not necessarily have to include the camera 105 and/or the Doppler sensor 106. The camera 105 and/or the Doppler sensor 106 may be provided outside the sensor box 100, similar to the door gate sensor 510, etc. The control device 101 may acquire the detection output of the camera 105 and/or the Doppler sensor 106 provided outside the sensor box 100 via a given interface.

The vital sensor 540 does not necessarily have to be attached to the resident. In one example, the vital sensor 540 is realized by an infrared sensor or the like installed at a distance from the resident, and outputs vital information (body temperature) by detecting the temperature of the resident. In another example, the vital sensor 540 is realized by a Doppler sensor 106. The Doppler sensor 106 irradiates microwaves toward the resident. The microwaves cause a Doppler effect due to slight displacement of the resident's chest caused by the beating of the heart, causing the frequency to fluctuate. The Doppler sensor 106 detects the heartbeat of the resident based on the fluctuation in frequency. In yet another example, the Doppler sensor 106 detects the breathing of the resident.

[Configuration of each element of the monitoring system]
Fig. 3 is a diagram showing the hardware configuration of the sensor box 100, the management server 200, and the mobile terminal 300 of the monitoring system. Fig. 4 is a diagram showing the hardware configuration of the cloud server 400. Below, an example of the configuration of each device in the monitoring system will be described with reference to Figs. 3 and 4.

(Sensor box 100)
The sensor box 100 includes a control device 101 , a read only memory (ROM) 102 , a random access memory (RAM) 103 , a communication interface 104 , a camera 105 , a Doppler sensor 106 , a wireless communication device 107 , and a storage device 120 .

The control device 101 controls the sensor box 100. The control device 101 is, for example, configured with at least one integrated circuit. The integrated circuit is, for example, configured with at least one CPU (Central Processing Unit), at least one ASIC (Application Specific Integrated Circuit), at least one FPGA (Field Programmable Gate Array), or a combination thereof.

An antenna (not shown) and other elements are connected to the communication interface 104. The sensor box 100 exchanges data with external communication devices via the antenna. The external communication devices include, for example, a management server 200, a mobile terminal 300, an access point AP, a cloud server 400, and other communication terminals.

In one implementation, the camera 105 is a near-infrared camera. The near-infrared camera includes an IR (Infrared) projector that projects near-infrared light. By using the near-infrared camera, an image showing the interior of the room 900 can be captured even at night. In another implementation, the camera 105 is a surveillance camera that receives only visible light. In still another implementation, a 3D sensor or a thermography camera may be used as the camera 105. The sensor box 100 and the camera 105 may be configured as an integral unit or as separate units.

The Doppler sensor 106 is, for example, a microwave Doppler sensor that emits and receives radio waves to detect the behavior (movement) of objects in the room 900. This allows the detection of biometric information of the resident 800 in the room 900. In one example, the Doppler sensor 106 emits microwaves in the 24 GHz band toward the bed 901 in each room 900 and receives the reflected waves reflected by the resident 800. The reflected waves are Doppler shifted due to the movements of the resident 800. The Doppler sensor 106 can detect the breathing condition and heart rate of the resident 800 from the reflected waves.

The wireless communication device 107 receives signals from the care call handset 500, door sensor 510, toilet sensor 520, odor sensor 530, vital sensor 540, and microphone 550, and transmits the signals to the control device 101. For example, the care call handset 500 includes a care call button 501, and when the care call button 501 is operated, it transmits a signal indicating that operation has occurred to the wireless communication device 107. The door sensor 510, toilet sensor 520, odor sensor 530, and vital sensor 540 each transmit their respective detection outputs to the wireless communication device 107.

The storage device 120 is, for example, a storage medium such as a hard disk or an external storage device. The storage device 120 stores the programs executed by the control device 101 and various data used to execute the programs. The various data may include behavioral information of the resident 800.

At least one of the above programs and data may be stored in a storage device other than the storage device 120, as long as the storage device is accessible by the control device 101. The storage device other than the storage device 120 may be, for example, a storage area of the control device 101 (e.g., cache memory, etc.), the ROM 102, the RAM 103, and/or an external device (e.g., the management server 200, the mobile terminal 300, etc.).

In the monitoring system of this embodiment, a pressure sensor 560 is provided in each room 900. The pressure sensor 560 is connected to the sensor box 100, and the control device 101 can acquire the detection output of the pressure sensor 560. The installation manner of the pressure sensor 560 in the room 900 will be described later with reference to FIG. 5.

(Behavioral Information)
The above behavioral information will be described. The behavioral information is, for example, information indicating that the resident 800 has performed a predetermined behavior. In one example, the predetermined behavior includes four behaviors: "getting up" indicating that the resident 800 has woken up, "getting out of bed" indicating that the resident 800 has left the bed, "falling down" indicating that the resident 800 has fallen down from the bed, and "falling down" indicating that the resident 800 has collapsed.

In one embodiment, the control device 101 generates behavioral information of the resident 800 associated with each room 900. The control device 101 generates the behavioral information based on images captured by a camera 105 installed in each room 900. For example, the control device 101 detects the head of the resident 800 from the images, and detects the resident 800's "getting up," "getting out of bed," "falling," and "falling off" based on the change over time in the size of the detected head of the resident 800. A specific example of the generation of behavioral information is described in more detail below.

First, the area in the living room 900 where the bed 901 is located, the first threshold Th1, the second threshold Th2, and the third threshold Th3 are stored in the memory device 120. The first threshold Th1 distinguishes the size of the resident's head between when the resident is in a recumbent position and when the resident is in a sitting position within the area where the bed 901 is located. The second threshold Th2 distinguishes whether the resident is in a standing position or not based on the size of the resident's head within the living room 900 excluding the area where the bed 901 is located. The third threshold Th3 distinguishes whether the resident is in a recumbent position or not based on the size of the resident's head within the living room RM excluding the area where the bed 901 is located.

The control device 101 extracts a moving object region from the target image as the area of the resident 800. The extraction of the moving object region is performed, for example, according to a background subtraction method or a frame subtraction method. The control device 101 further extracts the head region of the resident 800 from the extracted moving object region by pattern matching using a head model prepared in advance. For example, a circular or elliptical Hough transform technique is used to extract the head region. A threshold derived by a neural network trained for head detection may be used for pattern matching. The control device 101 detects "getting up," "getting out of bed," "falling over," and "falling down" from the position and size of the extracted head.

The control device 101 may determine that the behavior "getting up" has occurred when it detects that the head position extracted as described above is located within the area where the bed 901 is located and that the head size extracted as described above has changed from the size of a lying position to the size of a sitting position. A first threshold value Th1 is used to detect the change in head size.

The control device 101 may determine that the action "getting out of bed" has occurred when the position of the head extracted as described above moves from within the area where the bed 901 is located to outside the area where the bed 901 is located and when it detects that the size of the head extracted as described above has changed from a certain size to the size of a standing position. A second threshold value Th2 is used to detect that the size of the head has changed to that in a standing position.

The control device 101 may determine that the behavior "fall" has occurred when the head position extracted as described above moves from within the area where the bed 901 is located to outside the area where the bed 901 is located and the size of the head extracted as described above changes from a certain size to the size of the recumbent position. A third threshold value Th3 is used to detect the change in head size to the size of the recumbent position.

The control device 101 may determine that the behavior "fall" has occurred when the position of the head extracted as described above is located within the room 900 excluding the area where the bed 901 is located and when it detects that the size of the extracted head has changed from a certain size to the size of a lying position. A third threshold value Th3 is used to detect the change in head size to the size of a lying position.

As described above, in one specific example, the control device 101 of the sensor box 100 generates behavioral information of the resident 800. Note that in the monitoring system, other elements may generate behavioral information of the resident 800 using images of the inside of the room 900.

(Management Server 200)
The management server 200 includes a control device 201 , a ROM 202 , a RAM 203 , a communication interface 204 , a display interface 205 , an operation interface 207 , and a storage device 220 .

The control device 201 controls the management server 200. The control device 201 is, for example, configured by at least one integrated circuit. The integrated circuit is, for example, configured by at least one CPU, at least one ASIC, at least one FPGA, or a combination thereof.

An antenna (not shown) and other elements are connected to the communication interface 204. The management server 200 exchanges data with external communication devices via the antenna. The external communication devices include, for example, the sensor box 100.

The display interface 205 is connected to the display 206 and sends an image signal to the display 206 to display an image in accordance with instructions from the control device 201 or the like.

The operation interface 207 is, for example, a USB (Universal Serial Bus) terminal, and is connected to the input device 209. The operation interface 207 receives a signal indicating a user operation from the input device 209. The input device 209 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of accepting a user's input operation.

The storage device 220 is, for example, a storage medium such as a hard disk or an external storage device. In one implementation, the storage device 220 stores a program executed by the control device 201, but the program may be stored in another storage device accessible to the control device 201.

(Mobile terminal 300)
The mobile terminal 300 includes a control device 301 , a RAM 303 , a communication interface 304 , a display 305 , an input device 306 , and an internal memory 320 .

The control device 301 controls the mobile terminal 300. The control device 301 is, for example, configured with at least one integrated circuit. The integrated circuit is, for example, configured with at least one CPU, at least one ASIC, at least one FPGA, or a combination thereof.

An antenna (not shown) and other elements are connected to the communication interface 304. The mobile terminal 300 exchanges data with external communication devices via the antenna and the access point AP (FIG. 1). The external communication devices include, for example, the sensor box 100 and the management server 200.

The display 305 is realized, for example, by an organic EL (Electro Luminescence) display. The input device 306 is realized, for example, by a touch sensor overlaid on the display 305. The touch sensor accepts various operations on the mobile terminal 300 by touch operations and outputs the contents of the operations to the control device 301.

The built-in memory 320 is a storage medium such as an eMMC (Embedded MultiMediaCard). In one implementation example, the built-in memory 320 stores a program executed by the control device 301, but the program may be stored in a storage device other than the built-in memory 320 as long as the storage device is accessible by the control device 301.

(Cloud server 400)
The cloud server 400 includes a control device 401 , a ROM 402 , a RAM 403 , a communication interface 404 , and a storage device 420 .

The control device 401 controls the cloud server 400. In one implementation, the control device 401 is configured by at least one integrated circuit. In one implementation, the integrated circuit is configured by at least one CPU, at least one ASIC, at least one FPGA, or a combination thereof.

Elements such as an antenna (not shown) are connected to the communication interface 404. The cloud server 400 exchanges data with external communication devices via the antenna. In one implementation example, the external communication devices include the sensor box 100 and the management server 200.

The storage device 420 is a storage medium such as a hard disk or an external storage device. In one embodiment, the storage device 420 stores a program executed by the control device 401, but the program may be stored in another storage device accessible to the control device 401.

(The meaning of the term "program" in this specification)
In the monitoring system, various "programs" may be provided not as standalone programs but as part of any program. Various "programs" may be realized by any multiple programs working together. In one example, when the control device 101 of the sensor box 100 executes a first program to realize the function described in this specification, the first program may use some modules of the second program. Even if the first program does not include some modules for realizing the function, if it mainly contributes to the realization of the function, it does not deviate from the purpose of the program for realizing the function. In addition, the functions described in this specification may be realized in part or in whole by dedicated hardware. Furthermore, the device providing the functions described in this specification may provide some or all of its functions from an external device as a so-called cloud service.

(Pressure sensor installation mode)
5 is a diagram showing an example of the installation mode of pressure sensors in each room 900. The pressure sensor 560 includes measurement sheets 560A, 560B, and 560C. In one example, each of the measurement sheets 560A, 560B, and 560C is a capacitor-type pressure sensor arranged in a grid pattern. The pressure sensor 560 can output the distribution and total amount of pressure on each of the measurement sheets 560A, 560B, and 560C as a detection output.

The measuring sheet 560A is placed on the bed 901. This allows the measuring sheet 560A to detect the pressure (and its distribution) applied to the bed 901.

The measuring sheet 560B is placed on the seat (seating position) of the wheelchair 905. This allows the measuring sheet 560B to detect the pressure (and its distribution) applied to the seat of the wheelchair 905. In the example of FIG. 5, the wheelchair 905 is positioned beside the bed 901. The measuring sheet 560B may include a communication interface for wireless communication. This allows the measuring sheet 560B to communicate with the sensor box 100 regardless of whether the wheelchair 905 is placed inside or outside the room 900.

The measurement sheet 560C is placed on the floor surface of the living room 900. This allows the measurement sheet 560C to detect the pressure (and its distribution) applied to the floor surface of the living room 900.

[Derivation of indicators for certification of nursing care level]
The control device 101 of the sensor box 100 acquires detection outputs from various sensors, derives an index for certifying the level of care required of the resident 800, and outputs the derived index. In this sense, the sensor outputs a detection output representing the movement of the resident 800 to the control device 101. Such sensors are, for example, a Doppler sensor 106, a camera 105, a care call handset 500, a door sensor 510, a toilet sensor 520, an odor sensor 530, a vital sensor 540, a microphone 550, and/or a pressure sensor 560. In this specification, "derive" is used as a general term for identifying, acquiring, calculating, and the like.

The control device 101 acquires data from the Doppler sensor 106 via the interface 106A. The control device 101 acquires data from the camera 105 via the interface 105A. The control device 101 acquires data from the care call handset 500, the door sensor 510, the toilet sensor 520, the odor sensor 530, the vital sensor 540, the microphone 550, and the pressure sensor 560 via the wireless communication device 107. In this sense, each of the interface 106A, the interface 105A, and the wireless communication device 107 is an example of an interface in the sensor box 100 from which the control device 101 acquires detection output.

The control device 101 derives information for the certification of the level of care required or the level of support required by utilizing the portion of the sensor detection output that corresponds to the unconscious movements of the resident 800 (hereinafter, the certification of the level of care required and the certification of the level of support required are collectively referred to as the certification of the level of care required). Unconscious movements include, for example, movements during sleep (more specifically, the movement of turning over in bed) and reflexive behaviors (such as movements to avoid danger).

Residents are unlikely to make any intentional movements while sleeping. For this reason, turning over in bed is considered an example of an unconscious movement.

The types of unconscious actions are not limited to those mentioned above.
An example of information related to certification of the level of care required (information related to the level of care required or the end of the form) is an ADL index.

In one example, the control device 101 identifies the score for the item "turning over" as the ADL index. More specifically, the control device 101 identifies the number of times the resident 800 turns over in sleep during a given period (e.g., while asleep, from 8 p.m. to 5 a.m. the next morning) from an image of the room 900, and identifies the score associated with the number of times as the score for the item "turning over". Note that a conversion table may be stored in the storage device 120 as an example of information that associates the number of times of turning over with a score. The control device 101 may obtain the score associated with the number of times by referring to the conversion table. In this embodiment, the control device 101 may obtain the score by applying the number of times of turning over to a predetermined formula instead of the conversion table.

The control device 101 may specify the time (reaction time) from the occurrence of a given event (e.g., a call) to a reaction to the given event (e.g., the generation of a voice in response, a turning back movement, etc.) as an index related to a reflexive behavior, convert the specified reaction time into a score, and specify an index from the score. As another example, the control device 101 may specify the time from the start to the end of a turning back movement (reflexive behavior time), convert the specified reflexive behavior time into a score, and specify an index from the score. The control device 101 may specify that the resident has reacted to a given event by specifying a behavior that matches a predetermined pattern from a detection output such as an image from the camera 105, or may specify the reaction of the resident by applying the detection output such as an image from the camera 105 to a trained machine learning model and using the output of the machine learning model. Note that a conversion table may be stored in the storage device 120 as an example of information that associates a reaction time with a score and/or information that associates a reflexive behavior time with a score. The control device 101 may obtain a score associated with the reaction time by referring to the conversion table.

[Derivation of an index for the item "turning over"]
The control device 101 can identify the number of times the turning motion occurs in a given period of time, and derive an index for the item "turning motion" based on the number of times.

Figure 6 is a flowchart of an example of processing in the control device 101 for deriving an index for the item "turning over motion". In one implementation example, the control device 101 performs the processing of Figure 6 by having the processor execute a given program. The given program may be stored in the storage device 120. In this sense, the storage device 120 is an example of a recording medium that non-temporarily stores a program that causes the control device 101 to perform the processing of Figure 6. The given program may be stored in a storage device accessible by the control device 101, other than the storage device 120. In this case, the storage device other than the storage device 120 is an example of a recording medium that non-temporarily stores a program that causes the control device 101 to perform the processing of Figure 6.

6, in step S600, the control device 101 acquires images of the inside of the room 900 for a given period of time from the camera 105 for an indicator of the item "turning over motion." Note that in step S600, the control device 101 may receive images from the camera 105 in real time, or may receive images from the camera 105 that have been stored in a given storage device from the storage device.

In step S602, the control device 101 identifies the number of times the person turns over in their sleep in the image acquired from the camera 105. The identification of the number of times the person turns over in their sleep will be described later with reference to FIG. 7.

In step S604, the control device 101 obtains an index corresponding to the number of turning over movements identified in step S602.

In one implementation example, the control device 101 identifies the number of times the motion of turning over occurs in a given period of time, and identifies the score associated with the number of times as the score for the item "motion of turning over." Note that the storage device 120 may store a conversion table as an example of information relating the number of times the motion of turning over occurs in a given period of time and a score. The control device 101 may obtain the score associated with the number of times by referring to the conversion table. The conversion table is stored in the storage device 120, for example.

In one implementation, the higher the number of tossing movements in a given period, the higher the score, but the relationship between number and identification is not limited to this.

In step S606, the control device 101 outputs the index acquired in step S604 and ends the processing in Fig. 6. The output destination may be an output device such as a display connected to the control device 101, or may be another information device (such as the management server 200) with which the control device 101 can communicate.

Figure 7 is a diagram showing an example of skeletal information used to detect the motion of turning over. Figure 7 shows skeletal information of a person identified by the control device 101 for a certain frame of an image from the camera 105. The outline OL1 represents the outline of the person, point P10 represents the position of the center of gravity of the person, and points P11 and P12 represent the positions of the person's shoulders.

When a person lying down turns to the right or left and is photographed from the front, the distance between one shoulder and the other shoulder in the captured image is expected to be shorter. Also, when a person who has been turned to the right or left changes their body orientation so that they are now facing forward, the distance between one shoulder and the other shoulder in the image captured from the front is expected to be longer.

The control device 101 may count the number of times the resident turns over in their sleep, using the image from the camera 105. In counting the number of times the resident turns over in their sleep, the control device 101 may update the count value by incrementing it by one when it detects a change in the distance between the shoulders of a person (resident) in the image from distance DF, which indicates that the person is facing forward, to distance DS, which indicates that the person is facing to the side (right or left).

That is, the control device 101 extracts from the image from the camera 105 the portion corresponding to the period during which the distance between the shoulders changes from distance DF to distance DS, and identifies the turning over motion using the images (frames) from that period.

As described above with reference to Figures 6 and 7, the control device 101 derives and outputs an indicator of the resident's turning over movement.

In the process of FIG. 6, the control device 101 may determine the period during which the resident is asleep based on the image from the camera 105 instead of a predetermined given period as the target period for derivation of the index. This allows the movement of turning over in one's sleep to be more reliably identified as an unconscious movement of the resident. The control device 101 may determine the period during which the resident is asleep based on, for example, the detection output from the Doppler sensor 106 and/or the detection output from the vital sensor 540. For example, the control device 101 may determine whether the resident is asleep based on whether the mode of body movement determined based on the detection output from the Doppler sensor 106 is a mode of a sleeping state. The control device 101 may determine whether the resident is asleep based on whether the heart rate determined based on the detection output from the vital sensor 540 is a heart rate in a sleeping state.

The control device 101 may increment and update the count value of the turning motion when the distance between the shoulders of the person in the image changes from distance DS to distance DF. In this case, the control device 101 extracts a portion from the image from the camera 105 that corresponds to the period during which the distance between the shoulders changes from distance DS to distance DF, and identifies one turning motion using the images (frames) from that period.

The control device 101 may identify the turning over motion using detection output from a measuring sheet 560A placed on the bed instead of or in addition to the image from the camera 105. In this case, the control device 101 acquires the detection output from the measuring sheet 560A instead of or in addition to the control of step S600. Then, in step S602, the control device 101 identifies the number of times the resident has turned over using the detection output from the measuring sheet 560A instead of or in addition to the image from the camera 105.

An example of determining the number of tossing movements using the detection output from the measuring sheet 560A will be described. The storage device 120 may store a pattern (hereinafter also referred to as a "front pattern") corresponding to a state in which the person on the bed is facing forward and a pattern (hereinafter also referred to as a "sideways pattern") corresponding to a state in which the person on the bed is facing sideways, for the detection output from the measuring sheet 560A. In one implementation example, the front pattern has a larger area on the measuring sheet 560A where pressure above a given value is detected than the sideways pattern.

The control device 101 may update the count value of one turning motion by incrementing it by 1 each time the detection output of the measurement sheet 560A changes from a frontal pattern to a sideways pattern. Alternatively, the control device 101 may update the count value of one turning motion by incrementing it by 1 each time the detection output of the measurement sheet 560A changes from a sideways pattern to a frontal pattern.

[Calculation of indicators related to natural, unconscious behavior]
The control device 101 may use the detection output from the sensor to derive an index related to a behavior (habitual behavior) that is naturally performed without being conscious of the user. The index related to the habitual behavior may be an index derived based on, for example, a behavior immediately after waking up from sleep (movement of getting up from bed).

The index for habitual behavior may also be the item "movement of transferring to a wheelchair" or the item "movement of standing up." "Motion of transferring to a wheelchair" is an action on the way to a goal such as sitting in a wheelchair. In this embodiment, "movement of transferring" may refer to the action of transferring from a given place such as a bed to a wheelchair, or conversely, the action of moving from a wheelchair to a given place such as a bed. "Motion of standing up" is an action on the way to walking. Subjects who do not need much care or support may be so conscious of the previous action that they do not pay attention to the action on the way. It is considered that by analyzing these actions, it is possible to estimate whether the action includes acting. The derivation of each type of index will be explained below.

[Processing flow (deriving an index for the item "transfer to wheelchair" based on images)]
8 is a flowchart of an example of a process executed by the control device 101 to derive an ADL index. In the process of FIG. 8, a score for the item "movement of transferring to a wheelchair" is derived as an example of an ADL index. In one implementation example, the control device 101 performs the process of FIG. 8 by causing a processor to execute a given program.

Referring to FIG. 8, in step S800, the control device 101 acquires from the camera 105 an image of the room 900 corresponding to a certain time. The timing and length of time from which images corresponding to the certain time are acquired can be set appropriately within the monitoring system. In one example, the control device 101 may acquire images for a period including a time period during which the resident 800 is assumed to have transferred to a wheelchair in the room 900, which is designated by an administrator of the monitoring system as images corresponding to the certain time. In another example, the control device 101 may acquire images for each day as images corresponding to the certain time. The control device 101 may also acquire images from the camera 105 sequentially.

In step S802, the control device 101 calculates the rotation angle of the person's trunk in each frame of the image acquired in step S800.

In calculating the rotation angle of a person's trunk, the control device 101 first acquires the person's skeletal information from the image. In one example, a publicly known service (for example, "User Local Pose Estimation AI" by User Local Co., Ltd. <https://humanpose-ai.userlocal.jp/>) is used to acquire the person's skeletal information from the image. The control device 101 then identifies a line from the center of the head to the center of the chest from the acquired skeletal information, and calculates the rotation angle of the trunk as the angle between that line and a predetermined reference line.

Calculation of the rotation angle of a person's trunk will be described in more detail with reference to Figures 9 and 10. Figure 9 is a diagram showing the flow of transferring from a bed to a wheelchair. Flow 700 shown in Figure 9 includes images 701, 702, 703, and 704 at four different times, and shows the flow of the movement of a resident transferring from a bed to a wheelchair. Image 702 shows the state at a time later than image 701, image 703 shows the state at a time later than image 702, and image 704 shows the state at a time later than image 703.

9, heads H1, H2, H3, and H4 represent the heads of people identified from skeletal information in images 701, 702, 703, and 704, respectively. Chests C1, C2, C3, and C4 represent the chests of people identified from skeletal information in images 701, 702, 703, and 704, respectively.

Figure 10 shows the trunk rotation angles identified in each of the four images in Figure 9.

Flow 800 in Figure 10 corresponds to flow 700 in Figure 9. In Figure 10, frames 801, 802, 803, and 804 correspond to images 701, 702, 703, and 704 in Figure 9, respectively. Frame 801 shows a line L1 from head H1 to chest C1, and a reference line LS. The angle between line L1 and reference line LS is shown as "85°".

In frames 802, 803, and 804, lines L2, L3, and L4 are shown extending from the head H2, H3, and H4 to the chest C2, C3, and C4, respectively, and the angles between the lines L2, L3, and L4 and the reference line LS are shown as "75°", "40°", and "5°", respectively.

The reference line LS is used commonly for the information within frames 801-804, i.e., for images 701-704. The reference line LS may be identified based on the contour of the bed in each frame (e.g., parallel to a straight line that constitutes the contour).

Returning to FIG. 8, in step S804, the control device 101 determines whether the rotation angle of the person's trunk has changed from angle A1 to angle A2 in an example frame constituting the image acquired in step S800. The control device 101 repeats the control of step S804 until a change in the rotation angle of the person's trunk from angle A1 to angle A2 is observed (NO in step S804), and when a change from angle A1 to angle A2 is observed (YES in step S804), the control proceeds to step S806.

Here, a specific example of a change in the rotation angle of the trunk will be described with reference to Figure 11. Figure 11 is a diagram showing an example of a change in the rotation angle of the trunk.

In the graph of Figure 11, the vertical axis represents the rotation angle of the trunk, and the horizontal axis represents time. Line LM represents an example of the change over time in the rotation angle of the trunk identified in each frame of the image from camera 105. The trunk rotation angle of line LM changes from angle A1 to angle A2 between time T1 and time T2. When such a change in the rotation angle of the person's trunk is detected, the control device 101 advances control from step S804 to step S806.

Returning to Fig. 8, in step S806, the control device 101 determines the time required for the person in the image to transfer from the bed to the wheelchair (transfer time). In the example of Fig. 11, the control device 101 determines the time required for the trunk rotation angle to change from angle A1 to angle A2 in the image from the camera 105 (the time from time T1 to time T2) as the time required for the transfer.

In step S808, the control device 101 acquires a score corresponding to the transfer time identified in step S806 as an index. The score means the score for the item "transfer to wheelchair". The control device 101 may acquire the score by referring to the conversion table that associates the length of time required for the transfer operation with the score, as described above. The conversion table is stored, for example, in the storage device 120.

In step S810, the control device 101 outputs the index acquired in step S808 and terminates the processing of Fig. 8. The output destination may be an output device such as a display connected to the control device 101, or may be another information device (such as the management server 200) with which the control device 101 can communicate.

When the output destination is another information device, the control device 101 outputs the index via the communication interface 104. In this sense, the communication interface 104 is an example of an output unit that outputs the index. In this embodiment, the output unit may be included in an analysis unit that derives the index. When the output destination is a display, the control device 101 outputs the index via a hardware element that serves as an interface between the display and the sensor box. In this sense, the hardware element is an example of an output unit that outputs the index.

In the process described above with reference to Figure 8, the control device 101 extracts from the image from the camera 105 the portion where the rotation angle of the person's trunk in each frame changes from angle A1 to angle A2 (for example, the portion corresponding to the period from time T1 to time T2 in Figure 11), identifies the time to which this portion corresponds (transfer time: for example, the time from time T1 to time T2 in Figure 11), and obtains a score corresponding to this transfer time, thereby deriving an index for the transfer operation to the wheelchair.

In one implementation, it is assumed that the shorter the transfer time, the higher the corresponding score, but the relationship between transfer time and score is not limited to this.

The control device 101 may impose additional conditions on the extraction of the above-mentioned part. For example, the control device 101 may extract the above-mentioned part and derive the index when a change from angle A1 to angle A2 is observed in the rotation angle of the trunk, and further, when the person is located on the bed in the frame corresponding to angle A1 and outside the bed in the frame corresponding to angle A2, the control device 101 may execute the extraction of the above-mentioned part and the derivation of the index. That is, for example, when a person is located on the bed in both the frame corresponding to angle A1 and the frame corresponding to angle A2, or when a person is located outside the bed in both the frame corresponding to angle A1 and the frame corresponding to angle A2, the control device 101 does not derive the index. This can avoid the derivation of the index when a resident is exercising on the bed in a room or moving in a wheelchair, or when the resident is performing an action other than a transfer action to a wheelchair.

The control device 101 may derive an index for the transfer operation to a wheelchair by identifying the speed of the transfer operation instead of the transfer time and obtaining a score corresponding to the speed. The speed is identified, for example, as the reciprocal of the time it takes for the rotation angle of the trunk of the person in each frame to change from a first angle to a second angle. In this case, the conversion table associates the speed with the score. In one implementation example, it is assumed that the higher the speed, the higher the corresponding score, but the relationship between the speed and the score is not limited to this. The conversion table is stored, for example, in the storage device 120.

The control device 101 may derive an index based on the transfer time (or transfer speed) of one transfer operation. The control device 101 may derive an index based on a statistical value (e.g., an average value) of the transfer time (or transfer speed) of multiple transfer operations.

In step S810, the control device 101 may further output statistical values of the measured values (indexes, transfer times, etc.) in each of the multiple transfer operations as auxiliary numerical values for the index. The output statistical values are, for example, the minimum, maximum, average, standard deviation, and/or variance of the measured values.

In one implementation example, in step S810, the control device 101 may output, together with the indicator, the standard deviation for the transfer time for multiple transfer operations. By checking the value of the standard deviation, the administrator of the monitoring system can infer whether the resident's behavior includes acting.

In another implementation, the control device 101 may output the transfer time for each of multiple transfer movements. The administrator of the monitoring system can infer that the transfer movements used to derive the index included the resident's acting due to the presence of extremely short and/or long transfer times.

[Process flow (deriving an index for the item "transfer to wheelchair" based on pressure)]
Fig. 12 is a flowchart of another example of the process executed by the control device 101 for deriving the ADL index. In the process of Fig. 12, a score for the item "movement of transferring to a wheelchair" is derived as an example of the ADL index. In one implementation example, the control device 101 performs the process of Fig. 8 by causing the processor to execute a given program.

12, in step S1000, the control device 101 acquires detection values from the measurement sheet 560A (placed on the bed) and the measurement sheet 560B (placed on the wheelchair seat) corresponding to a certain period of time. "Corresponding to a certain period of time" in step S1000 may be interpreted as having the same meaning as in step S800. The control device 101 may also acquire detection values sequentially from the measurement sheets 560A and 560B.

In step S1002, the control device 101 checks the change in the detection value of the measurement sheet 560A placed on the bed every specific time (for example, every second) and determines whether the detection value has dropped to value Pa at the specific time. In one implementation example, the control device 101 determines that the detection value has dropped to value Pa based on the detection value of the measurement sheet 560A dropping from a value higher than value Pa to value Pa. If the control device 101 determines that the detection value has dropped to value Pa (YES in step S1002), it proceeds to control step S1004; otherwise (NO in step S1002), it continues the control of step S1002 for the specific time that has not yet been checked.

In step S1004, the control device 101 determines whether the detection value of the measurement sheet 560B installed on the wheelchair seat has risen to value Pb after it has been determined in step S1002 that the detection value of the measurement sheet 560A has fallen to value Pa. In one implementation example, the control device 101 determines that the detection value has risen to value Pb based on the fact that the detection value of the measurement sheet 560B has risen to value Pb from a value lower than value Pb. When the control device 101 determines that the detection value of the measurement sheet 560B has risen to value Pb after the detection value of the measurement sheet 560A has fallen to value Pa (YES in step S1004), the control proceeds to step S1006. On the other hand, if the control device 101 cannot determine that the detection value of the measurement sheet 560B has risen to a value Pb within a predetermined time (e.g., within 30 seconds) after the detection value of the measurement sheet 560A has fallen to a value Pa (NO in step S1004), the control device 101 returns the control to step S1002. As a result, the detection value of the measurement sheet 560A that has not yet been subject to step S1002 becomes the subject of step S1002.

In step S1006, the control device 101 determines the time required for the transfer to the wheelchair (transfer time) that occurred in the room in which the measurement sheets 560A and 560B are installed from the detection values of the measurement sheets 560A and 560B. In one implementation example, the control device 101 determines the transfer time as the length of time from the time when the detection value of the measurement sheet 560A falls to value Pa to the time when the detection value of the measurement sheet 560B reaches value Pb.

An example of determining the transfer time will be described in more detail with reference to Figure 13. Figure 13 is a diagram showing an example of changes in the detection values of measurement sheet 560A and measurement sheet 560B.

13, line La represents the change over time in the detection value of measurement sheet 560A, and line Lb represents the change over time in the detection value of measurement sheet 560B. In the graph including each of lines La and Lb, the vertical axis represents the detected pressure value, and the horizontal axis represents time.

In the example of Figure 13, line La drops to value Pa at time Ta, and line Lb reaches value Pb at time Tb. In this case, the control device 101 identifies the time from time Ta to time Tb (time Tx) as the transfer time.

Returning to FIG. 12, in step S1008, the control device 101 acquires a score corresponding to the transfer time identified in step S1006 as an index for the item "transfer to wheelchair." Acquisition of the score corresponding to the transfer time can be achieved, for example, in a manner similar to that described for step S808.

In step S1010, the control device 101 outputs the indicators obtained in step S1008 and terminates the processing of FIG. 12.

In the process described above with reference to Figure 12, the control device 101 extracts the detection output from the detection outputs of the measurement sheets 560A, 560B, the detection output for the period from when the detection value of the measurement sheet 560A drops to value Pa to when the detection value of the measurement sheet 560B rises to value Pb, determines the transfer time based on the detection output, derives an index (score) based on the determined transfer time, and outputs the derived index.

In the processing of Figure 12, the detection output of the measurement sheet 560A may be limited to the detection output of the portion of the area covered by the measurement sheet 560A where the resident is expected to be located immediately before transferring to the wheelchair.

[Derivation of an index for the item "standing up"]
The control device 101 can identify the speed of the standing up movement and derive an index for the item "standing up movement" based on the speed.

Figure 14 is a diagram showing an example of skeletal information used to detect the standing up motion. Figure 14 shows skeletal information of a person identified by the control device 101 for a certain frame of an image from the camera 105. Outer contour OL1 represents the outer contour of the person, point P10 represents the position of the center of gravity of the person, and area H10 represents the area identified as the person's head.

As shown in FIG 2, the sensor box 100 including the camera 105 is installed on the ceiling CL of the room 900. Therefore, the camera 105 captures an image of the resident 800 in the room 900 from above. Therefore, as the standing-up action progresses, the distance between the head of the resident 800 and the camera 105 becomes shorter, and as a result, the area of the head becomes larger in the image captured by the camera 105.

The control device 101 may determine the time required for the area of the area identified as the person's head in the image captured by the camera 105 to change from a first area to a second area, and determine the speed of the person's standing up motion based on the reciprocal of the time. That is, the control device 101 may extract a portion from the image from the camera 105 where the area of the area identified as the person's head changes from a first area to a second area, and determine the speed of the person's standing up motion based on the extracted portion.

The storage device 120 may store a conversion table as an example of information relating the speed of the rising motion to a score. The control device 101 may obtain a score associated with the specified speed by referring to the conversion table, and output the obtained score as an index for the item "rising motion." The conversion table is stored in the storage device 120, for example.

In one implementation, the faster the speed of the rising motion, the higher the score, but the relationship between speed and score is not limited to this.

The control device 101 may output statistical values of the measured values (index, speed of the standing up movement, etc.) for each of the multiple standing up movements as auxiliary numerical values for the index. The output statistical values are, for example, the minimum, maximum, average, standard deviation, and/or variance of the measured values.

In one implementation example, in step S810, the control device 101 may output the standard deviation of the speed together with the indicator. By checking the value of the standard deviation, the administrator of the monitoring system can infer whether the resident's behavior includes acting.

In another implementation, the control device 101 may output the speed of each standing up motion for multiple standing up motions. The administrator of the monitoring system can infer that the standing up motion used to derive the index included the resident's acting due to the presence of an extremely slow and/or fast standing up motion.

[Derivation of indicators for the item "walking movement"]
"Walking" is an important item in evaluating ADL. Therefore, in addition to deriving an index related to "natural behavior without awareness," it is more preferable to derive an index for "walking." The control device 101 can identify the speed of walking and derive an index for the item "walking" based on the speed.

Figure 15 is a diagram showing an example of skeletal information used to detect walking movements. Figure 15 shows skeletal information of a person identified by the control device 101 for a certain frame of an image from the camera 105. The outline OL1 represents the outline of the person, and point P10 represents the position of the center of gravity of the person.

The control device 101 may identify the speed of movement of the center of gravity of a person in an image captured by the camera 105 as the walking speed of the person. The storage device 120 may store a conversion table as an example of information relating the walking speed and a score. The control device 101 may obtain the score associated with the identified speed by referring to the conversion table, and output the obtained score as an index for the item "walking". The conversion table is stored in the storage device 120, for example.

In addition, the control device 101 may determine the speed of walking movement using the detection output of the measurement sheet 560C.

That is, the control device 101 may identify the trajectory of the position where the measurement sheet 560C detected the pressure as the trajectory of the movement of the resident in the room 900, identify the distance moved by the resident from the trajectory of the movement of the resident, and identify the walking speed based on the identified movement distance and the time required for the movement. In this case, the control device 101 may extract, from the detection output of the measurement sheet 560C, a portion corresponding to a period during which the position where the measurement sheet 560C detected the pressure changed by more than a predetermined distance as a portion for identifying the walking speed, and then identify the walking speed.

In one implementation, the faster the walking speed, the higher the score, but the relationship between speed and score is not limited to this.

The control device 101 may output statistical values of the measured values (index, walking speed, etc.) in each of the multiple walking movements as auxiliary numerical values for the index. The output statistical values are, for example, the minimum, maximum, average, standard deviation, and/or variance of the measured values.

In one implementation example, in step S810, the control device 101 may output the standard deviation of the walking speed together with the indicator. By checking the value of the standard deviation, the administrator of the monitoring system can infer whether the resident's behavior includes acting.

In another implementation, the control device 101 may output the walking speed for each of multiple walking movements. The administrator of the monitoring system can infer that the walking movements used to derive the index included the resident's acting due to the presence of extremely slow and/or fast walking speeds.

[Identifying behavior types using machine learning models]
In deriving the index, the control device 101 may use a machine learning model to identify the type of movement of the resident from an image of the inside of the room 900. The processor of the control device 101 executes a program that realizes the algorithm of the machine learning model, and the control device 101 functions as a classifier.

Figure 16 is a diagram showing an example of a machine learning model. In the example of Figure 16, the machine learning model 1500 includes a learner 1501 based on the LightGBM (Gradient Boosting) framework and a learner 1502 based on the LSTM (Long Short Term Memory) framework. Note that the machine learning model shown in Figure 16 is merely an example, and any type of model may be used to identify the type of resident's movement.

The learning device 1501 is subjected to a learning process using a plurality of frame images labeled with the type of human movement. More specifically, in one example, the learning device 1501 is configured to output a prediction result of the type of human movement in an input frame image by performing a learning process using frame images labeled with one of the following nine types:

Type (1): lying in bed Type (2): from lying to sitting on edge of bed Type (3): sitting on edge of bed Type (4): from sitting on edge of bed to standing Type (5): transferring from sitting on edge of bed to wheelchair Type (6): moving in wheelchair Type (7): from standing to walking Type (8): walking Type (9): other than types (1) to (8) above "lying in bed" means that the frame image includes a person lying on the bed 901. "from lying to sitting on edge of bed" means that the frame image includes a person changing their position on the bed 901 from lying to sitting on edge of bed. "sitting on edge of bed" means that the frame image includes a person sitting on the edge of bed 901. "from sitting on edge of bed to standing" means that the frame image includes a person transferring from the bed 901 to a wheelchair. "moving in wheelchair" means that the frame image includes a person moving in a wheelchair. "Standing to walking" means that the frame image includes a person who is in the process of changing from a standing state to a walking state. "Walking" means that the frame image includes a person who is walking.

The learning device 1502 is subjected to a learning process using multiple sets of frame images labeled with the type of human movement. Figure 17 is a diagram for explaining the machine learning of the learning device 1502.

In FIG. 17, set 1600 represents an example of a set of frame images used for machine learning by learner 1502. Set 1600 includes seven consecutive frame images 1601 to 1607. An example of the time difference between consecutive frame images is 0.4 seconds. In the learning process of learner 1502, set 1600 is labeled with the type of action labeled in frame image 1604, which is the center in time, and input to learner 1502. That is, the label attached to frame image 1604 among frame images 1601 to 1607 is used as the correct answer label. The types of labels include, for example, types corresponding to each of the nine types (type (1) to type (9)) described above. Note that the number of frame images included in one set, the label attached to which frame image among the frame images included in one set is to be the label attached to the set, and the time difference between consecutive frame images are all merely examples.

The learning device 1502 is configured to perform the above-described learning process, and when consecutive frame images are input, to output a prediction result of the type of human movement in a specific frame image among the consecutive frame images.

The machine learning model 1500 outputs prediction results in two stages for each frame of an image from the camera 105 using the learning device 1501 and the learning device 1502. More specifically, the machine learning model 1500 outputs a prediction result of the type of movement of each frame using the learning device 1501. Then, the machine learning model 1500 inputs a set consisting of a plurality of consecutive frames to the learning device 1502 in the manner shown in FIG. 17. The set of frames is input to the learning device 1502 together with the prediction result of the learning device 1501 for the plurality of frames constituting the set. More specifically, the set of frames is input to the learning device 1502 together with the prediction result of a specific frame in the set (the frame in the middle in time in the set in the example of FIG. 17). The learning device 1502 outputs a prediction result of the type of movement of a person for a specific frame in the set by inputting the set of frames.

The machine learning model 1500 outputs the prediction result output from the learning device 1502 as the final prediction result.

In addition, the machine learning model 1500 may make corrections to the prediction results output from the learning device 1502 and output the corrected prediction results as the final prediction results. More specifically, when the learning device 1502 outputs the same prediction results consecutively for a given number of frames or more, the machine learning model 1500 outputs the prediction results output for each of the consecutive frames as the final prediction results as they are. On the other hand, when the prediction results output by the learning device 1502 for a certain frame are not consecutive for a given number of frames including the frame, the machine learning model 1500 changes the prediction result of the frame to another prediction result output by the learning device 1502 for the consecutive frames before or after the frame, and outputs the changed prediction result as the final prediction result.

For example, assume that the machine learning model 1500 is configured to output the prediction result output by the learner 1502 as the final prediction result as is, on the condition that the learner 1502 outputs the same prediction result for 10 or more consecutive frames. In such a case, when the learner 1502 outputs prediction results of type (8) for 12 consecutive frames, then outputs a prediction result of type (6) for one frame, and then outputs prediction results of type (8) for 10 or more consecutive frames, the machine learning model 1500 changes the prediction result to type (6) for the frame to which the prediction result of type (8) was assigned, and outputs the changed prediction result as the final prediction result.

The control device 101 may extract a portion of the image from the camera 105 that contains consecutive frames whose prediction result is type (5), and use this to derive an ADL index for the item “transferring to a wheelchair.”

The control device 101 may extract a portion of the image from the camera 105 that contains consecutive frames whose prediction result is type (4) and use this to derive an ADL index for the item “standing up movement.”

The control device 101 may extract a portion of the image from the camera 105 in which frames having a prediction result of type (8) are consecutive, and use this to derive the ADL index for the item "walking movement." The control device 101 may extract a portion of the image from the camera 105 in which frames having a prediction result of type (7) or type (8) are consecutive, and use this to derive the ADL index for the item "walking movement."

The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. Furthermore, the inventions described in the embodiments and each modified example are intended to be implemented, as far as possible, either alone or in combination.

100, 100A, 100B Sensor box, 200 Management server, 560 Pressure sensor, 560A, 560B, 560C Measurement sheet, 701, 702, 703, 704 Image, 800, 800A, 800B Resident, 801, 802, 803, 804 Frame, 900, 900A, 900B Room, 901, 901A, 901B Bed, 905 Wheelchair, 1500 Machine learning model, 1501, 1502 Learning machine, 1600 Set, 1601, 1604, 1607 Frame image.

Claims (20)

an interface for acquiring detection output from a sensor, the detection output representing the movement of a person who is to be certified as requiring nursing care or assistance;
an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject;
The subject's unconscious movement is a movement of turning over in bed,
The analysis unit is
Identifying the number of times the detection output changes from a first pattern related to a turning motion to a second pattern as the number of times the subject turns over in their sleep;
deriving an index of the subject's turning over movement, which is information regarding a level of care or a level of support required, based on the number of times the subject turns over in bed during a given period of time;
An information processing device that outputs the index . The interface acquires an image of the subject's room as the detection output,
The information processing device according to claim 1 , wherein the analysis unit extracts from the image a portion corresponding to the turning over motion, the image being of a period during which the distance between one shoulder and the other shoulder of a person on a bed in the room changes from a first distance to a second distance different from the first distance. an interface for acquiring detection output from a sensor, the detection output representing the movement of a person who is to be certified as requiring nursing care or assistance;
an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject;
The subject's unconscious movement is a transfer movement,
The analysis unit is
determining at least one of a transfer time and a transfer speed based on a time required for the subject to change from a first state to a second state in the detection output;
deriving an index regarding a transfer motion of the subject, which is information regarding a level of care need or a level of support need, based on at least one of the transfer time and the transfer speed;
An information processing device that outputs the index . The interface acquires an image of the subject's room as the detection output,
The information processing device according to claim 3, wherein the analysis unit extracts from the image an image of a period during which a rotation angle of the person's trunk in the image changes from a first angle to a second angle different from the first angle, as a portion corresponding to the transfer movement of the subject. In the portion corresponding to the transfer motion of the subject,
When the rotation angle of the trunk of the person is the first angle, the person is positioned on a bed;
The information processing device according to claim 4 , wherein when a rotation angle of the trunk of the person is the second angle, the person is located outside the bed. The information processing device according to claim 4 , wherein the analysis unit derives the index based on a time taken for a rotation angle of the person's trunk to change from the first angle to the second angle. The information processing device according to claim 3 , wherein the interface acquires, as the detection output, a detection output of a first pressure sensor that detects pressure at a seating position before the transfer related to the transfer operation, and a detection output of a second pressure sensor that detects pressure at the seating position. the analysis unit extracts, from the detection output of the first pressure sensor and the detection output of the second pressure sensor, a detection output for a period from a first timing to a second timing as a portion corresponding to the transfer motion;
At the first timing, the detection output of the first pressure sensor changes so as to decrease to a first threshold value,
The information processing device according to claim 7 , wherein at the second timing, the detection output of the second pressure sensor changes so as to rise to a second threshold value. an interface for acquiring detection output from a sensor, the detection output representing the movement of a person who is to be certified as requiring nursing care or assistance;
an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject;
The subject's involuntary movement is a movement of standing up,
The analysis unit is
A speed of the standing-up movement is identified by analyzing a portion of the detection output corresponding to the subject's standing-up movement, and an index for the standing-up movement is derived based on the speed of the standing-up movement, which is information related to a level of care required or a level of support required certification;
An information processing device that outputs the index . The interface acquires an image of the subject's room as the detection output,
10. The information processing device according to claim 9, wherein the analysis unit extracts from the image an image of a period during which an area of a region corresponding to a person's head changes from a first area to a second area different from the first area as a portion corresponding to the standing up motion. an interface for acquiring detection output from a sensor, the detection output representing the movement of a person who is to be certified as requiring nursing care or assistance;
an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject;
The subject's involuntary movement is a walking movement,
The interface acquires an image of the subject's room as the detection output,
The analysis unit is
extracting, from the images, an image of a period during which a person moved a certain distance or more in the room as a portion corresponding to the walking movement of the subject ;
deriving an index of walking movement, which is information related to a certification of a level of nursing care or a level of support required, based on the extracted image;
An information processing device that outputs the index . The information processing device according to claim 1 , wherein the analysis unit further derives and outputs a statistical value of the index when the index of the same type is derived multiple times. The analysis unit is
inputting data of a plurality of consecutive frames constituting the image into a trained classifier, and obtaining an estimation result of a movement in the room for each of the plurality of frames;
deriving the indicator by utilizing a frame from among the plurality of frames in which the estimation result is the unconscious movement;
The information processing device according to any one of claims 2, 4 to 6, 10, and 11, wherein the classifier is subjected to a learning process using, as training data, images consisting of a series of frames including frames labeled with a type of action . The information processing device according to claim 1 , wherein the interface acquires, as the detection output, a detection output of a pressure sensor installed in a room of the subject. A sensor for acquiring a motion state of a subject;
An information processing device according to any one of claims 1 to 14 ,
An output device that displays information relating to the level of care required or the level of support required output from the information processing device. When executed by a processor of a computer, the computer is caused to:
obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance;
and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject.
The subject's unconscious movement is a movement of turning over in bed,
The analyzing step includes:
Identifying the number of times the detection output changes from a first pattern related to a turning motion to a second pattern as the number of times the subject turns over in their sleep;
deriving an index for the subject's turning over movement, which is information relating to the level of care or assistance required, based on the number of times the subject turns over in bed in a given period of time . When executed by a processor of a computer, the computer is caused to:
obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance;
and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject.
The subject's unconscious movement is a transfer movement,
The analyzing step includes:
determining at least one of a transfer time and a transfer speed based on a time required for the subject to change from a first state to a second state in the detection output;
deriving information regarding the level of care required or the level of assistance required, which is an index regarding the transfer movement of the subject, based on at least one of the transfer time and the transfer speed. When executed by a processor of a computer, the computer is caused to:
obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance;
and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject.
The subject's involuntary movement is a movement of standing up,
The analyzing step includes:
determining a speed of the subject's standing up motion by analyzing a portion of the detection output corresponding to the subject's standing up motion;
and deriving an index for the standing up movement, which is information relating to a level of nursing care or level of assistance required certification, based on the speed of the standing up movement. When executed by a processor of a computer, the computer is caused to:
obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance;
and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject.
the detection output includes an image of the subject's room;
The subject's involuntary movement is a walking movement,
The analyzing step includes:
extracting, from the images, an image of a period during which a person moves a certain distance or more in the living room as a portion corresponding to the walking movement of the subject;
and deriving an index for walking movement, which is information relating to the certification of the level of nursing care or level of assistance required, based on the extracted image. A computer-readable recording medium non-temporarily storing the program according to any one of claims 16 to 19 .

Images