JP7801855B2 - Estimation device, learning device, estimation system, estimation method, and program - Google Patents

Description

This disclosure relates to an estimation device, a learning device, an estimation system, an estimation method, and a program.

Currently, there are known monitoring systems that estimate the condition of a person being monitored from images obtained by capturing images of the person's room. The monitored person may be a resident of a nursing home or an inpatient in a hospital. For example, Patent Document 1 describes a care system that processes images captured by an infrared camera to detect abnormal conditions in the person being cared for.

The care system described in Patent Document 1 detects abnormal conditions such as when the care recipient wanders around or when the care recipient remains motionless for an extended period of time, and outputs a detection signal to alert the user to the abnormality when an abnormal condition is detected. The care system described in Patent Document 1 processes images from an infrared camera to identify the movements of the care recipient and determines whether the identified movements correspond to an abnormal condition.

Japanese Patent Application Laid-Open No. 2003-260095

However, in the care system described in Patent Document 1, it is considered difficult to appropriately define actions that correspond to an abnormal state and to appropriately determine whether or not actions identified through image processing correspond to an abnormal state. For this reason, it is considered difficult for the care system described in Patent Document 1 to accurately determine whether or not the person being cared for is in an abnormal state. Therefore, in order to appropriately support the monitoring of the person being watched over, there is a need for technology that can accurately estimate the condition of the person being watched over.

This disclosure has been made in consideration of the above-mentioned problems, and aims to provide an estimation device, learning device, estimation system, estimation method, and program that accurately estimate the state of a person being watched over.

In order to achieve the above object, an estimation device according to the present disclosure includes:
an image acquisition means for acquiring a first captured image from a first imaging device that captures an image of a room of a first person being watched over among the plurality of people being watched over ;
a parameter acquiring means for acquiring a first parameter which is a parameter related to the first watching target;
an estimation means for estimating the state of the first person being watched over from the first captured image acquired by the image acquisition means and the first parameter acquired by the parameter acquisition means, using a trained model acquired by machine learning using captured images obtained by capturing images of the rooms of each of the multiple people being watched over and parameters related to each of the multiple people being watched over;
and an output means for outputting the estimation result by the estimation means.

In this disclosure, the state of the person being watched over is estimated from captured images using a trained model obtained through machine learning to estimate the state of the person being watched over from captured images. Therefore, according to this disclosure, the state of the person being watched over can be estimated with high accuracy.

Configuration diagram of a monitoring system according to the first embodiment 1 is a block diagram of a learning device according to a first embodiment; 1 is a block diagram of an estimation device according to a first embodiment; Functional configuration diagram of an estimation system according to the first embodiment FIG. 1 is an explanatory diagram of an occupant state model according to the first embodiment; FIG. 10 is a diagram showing a group of captured images that are estimated to be in a falling state. FIG. 10 is a diagram showing a group of captured images that are estimated to be in a wandering state. FIG. 10 is a diagram showing a group of captured images that are estimated to be in a fever state. FIG. 10 is a diagram showing a captured image that is estimated to be in a crowded state. 1 is a flowchart showing a learning process executed by a learning device according to a first embodiment; 1 is a flowchart showing an estimation process executed by an estimation device according to a first embodiment; Functional configuration diagram of an estimation system according to a second embodiment FIG. 10 is an explanatory diagram of an occupant state model according to the second embodiment. 10 is a flowchart showing a learning process executed by a learning device according to a second embodiment. 10 is a flowchart showing an estimation process executed by an estimation device according to a second embodiment. Functional configuration diagram of an estimation system according to a third embodiment 10 is a flowchart showing an estimation process executed by an estimation device according to a third embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that identical or equivalent parts in the drawings are designated by the same reference numerals.

(Embodiment 1)
FIG. 1 is a diagram showing the configuration of a watching system 1000 according to a first embodiment. The watching system 1000 is a system for watching over a person being watched over, and a system that supports an administrator in watching over the person being watched over. The person being watched over is a person who is the target of watching over and is watched over by the administrator. The person being watched over is a resident of a nursing home, an inpatient in a hospital, a person receiving care at home, etc. The administrator is a person who watches over the person being watched over and is concerned about the health and safety of the person being watched over. The administrator is a staff member of a nursing home, a hospital staff member, a staff member of a watching service provider, etc. The watching system 1000 includes an estimation system 100, an imaging device 200, and a display device 300.

The estimation system 100 estimates the state of the person being watched over based on captured images captured by the imaging device 200. The estimation system 100 includes a learning device 110 that executes a learning process, an estimation device 120 that executes an estimation process, and a storage device 130 that stores a trained model. The learning process is a process of generating a trained model to be used in the estimation process. The estimation process is a process of estimating the state of the person being watched over from the captured image using the trained model. The state of the person being watched over is broadly divided into two states: a normal state and an abnormal state. In this embodiment, abnormal states include a fallen state, a wandering state, a fever state, and a crowded state.

The fallen state is a state in which a person has fallen. The wandering state is a state in which a person is walking around inside or outside the house. The fever state is a state in which a person has a fever and the amount of heat generated is higher than normal. The crowded state is a state in which multiple people are gathered in a short distance. The normal state is a state other than the abnormal state. The estimation system 100 outputs the estimation results obtained by the estimation process to the display device 300.

During the learning stage, the learning device 110 performs machine learning using learning data including normal captured images and abnormal captured images, and generates a trained model for estimating the state of the person being watched over from the captured images. A normal captured image is an image obtained by capturing an image of the room of the person being watched over when the person being watched over is in a normal state. An abnormal captured image is an image obtained by capturing an image of the room of the person being watched over when the person being watched over is in an abnormal state. In this embodiment, the learning device 110 generates one trained model for one person being watched over. As shown in FIG. 2, the learning device 110 includes a control unit 11, a memory unit 12, a display unit 13, an operation reception unit 14, a first communication unit 15, and a second communication unit 16.

The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), RTC (Real Time Clock), etc. The CPU is also called a central processing unit, central arithmetic unit, processor, microprocessor, microcomputer, DSP (Digital Signal Processor), etc., and functions as a central processing unit that executes processes and calculations related to the control of the learning device 110. In the control unit 11, the CPU reads programs and data stored in the ROM and uses the RAM as a work area to perform overall control of the learning device 110. The RTC is, for example, an integrated circuit with a timekeeping function. The CPU can determine the current date and time from the time information read from the RTC.

The memory unit 12 is equipped with non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), or EEPROM (Electrically Erasable Programmable ROM), and serves as a so-called auxiliary storage device. The memory unit 12 stores programs and data used by the control unit 11 to execute various processes. The memory unit 12 also stores data generated or acquired by the control unit 11 as it executes various processes.

The display unit 13 displays various images under the control of the control unit 11. For example, the display unit 13 displays a screen for accepting various operations from the user. The display unit 13 includes a touch screen, a liquid crystal display, etc. The operation acceptance unit 14 accepts various operations from the user and supplies information indicating the content of the accepted operations to the control unit 11. The operation acceptance unit 14 includes a touch screen, a button, a lever, etc.

The first communication unit 15 communicates with the imaging device 200 under the control of the control unit 11. For example, the first communication unit 15 receives captured images from the imaging device 200 at a predetermined interval. The first communication unit 15 communicates with the imaging device 200 in accordance with a well-known wired communication standard or a well-known wireless communication standard. Well-known wired communication standards include USB (Universal Serial Bus, registered trademark) and Thunderbolt (registered trademark). Well-known wireless communication standards include Wi-Fi (registered trademark), Bluetooth (registered trademark), and Zigbee (registered trademark). The first communication unit 15 is equipped with a communication interface that complies with various communication standards.

The second communication unit 16 communicates with the storage device 130 under the control of the control unit 11. For example, the second communication unit 16 reads a trained model from the storage device 130 and writes an updated trained model to the storage device 130. The second communication unit 16 communicates with the storage device 130 in accordance with a well-known wired communication standard or a well-known wireless communication standard. The second communication unit 16 has a communication interface that complies with various communication standards.

In the estimation stage, the estimation device 120 uses the trained model to estimate the state of the person being watched over from captured images, which serve as estimation data. As shown in FIG. 3 , the estimation device 120 includes a control unit 21, a memory unit 22, a display unit 23, an operation reception unit 24, a first communication unit 25, a second communication unit 26, and a third communication unit 27.

The control unit 21 includes a CPU, ROM, RAM, RTC, etc. The CPU is also called a central processing unit, central processing unit, processor, microprocessor, microcomputer, DSP, etc., and functions as a central processing unit that executes processing and calculations related to the control of the estimation device 120. In the control unit 21, the CPU reads programs and data stored in ROM and uses RAM as a work area to perform overall control of the estimation device 120. The RTC is, for example, an integrated circuit with a timekeeping function. The CPU can determine the current date and time from the time information read from the RTC.

The memory unit 22 is equipped with non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), or EEPROM, and serves as a so-called auxiliary storage device. The memory unit 22 stores programs and data used by the control unit 21 to execute various processes. The memory unit 22 also stores data generated or acquired by the control unit 21 as it executes various processes.

The display unit 23 displays various images under the control of the control unit 21. For example, the display unit 23 displays a screen for accepting various operations from the user. The display unit 23 includes a touch screen, an LCD display, etc. The operation acceptance unit 24 accepts various operations from the user and supplies information indicating the content of the accepted operations to the control unit 21. The operation acceptance unit 24 includes a touch screen, buttons, levers, etc.

The first communication unit 25 communicates with the imaging device 200 under the control of the control unit 21. For example, the first communication unit 25 receives captured images from the imaging device 200 at a predetermined interval. The first communication unit 25 communicates with the imaging device 200 in accordance with a well-known wired communication standard or a well-known wireless communication standard. The first communication unit 25 has a communication interface that complies with various communication standards.

The second communication unit 26 communicates with the storage device 130 in accordance with the control of the control unit 21. For example, the second communication unit 26 reads a trained model from the storage device 130 in accordance with the control of the control unit 21. The second communication unit 26 communicates with the storage device 130 in accordance with a well-known wired communication standard or a well-known wireless communication standard. The second communication unit 26 has a communication interface that complies with various communication standards.

The third communication unit 27 communicates with the display device 300 in accordance with the control of the control unit 21. For example, the third communication unit 27 transmits information indicating the estimation result received from the control unit 21 to the display device 300 in accordance with the control of the control unit 21. The third communication unit 27 communicates with the display device 300 in accordance with a well-known wired communication standard or a well-known wireless communication standard. The third communication unit 27 has a communication interface that complies with various communication standards.

The storage device 130 stores the trained model generated by the learning device 110. The trained model stored in the storage device 130 is updated by the learning device 110. The trained model stored in the storage device 130 is also used by the estimation device 120. The storage device 130 stores captured images generated by the imaging device 200 as appropriate. The storage device 130 includes a hard disk drive (HHD), a solid state drive (SSD), etc.

The imaging device 200 captures images of the surroundings of the person being watched over and generates a captured image. The imaging device 200 is basically installed so that the resident 210, who is the person being watched over, fits within the imaging range when the resident 210 is in the room 250. The imaging device 200 is installed, for example, on the ceiling of the room 250 where the resident 210 lives. In this embodiment, the imaging device 200 is an infrared camera. Therefore, the captured image generated by the imaging device 200 is an infrared image that visualizes infrared rays emitted from an object in two dimensions. The imaging device 200 is equipped with an infrared image sensor in which imaging elements that detect infrared rays are arranged two-dimensionally. An infrared image is also called a thermal image.

The captured images generated by the imaging device 200 may be still images or moving images. In this embodiment, the captured images generated by the imaging device 200 are still images, and the imaging device 200 generates the captured images by capturing images of the living room 250 at predetermined intervals. The imaging device 200 transmits the generated captured images to the estimation system 100. The captured images are used as learning data used by the learning device 110 in the learning process. The captured images are also used as estimation data used by the estimation device 120 in the estimation process. The imaging device 200 is connected to the estimation system 100 using a well-known communication standard.

The display device 300 displays the estimation results from the estimation system 100. The display device 300 receives information indicating the estimation results from the estimation system 100, and displays an image indicating the estimation results indicated by the received information. Hereinafter, the information indicating the estimation results will be simply referred to as the estimation results, where appropriate. The estimation results are broadly divided into estimation results indicating that the state of the resident 210 is normal, and estimation results indicating that the state of the resident 210 is abnormal. The estimation result indicating that the state of the resident 210 is abnormal includes at least one of the estimation results indicating that the state of the resident 210 is in a fallen state, the estimation result indicating that the state of the resident 210 is in a wandering state, the estimation result indicating that the state of the resident 210 is in a fever state, and the estimation result indicating that the state of the resident 210 is in a crowded state.

The display device 300 is placed in a location where the manager 310 can view the screen of the display device 300. For example, the display device 300 is placed in the management room 350 where the manager 310 is present. The display device 300 may display captured images in addition to images showing the estimation results. When the manager 310 is notified by the display device 300 that the condition of the resident 210 is abnormal, he can quickly take action according to the abnormal condition. The display device 300 is connected to the estimation system 100 via a well-known communication standard. The display device 300 is a computer that includes, for example, a control unit (not shown), a memory unit (not shown), a display unit (not shown), an operation reception unit (not shown), and a communication unit (not shown).

Next, the functions of the estimation system 100 will be described with reference to Figure 4. The learning device 110 functionally comprises an image acquisition unit 111, a model generation unit 112, and a label acceptance unit 113. The estimation device 120 functionally comprises an image acquisition unit 121, an estimation unit 122, and an output unit 123. Each of these functions is realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in ROM or memory units 12, 22. The CPU then executes the programs stored in ROM or memory units 12, 22 to realize each of these functions.

The image acquisition unit 111 acquires normal captured images and abnormal captured images from the imaging device 200 or the storage device 130. The normal captured images and abnormal captured images acquired by the image acquisition unit 111 are used as learning data for machine learning. The captured image used in one learning session may be a single captured image or a series of captured images. A series of captured images is a plurality of captured images arranged in chronological order, for example, a plurality of captured images captured by the imaging device 200 every predetermined time (for example, several seconds to several tens of minutes). In this embodiment, the captured images used in one learning session are a series of captured images. The image acquisition unit 111 acquires a large series of normal captured images and a large series of abnormal captured images as learning data.

A series of normal captured images is a group of captured images obtained by capturing images of the watching subject's room at regular intervals when the watching subject is in a normal state. A series of abnormal captured images is a group of captured images obtained by capturing images of the watching subject's room at regular intervals when the watching subject is in an abnormal state. The abnormal captured images include a fall captured image group, a wandering captured image group, a fever captured image group, and a crowded captured image group. The fall captured image group is a group of captured images obtained when the watching subject is in a fallen state. The wandering captured image group is a group of captured images obtained when the watching subject is in a fallen state. The fever captured image group is a group of captured images obtained when the watching subject has a fever. The crowded captured image group is a group of captured images obtained when the watching subject is crowded together.

The control unit 11, functioning as the image acquisition unit 111, acquires a series of captured images from the imaging device 200 via the first communication unit 15. Alternatively, the control unit 11, functioning as the image acquisition unit 111, acquires a series of captured images from the storage device 130 via the second communication unit 16. This series of captured images is either a normal captured image group or an abnormal captured image group. The image acquisition unit 111 is an example of an image acquisition means and a first image acquisition means provided in the learning device.

The model generation unit 112 generates a trained model for estimating the state of the person being watched over from the captured image through machine learning using the normal captured image and the abnormal captured image acquired by the image acquisition unit 111. This trained model is a model for estimating, from the captured image, whether the person being watched over has fallen, whether they are wandering around, whether they have a fever, and whether they are crowded together. Various algorithms can be adopted as the learning algorithm used by the model generation unit 112.

In this embodiment, we will explain an example in which k-means, an unsupervised learning algorithm, is used as the learning algorithm. Unsupervised learning is a method of learning the inherent features of training data by providing the training data to a learning device without including labels, which are the results to be output. k-means is a non-hierarchical clustering algorithm that classifies a given number of clusters into k using the cluster mean. The processing flow of k-means is explained below.

First, a cluster is randomly assigned to each xi, which is each piece of data. Next, Vj, the center of each cluster, is calculated based on the assigned data. Next, the distance between each xi and each Vj is calculated, and xi is reassigned to the closest central cluster. Then, if the cluster assignment for all xi remains unchanged through the above process, or if the amount of change falls below a preset threshold, it is determined that convergence has occurred and the process ends.

In this embodiment, the model generation unit 112 learns the state of the resident through unsupervised learning using learning data that includes a large number of groups of normal captured images and groups of abnormal captured images. Here, each cluster is labeled by a human user. This labeling is used to assign to each cluster whether the state of the resident is normal or not, and, if the state of the resident is abnormal, what type of abnormal state the resident is in.

The model generation unit 112 classifies the group of captured images acquired by the image acquisition unit 111 into one of multiple groups through clustering. Note that the above-mentioned clusters correspond to groups. The model generation unit 112 performs labeling on each of the multiple groups in accordance with instructions from the user. The model generation unit 112 basically updates an existing trained model using a newly acquired group of captured images. In other words, the model generation unit 112 generates a new trained model from the newly acquired group of captured images and a trained model stored in the storage device 130. The model generation unit 112 saves the generated new trained model in the storage device 130. The model generation unit 112 is an example of model generation means.

The label receiving unit 113 receives labels from the user. As described above, in this embodiment, machine learning is unsupervised data. Therefore, the user performs labeling for each of the multiple classified groups. The label receiving unit 113 receives operations associated with labeling from the user. Through this labeling, each of the multiple groups is set as a group indicating a normal state or a group indicating an abnormal state. A group indicating an abnormal state is one of a group indicating a fallen state, a group indicating a wandering state, a group indicating a fever state, and a group indicating a crowded state. A trained model is completed through labeling.

The image acquisition unit 121 acquires captured images from the imaging device 200, which captures images of the room of the person being watched over. The captured images acquired by the image acquisition unit 121 from the imaging device 200 are estimation data used to estimate the state of the person being watched over. The estimation data may be a single captured image or a series of captured images. In this embodiment, the estimation data is a series of captured images. In other words, the control unit 21, functioning as the image acquisition unit 121, acquires the series of captured images from the imaging device 200 via the first communication unit 25. This series of captured images is either a normal captured image group or an abnormal captured image group. The image acquisition unit 121 is an example of an image acquisition means and a second image acquisition means provided in the estimation device. The captured images acquired by the image acquisition unit 121 are an example of second captured images.

The estimation unit 122 estimates the state of the person being watched over from the captured images acquired by the image acquisition unit 121, using a trained model acquired by machine learning to estimate the state of the person being watched over from captured images. This trained model is generated by the model generation unit 112 and stored in the storage device 130. The estimation unit 122 estimates whether the state of the person being watched over is normal or abnormal, based on this trained model and the acquired series of captured images. The estimation unit 122 supplies the estimation result to the output unit 123. The estimation unit 122 is an example of an estimation means.

The output unit 123 outputs the estimation result obtained by the estimation unit 122. The output destination of the output unit 123 can be adjusted as appropriate. In this embodiment, the output destination of the output unit 123 is the display device 300. In other words, the control unit 21 functioning as the output unit 123 transmits information indicating the estimation result to the display device 300 via the third communication unit 27. The output unit 123 is an example of an output means.

Next, with reference to Figure 5, the trained model generated by the model generation unit 112 will be described. In this embodiment, the trained model is a resident state model for estimating the state of a resident from a series of captured images. Therefore, the input of the resident state model is a series of captured images. The output of the resident state model is the state of the resident 210. The states of the resident 210 include a normal state, a fallen state, a wandering state, a fever state, a crowded state, etc.

Next, with reference to Figures 6, 7, 8, and 9, a group of captured images in which the state of the resident 210 is estimated to be abnormal in the resident state model will be described. Note that in the captured images, the hatched areas correspond to the heat source that emits heat, and are the areas corresponding to the resident 210. Also, in the captured images, the darkly hatched areas correspond to the heat source that emits strong heat, and are the areas where the body temperature of the resident 210 is high. Note that the position and movement of the resident 210 can be identified from the position and movement of the heat source.

Figure 6 shows a group of captured images that indicate that the resident 210 is in a fallen state. If the resident 210 falls in the living room 250, the resident 210 is likely to transition from a standing state to a lying state in a position different from the sleeping position. In particular, if the resident 210 falls due to fainting, there is a high possibility that the resident will lie motionless for a long period of time. Fainting refers to a sudden loss of consciousness and collapse due to cerebral anemia, cerebral hemorrhage, or the like. Therefore, a resident state model is generated so that a group of captured images that indicates the resident 210 transitioning from a standing state to a lying state in a position different from the sleeping position and lying down for a long period of time is estimated to be a group of captured images that indicate a fallen state. In other words, in unsupervised learning, the user assigns a label indicating a fallen state to a group to which a group of captured images that indicate such a fallen state belongs.

The group of captured images shown in Figure 6 is a group of captured images captured in the order of image 20AA, image 20AB, and image 20AC. Image 20AA is an image captured when resident 210 is standing in a position different from his usual sleeping position. Images 20AB and 20AC are images captured when resident 210 is lying down in a position different from his usual sleeping position. Image 20A is an image captured when resident 210 is lying down in his usual sleeping position. The usual sleeping position is, for example, a position where resident 210 is detected most of the time during the nighttime hours.

Figure 7 shows a group of captured images that indicate that the resident 210 is in a wandering state. When the resident 210 wanders, the resident 210 often leaves the room 250 for a long period of time after exhibiting pre-actions that suggest wandering. Pre-actions that suggest wandering include unusual nighttime behavior that occurs before wandering. Pre-actions that suggest wandering include, for example, moving around the room 250, entering and leaving the room 250, etc. Therefore, a resident state model is generated so that a group of captured images that show the resident 210 being absent from the room 250 for a long period of time after exhibiting pre-actions that suggest wandering is estimated to be a group of captured images that indicate a wandering state. In other words, in unsupervised learning, the user assigns a label indicating a wandering state to a group to which a group of captured images that indicate such a wandering state belongs.

The group of captured images shown in Figure 7 is a group of captured images captured in the order of image 20BA, image 20BB, image 20BC, image 20BD, and image 20BE. Images 20BA, 20BB, and 20BC are captured when resident 210 was engaging in preparatory behavior suggestive of wandering, and are captured when resident 210 was moving around in room 250. Images 20BD and 20BE are captured when resident 210 was away from room 250.

Figure 8 shows a group of captured images that indicate that the occupant 210 is in a fever state. When the occupant 210 has a fever, the captured images show that the amount of heat generated by the occupant 210 is higher than normal for a long period of time. Therefore, an occupant state model is generated so that a group of captured images that show that the amount of heat generated by the occupant 210 is higher than normal for a long period of time is estimated to be a group of captured images that represent a fever state. In other words, in unsupervised learning, the user assigns a label representing a fever state to a group to which a group of captured images that represent such a fever state belongs.

The group of captured images shown in Figure 8 is a group of captured images captured in the order of image 20CA, image 20CB, and image 20CC. Images 20CA, 20CB, and 20CC are captured when the occupant 210 continues to emit stronger heat radiation than normal. Image 20C is captured when the occupant 210 is emitting normal heat radiation. Normal heat radiation is, for example, the heat radiation of the occupant 210 that is detected most of the time.

Figure 9 shows a captured image in which the state of the occupant 210 is estimated to be a crowded state. When multiple people, including the occupant 210, are crowded together, multiple heat sources will be close to each other in the captured image. Therefore, an occupant state model is generated so that a captured image showing multiple heat sources being close to each other is estimated to be an image representing a crowded state. In other words, in unsupervised learning, the user assigns a label representing a crowded state to the group to which the captured image representing such a crowded state belongs. Note that a crowded state can be estimated from a single captured image. Therefore, when estimating a crowded state, a single captured image is sufficient as input to the occupant state model. Image 20D shown in Figure 9 is an image captured when multiple people, including the occupant 210, are crowded together in the living room 250.

Next, the learning process performed by the learning device 110 will be explained with reference to the flowchart in Figure 10.

The control unit 11 included in the learning device 110 acquires learning data (step S101). The learning data includes a large number of groups of normal captured images and a large number of groups of abnormal captured images. A series of captured images acquired by the imaging device 200 may be supplied directly to the learning device 110 as learning data, or may be stored in the storage device 130 as learning data and supplied to the learning device 110 as appropriate. In other words, the control unit 11 acquires the learning data from the imaging device 200 or the storage device 130.

When the control unit 11 completes the processing of step S101, it performs unsupervised learning (step S102). Specifically, the control unit 11 performs unsupervised learning using a large number of captured images included in the acquired learning data, and classifies the large number of captured images into multiple groups. When the control unit 11 completes the processing of step S102, it performs labeling (step S103). The control unit 11 assigns a label to each of the multiple groups in accordance with the operation received from the user by the operation receiving unit 14. This labeling completes the trained model.

When the control unit 11 completes the processing of step S103, it saves the trained model (step S104). The control unit 11 saves the trained model completed by labeling in the storage device 130. When the control unit 11 completes the processing of step S104, it completes the training process.

Next, the estimation process performed by the estimation device 120 will be described with reference to the flowchart in Figure 11.

The control unit 21 included in the estimation device 120 acquires a group of captured images from the imaging device 200 (step S201). After completing the processing of step S201, the control unit 21 inputs the acquired group of captured images into the trained model (step S202). The trained model estimates the state of the resident 210 from the input group of captured images. The state of the resident 210 is either a normal state or an abnormal state. The abnormal state is either a fallen state, a wandering state, a fever state, or a crowded state.

When the control unit 21 completes the processing of step S202, it obtains the estimation result from the trained model (step S203). When the control unit 21 completes the processing of step S203, it outputs the estimation result to the display device 300 (step S204). The display device 300 displays an image showing the estimation result supplied from the estimation device 120. When the control unit 21 completes the processing of step S204, it completes the estimation processing.

In this embodiment, a trained model is generated by machine learning using normal captured images and abnormal captured images, and the generated trained model is used to estimate the state of the person being watched over from the captured images. According to this embodiment, the state of the person being watched over is estimated from a trained model corresponding to the person being watched over, so the state of the person being watched over can be estimated with high accuracy.

In addition, in this embodiment, a trained model capable of estimating a fallen state, a wandering state, a fever state, and a crowded state is used to estimate whether the person being watched over is in a normal state, a fallen state, a wandering state, a fever state, or a crowded state. Therefore, according to this embodiment, it is possible to accurately estimate whether the person being watched over is in a normal state, a fallen state, a wandering state, a fever state, or a crowded state.

(Embodiment 2)
In the first embodiment, an example in which a trained model is prepared for each person being watched over is described. In the second embodiment, an example in which one trained model is prepared in common for multiple people being watched over is described. Note that the description of the same configurations and functions as those in the first embodiment will be omitted or simplified as appropriate.

Figure 12 shows the functional configuration of an estimation system 100 according to this embodiment. The learning device 110 according to this embodiment functionally comprises an image acquisition unit 111, a model generation unit 112, a label acceptance unit 113, and a parameter acquisition unit 114. The estimation device 120 according to this embodiment functionally comprises an image acquisition unit 121, an estimation unit 122, an output unit 123, and a parameter acquisition unit 124.

The image acquisition unit 111 acquires captured images obtained by capturing images of each of the rooms of the multiple people being watched over. The image acquisition unit 111 acquires a series of captured images as learning data from each of the multiple imaging devices 200 installed in the rooms of the multiple people being watched over. This series of captured images is either a normal captured image group or an abnormal captured image group.

The parameter acquisition unit 114 acquires parameters for each of the multiple monitored persons. When the image acquisition unit 111 acquires a series of captured images, the parameter acquisition unit 114 acquires parameters corresponding to the series of captured images as learning data. These parameters indicate the attributes of the monitored persons. Examples of these parameters include the monitored persons' age, gender, physical characteristics, and health status. Physical characteristics include height, weight, normal body temperature, and average heart rate. Health status includes whether the monitored persons require care or have any illnesses.

The parameter acquisition unit 114 acquires a first parameter, which is a parameter of the first person being watched over, when acquiring a series of captured images from an imaging device 200 that captures images of the room of the first person being watched over. For example, assume that parameter information in which the identification information of the person being watched over, the identification information of the imaging device 200, and parameters are associated is stored in the storage device 130. In this case, the parameter acquisition unit 114 can identify and acquire, from the parameter information stored in the storage device 130, parameters related to the person being watched over whose image is captured by the imaging device 200 that acquired the series of captured images.

The model generation unit 112 generates a trained model for estimating the state of the person being watched over from the captured images through machine learning using learning data including the captured images acquired by the image acquisition unit 111 and the parameters acquired by the parameter acquisition unit 114. For example, the model generation unit 112 classifies combinations of a series of captured images and parameters into one of multiple groups. The model generation unit 112 also labels each of the multiple groups in accordance with instructions from the user.

The model generation unit 112 stores the trained model completed by labeling in the storage device 130. This trained model is a model obtained by machine learning using captured images acquired for each of the multiple monitored individuals and parameters related to each of the multiple monitored individuals. This trained model is a common trained model for multiple monitored individuals.

The label receiving unit 113 receives labels from the user. The user performs labeling for each of the multiple classified groups. The user can perform labeling by taking into consideration a series of captured images and parameters. For example, consider a case where there is a first group containing a combination of a series of captured images showing a fever of 37 degrees and a parameter indicating a normal body temperature of 36 degrees, and a second group containing a combination of a series of captured images showing a fever of 37 degrees and a parameter indicating a normal body temperature of 36.5 degrees. In this case, the user may label the first group with a label indicating a fever state and the second group with a label indicating a normal state.

The image acquisition unit 121 acquires a first captured image from a first imaging device that captures an image of the room of a first person being watched over among multiple people being watched over. The first imaging device is one of the multiple imaging devices 200 that captures an image of the room of the first person being watched over. The parameter acquisition unit 124 acquires a first parameter, which is a parameter related to the first person being watched over among the multiple people being watched over. For example, the parameter acquisition unit 124 can identify and acquire parameters related to the first person being watched over that are captured by the first imaging device from parameter information stored in the storage device 130. The parameter acquisition unit 124 is an example of a parameter acquisition means.

The estimation unit 122 uses the trained model to estimate the state of the first watching target person from the first captured image acquired by the image acquisition unit 121 and the first parameters acquired by the parameter acquisition unit 124. The output unit 123 outputs the estimation result by the estimation unit 122.

Now, with reference to Figure 13, the trained model generated by the model generation unit 112 will be described. In this embodiment, the trained model is a resident state model for estimating the state of a resident from a series of captured images and parameters. Therefore, the input of the resident state model is the series of captured images and parameters. Furthermore, the output of the resident state model is the state of the resident 210. The states of the resident 210 include a normal state, a fallen state, a wandering state, a fever state, a crowded state, etc.

Next, the learning process performed by the learning device 110 will be explained with reference to the flowchart in Figure 14.

The control unit 11 included in the learning device 110 acquires learning data including a group of captured images and parameters (step S301). For example, the control unit 11 acquires a series of captured images corresponding to each resident 210 from multiple imaging devices 200. Alternatively, the control unit 11 may acquire a series of captured images corresponding to each resident 210 from a storage device 130 that stores a series of captured images acquired by multiple imaging devices 200. The control unit 11 also acquires parameters corresponding to each resident 210 from the storage device 130. In other words, the control unit 11 acquires learning data from multiple imaging devices 200 and the storage device 130.

When the control unit 11 completes the processing of step S301, it performs unsupervised learning (step S302). Specifically, the control unit 11 performs unsupervised learning using the many combinations included in the acquired learning data, and classifies the many combinations into multiple groups. These combinations are combinations of captured images and parameters. When the control unit 11 completes the processing of step S302, it performs labeling (step S303). The control unit 11 assigns a label to each of the multiple groups in accordance with the operation received from the user by the operation receiving unit 14. A trained model is generated through this labeling.

When the control unit 11 completes the processing of step S303, it saves the trained model (step S304). The control unit 11 saves the trained model completed by labeling in the storage device 130. When the control unit 11 completes the processing of step S304, it completes the training process.

Next, the estimation process performed by the estimation device 120 will be described with reference to the flowchart in Figure 15.

The control unit 21 included in the estimation device 120 acquires a first group of captured images from the first imaging device (step S401). Upon completing the processing of step S401, the control unit 21 acquires a first parameter (step S402). The control unit 21 acquires, from the storage device 130, a first parameter that is a parameter related to the first resident residing in the room imaged by the first imaging device. Upon completing the processing of step S402, the control unit 21 inputs the first group of captured images and the first parameter into the trained model (step S403). The trained model estimates the state of the resident 210 from the input first group of captured images and first parameter.

When the control unit 21 completes the processing of step S403, it obtains the estimation result from the trained model (step S404). When the control unit 21 completes the processing of step S404, it outputs the estimation result to the display device 300 (step S405). When the control unit 21 completes the processing of step S405, it completes the estimation processing.

In this embodiment, the state of the first watched-over person is estimated from a first captured image acquired from a first imaging device capturing an image of the room of the first watched-over person and a first parameter related to the first watched-over person, using a trained model acquired by machine learning using captured images obtained by capturing images of the rooms of each of the multiple watched-over people and parameters related to each of the multiple watched-over people. According to this embodiment, the state of the first watched-over person is appropriately estimated based on the first parameter from a trained model common to the multiple watched-over people, allowing for accurate estimation of the state of the watched-over person. In other words, according to this embodiment, even when a trained model common to the multiple watched-over people is used, the attributes of the watched-over people are taken into consideration, making it possible to make appropriate estimations based on the attributes.

(Embodiment 3)
In the first embodiment, an example was described in which the output of the trained model itself is treated as the estimation result. In the third embodiment, an example is described in which an estimation result is obtained based on the output of the trained model. Specifically, in the third embodiment, the trained model is used to acquire estimation material for estimating the state of the person being watched over, and the state of the person being watched over is estimated based on the acquired estimation material. Note that the description of the same configurations and functions as those in the first and second embodiments will be omitted or simplified as appropriate.

Figure 16 shows the functional configuration of an estimation system 100 according to this embodiment. The learning device 110 according to this embodiment functionally comprises an image acquisition unit 111, a model generation unit 112, and a label acceptance unit 113. The estimation device 120 according to this embodiment functionally comprises an image acquisition unit 121, an estimation unit 122 including a state determination unit 125 and a state estimation unit 126, and an output unit 123.

The state determination unit 125 uses the trained model to determine whether the state of the person being watched over is a suspected abnormal state from the captured image. A suspected abnormal state is a state in which an abnormality is suspected, and there is room for it to be considered an abnormal state. In this embodiment, the trained model is a model for determining whether the state of the person being watched over is a suspected abnormal state. If it is determined to be a suspected abnormal state, other conditions are taken into consideration to estimate whether it is an abnormal state. On the other hand, if it is determined not to be a suspected abnormal state, it is estimated to be a normal state. In other words, a suspected abnormal state is a necessary condition for estimating an abnormal state. The state determination unit 125 is an example of a state determination means.

The state estimation unit 126 estimates the state of the person being watched over based on the determination result by the state determination unit 125. For example, if the state determination unit 125 determines that the state of the person being watched over is a suspected abnormal state, and other conditions are met, the state estimation unit 126 estimates that the state of the person being watched over is an abnormal state. On the other hand, if the state determination unit 125 determines that the state of the person being watched over is not a suspected abnormal state, the state estimation unit 126 estimates that the state of the person being watched over is a normal state. The output unit 123 outputs the estimation result by the state estimation unit 126. The state estimation unit 126 is an example of a state estimation means.

The suspected abnormal state and other conditions can be adjusted as appropriate. For example, if the state determination unit 125 determines that the state of the person being watched over is in a suspected abnormal state at a rate equal to or greater than a threshold within a unit time, the state estimation unit 126 can estimate that the state of the person being watched over is in an abnormal state. In this case, the suspected abnormal state may be any state. Also, in this case, the other condition is that the state of the person being watched over is in a suspected abnormal state at a rate equal to or greater than a threshold within a unit time.

For example, consider a case where the person being monitored is deemed to be in a feverish state when a heat source of 37 degrees or higher is detected 80% or more of the time over a 10-minute period. In this case, a suspected abnormal state is a state in which the person being monitored has a body temperature of 37 degrees or higher. Another condition in this case is that the person being monitored is in a suspected abnormal state 80% or more of the time over a 10-minute period. In this case, the trained model only needs to be able to determine whether the person being monitored has a body temperature of 37 degrees or higher from captured images. In other words, in this case, the trained model does not need to be able to determine the percentage of time the person being monitored is in a suspected abnormal state.

As another example, if the area around the person being watched is considered to be crowded, a suspected abnormal state would be one in which multiple people, including the person being watched, are crowded together. This state includes a state in which the person being watched is receiving medical treatment, medical intervention, or nursing care services from a medical professional, or a state in which two people being watched pass each other while passing by. In this case, another condition is that the person being watched remains in a suspected abnormal state for a certain period of time (e.g., 15 minutes). In this case, the trained model only needs to be a model that can determine whether the person being watched is in a crowded state from captured images.

As another example, consider a case where the person being watched over is deemed to have fainted when the person is detected to have been lying still in a position other than the sleeping position for five minutes. In this case, the suspected abnormal state is a state of lying down in a position other than the sleeping position. Another condition in this case is that the person being watched over has been in a suspected abnormal state for five consecutive minutes. In this case, the trained model only needs to be able to determine from the captured image whether the person being watched over is lying down in a position other than the sleeping position. In other words, in this case, the trained model does not need to be able to determine the length of time the person being watched over has been in a suspected abnormal state.

Furthermore, the suspected abnormal state may include a first suspected abnormal state and a second suspected abnormal state, and the state estimation unit 126 may estimate that the state of the person being watched over is an abnormal state if, after a period in which the state determination unit 125 determines that the state of the person being watched over is the first suspected abnormal state, a period in which the state determination unit 125 determines that the state of the person being watched over is the second suspected abnormal state continues.

For example, consider a case in which the person being watched over is deemed to be wandering when prior behavior suggestive of wandering continues for 10 minutes or more, followed by the person being absent for 10 minutes or more. In this case, the suspected abnormal state includes a first suspected abnormal state, in which prior behavior suggestive of wandering is occurring, and a second suspected abnormal state, in which the person is absent from room 250. Furthermore, in this case, another condition is that the first suspected abnormal state continues for 10 minutes or more, followed by the second suspected abnormal state continuing for 10 minutes or more. In this case, the trained model only needs to be able to determine from the captured image whether prior behavior suggestive of wandering has occurred, and whether the person being watched over is absent from the captured image. In other words, in this case, the trained model does not need to be able to determine the duration of the first suspected abnormal state and the second suspected abnormal state, or the order in which the first suspected abnormal state and the second suspected abnormal state occur.

As another example, consider a case where the person being monitored is deemed to have collapsed due to poor health if they remain motionless for five minutes or more after a slight fever has continued for 20 minutes or more. In this case, the suspected abnormal state includes a first suspected abnormal state, which suggests a slight fever caused by poor health, and a second suspected abnormal state, which suggests that their health has worsened and they have become motionless in room 250, requiring rescue. Another condition in this case is that the first suspected abnormal state continues for 20 minutes or more, followed by the second suspected abnormal state for five minutes or more. In this case, the trained model only needs to be able to determine whether or not the captured image indicates a first suspected abnormal state suggesting a slight fever, and whether or not the captured image indicates a second suspected abnormal state in which the person being monitored has collapsed and is motionless. In other words, in this case, the trained model does not need to be able to determine the duration of the first suspected abnormal state and the second suspected abnormal state, the order in which the first suspected abnormal state and the second suspected abnormal state occur, etc.

Also, consider a case in which the monitored person is in an upright position as the first suspected abnormal state, and in which the monitored person is lying down in a position other than the sleeping position as the second suspected abnormal state. For example, if the first suspected abnormal state continues for 20 minutes or more, followed by the second suspected abnormal state for 2 minutes or more, the monitored person is deemed to have fainted. In this case, the suspected abnormal state includes the first suspected abnormal state suggesting an upright position, and the second suspected abnormal state suggesting a state of immobility in a position other than the normal sleeping position. Another condition in this case is that the first suspected abnormal state continues for 20 minutes or more, followed by the second suspected abnormal state for 2 minutes or more. In this case, the trained model only needs to be able to distinguish between the presence of the first suspected abnormal state suggesting an upright position from the captured image and the presence of the second suspected abnormal state in which the monitored person is lying down in a position other than the sleeping position and immobility from the captured image. In other words, in this case, the trained model does not need to be able to determine the duration of the first suspected abnormal state and the second suspected abnormal state, the order in which the first suspected abnormal state and the second suspected abnormal state occur, etc.

Furthermore, the state estimation unit 126 may estimate that the state of the person being watched over is an abnormal state if the state determination unit 125 determines that the state of the person being watched over is a suspected abnormal state during a first time period, and may estimate that the state of the person being watched over is a normal state if the state determination unit 125 determines that the state of the person being watched over is a suspected abnormal state during a second time period different from the first time period.

For example, consider a case where the person being watched over is deemed to be in a wandering state when they move around room 250 during the night and then disappear from room 250. In this case, the suspected abnormal state is a state in which they move around room 250 and then disappear from room 250. In this case, another condition is that it is during the night. In this case, the trained model only needs to be able to determine from the captured image that the person being watched over has moved around room 250 and then disappeared from room 250. In other words, in this case, the trained model does not need to be able to determine the time period in which the person being watched over is in a suspected abnormal state.

Also, if the person being watched over has a fever during a time other than mealtimes, the person being watched over may be considered to have a fever. In this case, the suspected abnormal state is when the person being watched over has a fever. Because it is difficult to determine whether the person being watched over has a fever during mealtimes because they are eating high-temperature food, the other condition is set to be a time period other than mealtimes. In this case, the trained model only needs to be able to determine whether the person being watched over has a fever from captured images. In other words, in this case, the trained model does not need to be able to determine the time periods when the person being watched over is in a suspected abnormal state.

As another example, if the person being watched over transitions from an upright position to a motionless lying state during a time other than sleeping hours, the person being watched over may be deemed to have fainted. In this case, a suspected abnormal state is when the person being watched over transitions from an upright position to a motionless lying state. In this case, another condition is that it is daytime. In this case, the trained model only needs to be able to determine from the captured image that the person being watched over transitions from an upright position to a motionless lying state without passing through a sitting position. In other words, in this case, the trained model does not need to be able to determine the time period during which the person being watched over is in a suspected abnormal state.

Next, the estimation process performed by the estimation device 120 will be described with reference to the flowchart in Figure 17.

The control unit 21 included in the estimation device 120 acquires a group of captured images from the imaging device 200 (step S501). After completing the processing of step S501, the control unit 21 inputs the group of captured images into the trained model (step S502). The trained model determines the state of the resident 210 from the input group of captured images.

When the control unit 21 completes the processing of step S502, it obtains a discrimination result from the trained model (step S503). This discrimination result is a discrimination result as to whether or not the state of the occupant 210 is a suspected abnormal state. When the control unit 21 completes the processing of step S503, it determines whether or not the state of the occupant 210 is a suspected abnormal state (step S504). If the control unit 21 determines that the state of the occupant 210 is not a suspected abnormal state (step S504: NO), it estimates that the state of the occupant 210 is a normal state (step S505). If the control unit 21 determines that the state of the occupant 210 is a suspected abnormal state (step S504: YES), it estimates the state of the occupant 210 based on other conditions (step S506).

For example, if the suspected abnormal state is a state in which the resident 210 has a body temperature above the threshold, the control unit 21 determines whether the resident 210 has a body temperature above the threshold at a rate above the threshold within a unit time. If the resident 210's current body temperature is above 37°C and the resident 210's body temperature has been above 37°C 80% or more of the time over the past 10 minutes, the control unit 21 estimates that the resident 210 is in a feverish state. If the resident 210's current body temperature is above 37°C and the resident 210's body temperature has not been above 37°C 80% or more of the time over the past 10 minutes, the control unit 21 estimates that the resident 210 is in a normal state.

When the control unit 21 completes the processing of step S505 or step S506, it outputs the estimation result to the display device 300 (step S507). When the control unit 21 completes the processing of step S507, it completes the estimation processing.

In this embodiment, a trained model is used to determine whether the state of the person being watched over is a suspected abnormal state from a captured image, and the state of the person being watched over is estimated based on the results of this determination. In this embodiment, for example, a trained model is used to determine whether the state of the person being watched over is a suspected abnormal state, which is easier to determine than whether the state of the person being watched over is an abnormal state. In this way, this embodiment can simplify the determination content using a trained model. As a result, this embodiment can estimate the state of the person being watched over with even greater accuracy.

According to this embodiment, for example, only the temporary state of the person being watched over is determined using the trained model, and the combination of multiple determination results, the order of multiple determination results, the duration of the same determination result, the time period, the attributes of the person being watched over, etc. can be determined without using the trained model. This configuration can improve the accuracy of determination using the trained model and flexibly adjust the method for estimating the state of the person being watched over without changing the trained model.

(Modification)
Although the embodiments have been described above, modifications and applications in various forms are possible. It is up to the discretion of the individual to adopt any of the configurations, functions, and operations described in the above embodiments. Furthermore, in addition to the above-described configurations, functions, and operations, additional configurations, functions, and operations may be adopted. Furthermore, the configurations, functions, and operations described in the above embodiments can be freely combined.

In the first embodiment, an example has been described in which the estimation system 100 includes three devices: the learning device 110, the estimation device 120, and the storage device 130. The configuration of the estimation system 100 is not limited to this configuration. For example, if one device has the functions of the learning device 110, the functions of the estimation device 120, and the functions of the storage device 130, the estimation system 100 may include this one device. Furthermore, the learning device 110, the estimation device 120, and the storage device 130 may be installed anywhere. For example, the learning device 110, the estimation device 120, and the storage device 130 may be installed within a single facility, or at least one of the learning device 110, the estimation device 120, and the storage device 130 may be installed on the cloud.

In embodiment 1, an example was described in which the display device 300 is provided separately from the learning device 110 and the estimation device 120. For example, if the learning device 110 or the estimation device 120 has the functions of the display device 300, the display device 300 does not need to be provided. In embodiment 1, an example was described in which the output unit 123 outputs the estimation result to the display device 300. The output unit 123 may output the estimation result to an audio output device. In this case, the audio output device notifies the administrator 310 of the estimation result by audio.

In the first embodiment, an example was described in which the abnormal state is one of four states: a fallen state, a wandering state, a fever state, and a crowded state. The type of abnormal state can be adjusted as appropriate. For example, the abnormal state may be at least one of the four states above, or may include at least one of the four states above plus other states, or may not include the four states above but may include other states.

The method for estimating the above four states can also be adjusted as appropriate. For example, in embodiment 1, an example was described in which the state of the people being watched over is estimated to be a crowded state when multiple heat sources are close together in the captured image. When multiple heat sources are close together in the captured image, and at least one heat source has a heat output equal to or greater than a threshold, the state of the resident 210 may be estimated to be a risky crowded state.

In the second embodiment, an example was described in which the parameters used for learning were attributes of the person being watched over. The parameters used for learning can be adjusted as appropriate. For example, the parameters used for learning may be the time of day, season, weather, etc.

In embodiment 1, one trained model is prepared for one person being watched over, and in embodiment 2, an example is described in which one trained model is prepared for multiple people being watched over. The correspondence between the person being watched over and the trained model can be adjusted as appropriate. For example, multiple people being watched over may be classified into multiple groups based on various perspectives, and one trained model may be prepared for each group. In this case, for example, multiple people being watched over may be classified into multiple groups based on age, gender, physical characteristics, health level, place of residence, etc. With this configuration, an appropriate trained model is generated according to the attributes of the person being watched over, so it is expected that the condition of the person being watched over can be estimated with high accuracy.

Note that a trained model generated using captured images of a certain person being watched over may be used to estimate the state of other people being watched over, or may be updated using captured images of other people being watched over. Similarly, a trained model generated using captured images of people being watched over who belong to a certain group may be used to estimate the state of people being watched over who belong to another group, or may be updated using captured images of people being watched over who belong to another group.

Furthermore, the learning algorithm employed by the learning device 110 is not limited to the example shown in embodiment 1. For example, deep learning that extracts the features themselves may be employed, or other known learning algorithms may be employed. Furthermore, when implementing unsupervised learning, clustering may be performed using a method other than non-hierarchical clustering using the k-means method. For example, hierarchical clustering using the shortest distance method may be employed, or clustering using other known methods may be employed. Furthermore, instead of unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, etc. may be employed as the learning algorithm.

In the above embodiments, the control units 11 and 21 function as the units shown in Figures 4, 12, and 16 by having the CPU execute programs stored in the ROM or memory units 12 and 22. However, in the present disclosure, the control units 11 and 21 may be dedicated hardware. Dedicated hardware includes, for example, a single circuit, a composite circuit, a programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these. When the control units 11 and 21 are dedicated hardware, the functions of each unit may be realized by individual hardware, or the functions of each unit may be realized together by a single piece of hardware. Furthermore, some of the functions of each unit may be realized by dedicated hardware, and other functions may be realized by software or firmware. In this way, the control unit 11 can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these.

By applying an operating program that defines the operation of the learning device 110 and estimation device 120 according to the present disclosure to a computer such as an existing personal computer or information terminal device, it is possible to cause the computer to function as the learning device 110 and estimation device 120 according to the present disclosure. Furthermore, such programs can be distributed in any manner, and may be stored on a computer-readable recording medium such as a CD-ROM (Compact Disk ROM), DVD (Digital Versatile Disk), MO (Magneto Optical Disk), or memory card, or distributed via a communications network such as the Internet.

This disclosure allows for various embodiments and modifications without departing from the broad spirit and scope of the disclosure. Furthermore, the above-described embodiments are intended to explain the disclosure and do not limit the scope of the disclosure. In other words, the scope of the disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are deemed to be within the scope of the disclosure.

This disclosure is applicable to monitoring systems.

11, 21 Control unit, 12, 22 Memory unit, 13, 23 Display unit, 14, 24 Operation reception unit, 15, 25 First communication unit, 16, 26 Second communication unit, 20A, 20AA, 20AB, 20AC, 20BA, 20BB, 20BC, 20BD, 20BE, 20C, 20CA, 20CB, 20CC, 20D Image, 27 Third communication unit, 100 Estimation system, 110 Learning device, 111, 121 Image acquisition unit, 112 Model generation unit, 113 Label reception unit, 114, 124 Parameter acquisition unit, 120 Estimation device, 122 Estimation unit, 123 Output unit, 125 State determination unit, 126 State estimation unit, 200 Imaging device, 210 Resident, 250 Living room, 300 Display device, 310 Administrator, 350 Centralized control room, 1000 Monitoring system

Claims (5)

an image acquisition means for acquiring a first captured image from a first imaging device that captures an image of a room of a first person being watched over among the plurality of people being watched over ;
a parameter acquiring means for acquiring a first parameter which is a parameter related to the first watching target;
an estimation means for estimating the state of the first person being watched over from the first captured image acquired by the image acquisition means and the first parameter acquired by the parameter acquisition means, using a trained model acquired by machine learning using captured images obtained by capturing images of the rooms of each of the multiple people being watched over and parameters related to each of the multiple people being watched over;
and an output means for outputting the estimation result by the estimation means.
Estimation device. an image acquisition means for acquiring a normal captured image obtained by capturing an image of each of the rooms of the multiple people being watched over when the state of each of the multiple people being watched over is normal, and an abnormal captured image obtained by capturing an image of each of the rooms of the multiple people being watched over when the state of each of the multiple people being watched over is abnormal;
a parameter acquisition means for acquiring parameters relating to each of the plurality of watching over targets;
and a model generation means for generating a trained model for estimating the state of the first person being watched over from a first captured image obtained by capturing an image of a room of a first person being watched over among the plurality of people being watched over and a first parameter which is a parameter related to the first person being watched over, by machine learning using the normal captured image and the abnormal captured image acquired by the image acquisition means and the parameters acquired by the parameter acquisition means.
Learning device. An estimation system including a learning device and an estimation device,
The learning device
a first image acquisition means for acquiring a normal captured image obtained by capturing an image of each of the rooms of the multiple people being watched over when the state of each of the multiple people being watched over is normal, and an abnormal captured image obtained by capturing an image of each of the rooms of the multiple people being watched over when the state of each of the multiple people being watched over is abnormal;
a first parameter acquisition means for acquiring parameters relating to each of the plurality of watching over targets;
a model generation means for generating a trained model for estimating the state of the first person being watched over from a first captured image obtained by capturing an image of a room of a first person being watched over among the plurality of people being watched over and a first parameter which is a parameter related to the first person being watched over, by machine learning using the normal captured image and the abnormal captured image acquired by the first image acquisition means and the parameters acquired by the first parameter acquisition means,
The estimation device includes:
a second image acquisition means for acquiring the first captured image from a first imaging device that captures an image of the room of the first person being watched over;
a second parameter acquisition means for acquiring the first parameter;
an estimation means for estimating a state of the first watching target person from the first captured image acquired by the second image acquisition means and the first parameter acquired by the second parameter acquisition means , using the trained model generated by the model generation means;
and an output means for outputting the estimation result by the estimation means.
Estimation system. the estimation device acquires a first captured image from a first imaging device that captures an image of a room of a first watching target person among the plurality of watching targets;
the estimation device acquires a first parameter which is a parameter related to the first watching target;
the estimation device estimates the state of the first watching target from the first captured image and the first parameter using a trained model acquired by machine learning using captured images obtained by capturing images of the rooms of the plurality of watching targets and parameters related to each of the plurality of watching targets ;
Estimation method. Computer,
an image acquisition means for acquiring a first captured image from a first imaging device that captures an image of a room of a first person being watched over among the plurality of people being watched over ;
a parameter acquisition means for acquiring a first parameter which is a parameter related to the first watching target;
an estimation means for estimating a state of the first person being watched over from the first captured image acquired by the image acquisition means and the first parameter acquired by the parameter acquisition means, using a trained model acquired by machine learning using captured images obtained by capturing images of the rooms of each of the plurality of people being watched over and parameters related to each of the plurality of people being watched over;
functioning as an output means for outputting the estimation result by the estimation means;
program.