JP7659484B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

At care facilities, homes, and other care sites, caregivers monitor those receiving care (those requiring care) to understand their condition and provide assistance. This monitoring task places a heavy burden on caregivers, and support is needed to reduce this burden. One way to provide such support would be to automate the monitoring process as much as possible, but in order to do so, it is necessary to at least identify the person receiving care.

Regarding toilets, Patent Document 1 describes a technology that collects excretion information indicating the contents of excrement excreted into the toilet bowl without the need to ask the toilet user, and prevents foreign objects other than excrement from being flushed even if they are found in the toilet bowl. In the technology described in Patent Document 1, a server receives facial image data captured by a camera, performs facial recognition processing by comparing the features of the data with pre-stored authentication data, and obtains identification data associated with the matching authentication data, thereby identifying the toilet user.

International Publication No. 2021/024584

As mentioned above, caregivers monitor and assist the condition of those in need of care. By introducing a system in which a server performs face recognition processing based on image data to identify toilet users, as in the technology described in Patent Document 1, the burden on caregivers can be reduced. However, in this case, the face image data captured by the camera will be stored on the server. Because information about toilet use is sensitive information, leaving the face image data on the server is not appropriate as it places a heavy mental burden on users and caregivers. Therefore, it is desirable to use a small device (toilet sensor) that is installed in the toilet and has face recognition, notification, and excrement analysis functions, and to prevent face image data from leaking onto a network such as an external server.

When such a configuration is adopted, the person who enters the restroom is identified by a device with a facial recognition function installed in the restroom, and the caregiver is notified of the user's entry. In this case, the facial image data used for facial recognition is not stored externally, which can reduce the mental burden on the user and caregiver. However, facial recognition using a toilet sensor has the following major problems (1) and (2).

(1) Processing power issues arise due to the need to perform multiple facial recognition tests simultaneously.
(2) The problem of increased rates of misidentification due to caregivers wearing masks.

Regarding (1) above, toilet space arises when there is a user who is unable to enter or leave the toilet without the assistance of a caregiver. When using the toilet, if one or more caregivers enter the toilet to accompany the user, and both the user and the caregiver enter the imaging range of the face recognition camera, multiple face recognitions must be performed simultaneously. Therefore, in such cases, the accuracy of identifying the user decreases and the processing time increases.

Regarding (2) above, in addition to the problem of (1) above, in nursing facilities, users do not wear masks, and although caregivers usually do not wear masks either, they may wear masks while working during epidemic periods to prevent infection. In this case, when assisting with excretion, the user and the caregiver wearing a mask will be within the imaging range of the face recognition camera. Recognizing a face wearing a mask cannot use the information of the part of the face covered by the mask, and matching is performed from limited feature information, which makes it prone to misrecognition, and is a factor that reduces the recognition accuracy. As a result, a problem may occur in which a face that actually belongs to a caregiver is mistakenly recognized as that of the user.

In addition, because identifying a person through facial recognition needs to be linked with a function to notify caregivers, such as for toilet assistance or rushing to prevent accidents, not only accuracy but also real-time performance is required, so it is desirable to improve the processing power and accuracy of facial recognition processing. On the other hand, if such processing is to be performed without transmitting facial image data over a network, it is desirable for the toilet sensor to be implemented as an edge device incorporating a space-saving, power-saving CPU (Central Processing Unit). Therefore, it is desirable to improve the processing power and accuracy of facial recognition processing by methods other than improving the performance of CPUs.

As described above, while improvements have been made to install toilet sensors to reduce the burden of excretion management in caregiving, there have been issues with the accuracy and processing power when identifying users who require caregiver assistance through facial recognition. Furthermore, such issues can also arise when managing entry to various rooms, not just toilets, and when entering a room with an accompanying person.

The objective of the present disclosure is to provide an information processing device, an information processing method, and a program that solve the above-mentioned problems. The above-mentioned problem is to make it possible to perform face authentication of a user in real time with high accuracy and without reducing processing power, and to output the authentication result, when a person accompanying a user of a room is wearing a mask.

The information processing device according to the first aspect of the present disclosure includes a sensor that detects the entry of a person into a room, and an imaging device that captures an image of the person who has entered the room to obtain image data when the sensor detects the entry of a person. The information processing device includes a determination unit that determines whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, thereby provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or not. The information processing device includes a face authentication unit that performs face authentication processing based on face image data of the person to be authenticated in the image data when the person is a person to be authenticated as a result of the provisional determination by the determination unit, and an output unit that outputs the result of the face authentication processing to a notification destination. The face authentication unit performs the face authentication processing by comparing the face image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated.

In an information processing method according to a second aspect of the present disclosure, a sensor detects the entry of a person into a room, and when the sensor detects the entry of a person, an imaging device captures an image of the person who has entered the room to obtain image data. The information processing method determines whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determines whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated. If the result of the provisional determination is that the person is a person to be authenticated, the information processing method executes a facial recognition process based on facial image data of the person to be authenticated in the image data, and outputs the result of the facial recognition process to a notification destination. The facial recognition process is executed by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated.

The program according to the third aspect of the present disclosure is a program for causing a computer to execute information processing. The information processing inputs a result of detection of a person entering a room by a sensor, and when the sensor detects the entry of a person, causes an imaging device to capture an image of the person who entered the room to obtain image data. The information processing determines whether the person who entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determines whether the person who entered the room is a person to be authenticated as a user of the room or is the person to be authenticated. If the result of the provisional determination is that the person is a person to be authenticated, the information processing executes a face recognition process based on face image data of the person to be authenticated in the image data, and outputs the result of the face recognition process to a notification destination. The face recognition process is executed by comparing the face image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated.

The present disclosure provides an information processing device, information processing method, and program that can perform face authentication of a user in real time and with high accuracy without reducing processing power when a person accompanying a user of a room is wearing a mask, and output the authentication result.

1 is a block diagram showing a configuration example of an information processing device according to a first embodiment; FIG. 11 is a diagram illustrating a configuration example of an information processing system including an information processing device according to a second embodiment. 3 is a side view showing an example of installation of an information processing device in the information processing system of FIG. 2. 3 is a block diagram showing an example of a configuration of an information processing device in the information processing system of FIG. 2. 3 is a schematic diagram for explaining an example of a process flow for generating and registering face model data in the information processing system of FIG. 2. 5 is a schematic diagram for explaining an example of face authentication processing in the information processing device of FIG. 4 when a caregiver is not wearing a mask. FIG. 5 is a schematic diagram for explaining an example of face authentication processing in the information processing device of FIG. 4 when a caregiver is wearing a mask. FIG. 3 is a diagram for explaining an example of a notification in a terminal device in the information processing system of FIG. 2 . 5 is a flow diagram for explaining an example of face authentication processing in the information processing device of FIG. 4. FIG. 11 is a schematic diagram illustrating a configuration example of an information processing device according to a third embodiment. FIG. 2 illustrates an example of a hardware configuration of the apparatus.

Hereinafter, the embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and duplicate descriptions will be omitted as appropriate.

Hereinafter, the embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and duplicate descriptions will be omitted as appropriate.

<Embodiment 1>
FIG. 1 is a block diagram showing an example of the configuration of an information processing apparatus according to the first embodiment.
As shown in FIG. 1, an information processing device 1 according to this embodiment can include a sensor 1a, an imaging device 1b, a determination unit 1c, a face authentication unit 1d, and an output unit 1e.

Sensor 1a is a sensor that detects the entry of a person into a room that is the subject of entry monitoring, and can be a human presence sensor installed in a position where it can detect entry. Various rooms, such as toilet cubicles and bathrooms, can be used as rooms that are the subject of entry monitoring.

The following description will be given using an example where the authentication subject, who is the person to be monitored for entry, is a care recipient, in other words, the room to be monitored for entry is a room used by the care recipient; however, other types of authentication subjects can also be the subject of entry monitoring. Here, the authentication subject refers to a person for whom face authentication is performed by the face authentication unit 1d and the results are output. A care recipient can also be referred to as a person receiving care, a person requiring care, or a person requiring assistance. A caregiver can be a caregiver, a doctor, or the like, and can also be referred to as a caregiver.

When the person to be authenticated is a person receiving care, the rooms to be monitored include the toilet and bathroom mentioned above. The information processing device 1 can be used in places where it is necessary to watch over people requiring care, such as elderly care facilities, private homes, and hospitals.

Sensor 1a may be, for example, an infrared sensor or an optical sensor. Sensor 1a is not limited to infrared, and may be a sensor using any method, such as image recognition or ultrasonic waves. It is also preferable that sensor 1a be able to detect exit from a room after entry. In a simple example, sensor 1a can detect that a person has left the room if a person is detected after entry.

When the sensor 1a detects a person entering the room, the imaging device 1b captures the person entering the room to obtain image data. The imaging device 1b can be, for example, a camera that captures still images and/or videos, and can be called a face recognition camera because the captured image data is used in the face recognition unit 1d at a subsequent stage. Furthermore, the imaging device 1b is not limited to a visible light camera, and may be an infrared light camera, etc.

The determination unit 1c provisionally determines whether the person who has entered the room is a person to be authenticated as a user of the room (a person receiving care in this example) or not, by determining whether the person is wearing a mask based on the image data obtained by the imaging device 1b. A provisional determination by the determination unit 1c that the person is not a person receiving care means that the person is provisionally determined to be a caregiver. The determination unit 1c can also be configured to determine whether the person who has left the room is a person receiving care.

If the result of the provisional judgment by the judgment unit 1c is that the person is the person to be authenticated (in this example, a person receiving care), that is, if the person is not wearing a mask, the face authentication unit 1d executes face authentication processing based on the face image data of the person to be authenticated in the image data. If the image data includes face image areas of multiple people, the face authentication processing can be executed based on the data of the face image areas of the person to be authenticated among all the people.

The facial authentication unit 1d performs the facial authentication process by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated (a person receiving care in this example) and for each attendant (a caregiver in this example) who may accompany the person to be authenticated and enter the room. Therefore, if the result of the provisional determination by the determination unit 1c is that the person is an attendant (a caregiver in this example), that is, a person wearing a mask, the facial authentication unit 1d does not perform the facial authentication process and ends the subsequent processing.

The above feature data can be, for example, data indicating facial feature points. This comparison can also be performed, for example, by inputting facial image data of the care recipient who has entered the room into a learning model (trained model) that has been machine-trained using learning data that includes the feature data and the image data (facial image data) on which the feature data is based.

This type of facial recognition processing makes it possible to identify the person who has entered the target room from the facial image data of the person who has become the authentication target through the provisional determination, that is, to identify (identify) the individual who has entered the target room. The person identified here is the result of the above-mentioned comparison, and may therefore be an attendant through the provisional determination. The facial recognition unit 1d can also be called an identification unit, as it identifies individuals in this way.

The determination unit 1c and the face authentication unit 1d may have some common components. For example, the determination unit 1c may use intermediate data from the recognition process in the face authentication unit 1d. In other words, the determination unit 1c may be included as part of the function of the face authentication unit 1d.

In addition, it is possible that the users of the room may be people who do not fall into the category of either a care recipient or a caregiver (such as family members visiting a patient), but such people may fail the facial recognition process and be determined to be unknown people.

The output unit 1e outputs the result of the face recognition process to the notification destination. The output unit 1e can be, for example, a communication unit configured with a wired or wireless communication interface. The output unit 1e can, for example, transmit information indicating the result of the face recognition process to a destination terminal device (not shown) via the communication unit. The output unit 1e can, for example, transmit this information to an external server (not shown) via the communication unit, so that the server can transfer the notification information to the destination terminal device. The notification destination can be set in advance, and it is preferable that the notification destination be changeable.

The output unit 1e can also output the result to the notification destination regardless of the result of the facial recognition process. However, if the output unit 1e authenticates an attendant (a caregiver in this example) in the above facial recognition process, it can also exclude the result of the facial recognition process for the attendant and output the result of the facial recognition process to the notification destination. In other words, the output unit 1e can output the result to the notification destination when the person to be authenticated (the person being cared for in this example) is authenticated, and can not output the result when the attendant is authenticated. In addition, in the case of an unknown person, the output unit 1e can either notify that an unknown person has entered the room or not notify, and it is sufficient to determine in advance which process to execute.

The information processing device 1 may also have a control unit (not shown), which may include the above-mentioned determination unit 1c, face authentication unit 1d, and output unit 1e (or a part thereof). The control unit may be realized, for example, by a CPU (Central Processing Unit), a working memory, and a non-volatile storage device storing a program. The program may be a program for causing the CPU to execute a process of inputting the detection result from the sensor 1a, a process of inputting image data captured by the imaging device 1b, and the processes of the determination unit 1c, face authentication unit 1d, and output unit 1e. The control unit provided in the information processing device 1 may also be realized, for example, by an integrated circuit. The above-mentioned storage device may also store a notification destination, and the output unit 1e may output the result of the face authentication process by referring to the notification destination.

In addition, the information processing device 1 according to this embodiment is assumed to be configured as a single information processing device, but it can also be configured as multiple devices with distributed functions. In the latter case, each device is provided with a control unit, a communication unit, and, if necessary, a memory unit, etc., and these multiple devices can be connected by wireless or wired communication as necessary to work together to realize the functions of the information processing device 1.

Here, the information processing method according to the present embodiment will be briefly described. This information processing method performs the following information processing. In the above information processing, the sensor 1a detects the entry of a person into a room, and when the sensor 1a detects the entry of a person, the imaging device 1b captures the person who entered the room to obtain image data. Furthermore, the above information processing determines whether the person who entered the room is wearing a mask based on the image data obtained by the imaging device 1b, thereby provisionally determining whether the person who entered the room is a person to be authenticated as a user of the room or not. Furthermore, in the above information processing, if the result of the provisional determination is that the person is a person to be authenticated, face recognition processing is performed based on face image data of the person to be authenticated in the image data, and the result of the face recognition processing is output to the notification destination. The face recognition processing is performed by comparing the face image data with feature data indicating face features stored in advance for each person to be authenticated and for each attendant. Here, when an attendant is authenticated as a result of the face recognition processing, the output to the notification destination can also be performed by excluding the result of the face recognition processing for the attendant. Furthermore, the above-mentioned program can be a program for causing a computer to execute the above-mentioned information processing.

As described above, in this embodiment, when a person accompanying a user of a room (e.g., a caregiver or other attendant) is wearing a mask, face authentication of the user can be performed in real time with high accuracy without reducing processing power, and the authentication result can be output.

For example, in this embodiment, the facial feature data of the caregiver is preregistered and the facial recognition process is performed, and the mask detection function is provided, so that highly accurate facial recognition processing can be achieved even with a space-saving, power-saving CPU built into an information processing device as an edge device. To give a more specific example, in this embodiment, these functions allow a caregiver wearing a mask to be excluded from facial recognition targets when entering the restroom to clean the restroom, by simply performing a mask determination process, which is a lighter process compared to the facial recognition process. Therefore, in this embodiment, facial recognition processing can be performed without significantly impairing performance even when multiple caregivers are assisting one user. Also, in this embodiment, a configuration can be adopted in which a notification is not issued even when a caregiver not wearing a mask enters the restroom.

<Embodiment 2>
The second embodiment will be described with reference to Figures 2 to 9, focusing on the differences from the first embodiment, but the various examples described in the first embodiment can also be applied. Figure 2 is a diagram showing a configuration example of an information processing system including an information processing device according to the second embodiment, and Figure 3 is a side view showing an example of installation of the information processing device in the information processing system of Figure 2.

The information processing system according to this embodiment (hereinafter, this system) can be a system in which a part is installed in a toilet and sends notifications to a notification destination (e.g., a caregiver, etc.), and will be described in detail below.

As shown in FIG. 2, the system can include an information processing device 60 attached to the toilet bowl 10, a server 40 wirelessly connected to the information processing device 60, and a terminal device 50 wirelessly connected to the server 40. These connections can be made, for example, within a single wireless LAN (Local Area Network). For example, the system can be configured so that the information processing device 60 can directly transmit information to the terminal device 50. In the following, an example will be described in which the information processing device 60 is installed in the toilet bowl 10 and also acquires excretion information, but excretion information can be acquired if at least a part of it is installed in the toilet bowl 10, such as a camera for capturing image data to obtain excretion information.

The toilet bowl 10 can have an edge 10c that forms a water-retaining area between the side surface 10a and the edge 10c, and can also have a flat surface 10b that continues from the edge 10c. The flat surface 10b can be provided with a toilet seat 11 that includes, for example, a toilet seat body 11a equipped with a warm water cleaning function for the user to wash, and a toilet seat cover 11b for covering the toilet seat body 11a.

The information processing device 60 can be a device installed in the toilet (a toilet-mounted device in this example), and notifies the terminal device 50 of detection events, such as the detection of the entry of a care recipient, directly or via the server 40. Here, an example will be described in which the information processing device 60 is an excretion information collection device with a function of acquiring excretion information. In this case, excretion-related events, such as the start and end of excretion, can also be notified as detection events. Although not described in detail, the information processing device 60 can also be configured to record excretion, present excretion information (notification, etc.), and predict excretion in cooperation with the server 40 and the terminal device 50. The information processing device 60 and the toilet 10 can, for example, constitute a toilet with a function of outputting excretion information.

The shape of the information processing device 60 is not limited to the shape shown in FIG. 2, and it can be configured so that all or part of its functions are embedded in the toilet seat body 11a, for example. Also, part of the functions of the information processing device 60 can be provided on the toilet seat body 11a side.

The information processing device 60 is an example of the information processing device 1 in FIG. 1, and has the functions of each part of the information processing device 1. Before describing a detailed example of the information processing device 60, we will briefly describe the main components, including the components corresponding to each part of the information processing device 1 in FIG. 1.

As shown in Figures 2 and 3, the information processing device 60 according to this embodiment can include a human presence sensor 24, an imaging device 25, an information collection unit 26, a storage housing 61, a bridge unit 62, an inner housing 63, a control unit 67, a separate housing 68, and an outer housing 69.

The human presence sensor 24 is an example of a sensor 1a, and is a sensor that detects a person entering a toilet stall, such as a user (customer) of the toilet bowl 10, and examples of such sensors include an infrared sensor and an optical sensor.

The imaging device 25 is an example of the imaging device 1b, and is a device that captures the face of a person entering a toilet stall, such as a toilet user, and can be, for example, a camera that captures still images and/or videos (a face recognition camera). Furthermore, the imaging device 25 is not limited to a visible light camera, and may be an infrared light camera, etc.

The information collecting unit 26 is disposed in the inner housing 63. However, the information collecting unit 26 is disposed so that information can be collected from the accumulation portion of the toilet bowl 10 (the area where excrement is excreted) by, for example, leaving the information collecting surface of the information collecting unit 26 exposed from an opening in the inner housing 63. The information collecting surface corresponds to, for example, a lens surface if the information collecting unit 26 is a camera, or a detection surface if the information collecting unit 26 is a sensor.

The information collecting unit 26 is a part that collects information about the excrement in the toilet bowl 10, and can be, for example, an imaging device that collects the contents of the excrement, such as the shape and color of the excrement, through an optical lens, or a distance sensor that optically measures distance. The excrement information can be information that indicates the contents of the excrement, and in a simpler example, it can be information that indicates whether the excrement is stool (feces) or urination (urine). The excrement information can also include other information, such as information that indicates the color of the excrement, and if it is a solid, its shape. The information collecting unit 26 may simply be configured to detect the presence or absence of excrement and collect the information about the excrement.

The information collecting unit 26 can set the area including the water-filled portion of the toilet bowl (excretion range) as the collection range, and this excretion range can also be called the planned excretion range. By installing the information collecting unit 26 so that such an excretion range is included in the imaging range, the captured imaging data will include excrement and the like as subjects. Of course, it is preferable that the excretion range is a range in which the user is not reflected, and it is also preferable that the information collecting unit 26 is installed so that lenses and the like are not visible to the user.

The information collection unit 26, the human presence sensor 24, and the imaging device 25 are connected to the control unit 67 by wired or wireless communication. Note that FIG. 2 shows an example in which the information collection unit 26 is connected to the control unit 67 by cable K1, and the human presence sensor 24 and the imaging device 25 are connected to the control unit 67 by cable K2.

The control unit 67 controls the human sensor 24, the imaging device 25, and the information collection unit 26. For example, the control unit 67 can receive the detection result of the human sensor 24, can instruct the imaging device 25 to capture an image and receive the captured image, and can instruct the information collection unit 26 to collect information on excrement and receive the collected information. The control unit 67 can also be configured to execute the functions of the determination unit 1c, the face authentication unit 1d, and the output unit 1e. The control unit 67 can be realized, for example, by a CPU, a working memory, and a non-volatile storage device that stores a program. This program can be, for example, a program that causes the CPU to execute the above-mentioned processes. The control unit 67 can also be realized, for example, by an integrated circuit.

The storage housing 61 is a housing that is placed on the outside of the toilet bowl 10 and that stores the control unit 67, and is hereinafter described as the control box 61. The bridge 62 is a part that bridges the inner housing 63 and the outer housing 69. In other words, the bridge 62 is a mechanism that connects the inner housing 63 and the outer housing 69, and is installed on the edge 10c of the toilet bowl 10. The outer housing 69 is a housing that includes a clamp mechanism for fixing the inner housing 63 and the toilet bowl 10. The outer housing 69 is a housing that is placed on the outside of the toilet bowl 10, and its shape does not matter as long as it does not interfere with excretion.

The above-mentioned clamp mechanism (fixing clamp mechanism) can be a mechanism that fixes the inner housing 63 and the outer housing 69 to the edge portion 10c, for example, by adjusting the distance between the inner side of the edge portion 10c via the bridge 62 and clamping it. This allows the inner housing 63 of the information processing device 60 to be reliably fixed and installed regardless of the shape of the toilet bowl 10, such as the shape of the edge portion 10c. Furthermore, with such a mechanism, the shape and components of the fixing clamp mechanism are not important. The main components of the fixing clamp mechanism can be provided on the outer housing 69.

The inner housing 63 is a housing in which the information collection unit 26 is arranged, and will be described below as the sensor box 63. The sensor box 63 is equipped with, for example, an optical sensor for collecting information about excrement, and is disposed inside the toilet bowl 10.

The separate housing 68 is a housing in which the human presence sensor 24 and the imaging device 25 are arranged. The separate housing 68 has the human presence sensor 24 for human detection and the imaging device (camera) 25 for face authentication arranged inside or partially outside, and will be described below as the human detection/identification box 68. The human detection/identification box 68 can be equipped with a camera adjustment mechanism for adjusting the direction of the camera horizontally. The imaging device 25 can be exposed from the human detection/identification box 68, for example, so that the face of the toilet user can be imaged, but it is sufficient if the face can be imaged. The human presence sensor 24 can be exposed from the human detection/identification box 68, so that the toilet user can be detected, but it is sufficient if the human presence sensor 24 and the imaging device 25 are connected to the control unit 67 by wire or wireless communication, so that information can be sent and received.

The human detection/identification box 68 can be disposed at a position separated from the control box 61 and the toilet 10, as exemplified in Figures 2 and 3. Figures 2 and 3 show an example in which the human detection/identification box 68 is installed on the wall W of the toilet in which the toilet 10 is installed. Of course, the human detection/identification box 68 can be installed on other places besides the wall W, such as the ceiling of the room or a shelf installed in the room. The human detection/identification box 68 can adopt a wired connection via cable K2 to the control box 61, or a wireless connection, and in either case, it can be used at a position separated from the control box 61. The human detection/identification box 68 rotates horizontally, and a mechanism for fixing the orientation of the rotating box can be provided.

In order to facilitate the detection and imaging of people, the human presence sensor 24 and the imaging device 25 are preferably disposed so that the detection surface and imaging lens are located at a position at least higher than the bridge 62. It is more preferable to provide the detection surface and imaging lens at a position that is a predetermined height higher, taking into account the height of the toilet seat 11.

In addition, in the configuration of the information processing device 60 illustrated in FIG. 2, the control box 61 having the human detection/identification box 68 and the control unit 67, and the sensor box 63 having an excretion information collection function are connected by cables K1 and K2. However, the human detection/identification box 68 is disposed on the wall W of the toilet, and the box having the control unit 67 is disposed on the side of the toilet bowl 10, making the installation easy. The control box 61 can also be disposed on the back side of the toilet bowl 10, in which case the installation is also easy. It is also desirable that both cables K1 and K2 include a power supply line that supplies power from the control box 61. Note that power is supplied to the control box 61 from a power cable not shown.

In FIG. 2, it is assumed that cable K1 passes through the inside of bridge 62, but cable K1 can also be connected to information collection unit 26, led out from sensor box 62, and directly connected to control box 61. In that case, cable K1 passes through the upper surface side of edge 10c of toilet bowl 10 (passes between the upper surface of edge 10c and toilet seat body 11a).

The server 40 may include a control unit 41 for controlling the whole server 40, and a storage unit 42 for storing, for example, notification conditions and various acquired information (and information generated based on the notification conditions). The control unit 41 may be configured to transfer notifications to the terminal device 50 as a part of the functions of the output unit 1e in FIG. 1. For example, the storage unit 42 may store a notification destination for each detected event and for each assisted user as a notification condition. This allows the server 40, which has a part of the functions of the output unit 1e, to output to the notification destination in response to the occurrence of a detected event according to the notification conditions. Note that the number of notification destinations for a certain detected event may be one or more. In addition, the server 40 preferably includes a setting unit for setting notification conditions from an external device such as the terminal device 50, which allows the notification conditions to be set according to the operation mode.

The control unit 41 can be realized, for example, by a CPU, a working memory, and a non-volatile storage device that stores a program. The control unit 41 can also be realized, for example, by an integrated circuit. This storage device can also be used as the storage unit 42, and this program can be a program for causing the CPU to realize the functions of the server 40.

The terminal device 50 is a terminal device held by a caregiver who is caring for (or will be caring for) a care recipient who is a toilet user, and can be a portable information processing device, but can also be a stationary device. In the former case, the terminal device 50 can be a smart device such as a mobile phone (including what are called smartphones), a tablet, or a mobile PC, but can also be an electronic bulletin board, etc. Although not shown, the terminal device 50 can also be equipped with a control unit that controls the entire device, and a storage unit, and this control unit can be realized, for example, by a CPU, working memory, storage device, etc., just like the control unit 41. In addition, the program stored in this storage device can be a program for causing the CPU to realize the functions of the terminal device 50.

Next, a detailed example of the information processing device 60 will be described with reference to FIG. 4. FIG. 4 is a block diagram showing an example of the configuration of an information processing device in the information processing system of FIG. 2. The information processing device 60 can be composed of two devices, for example, as shown in FIG. 2 and FIG. 3. The information processing device 60 can also be called a "toilet sensor" or "toilet sensor device" because it has various sensors installed in the toilet. The toilet sensor corresponds to a so-called edge in this system, which monitors the use of the toilet by the care recipient via a communication network.

The control box 61 can house, for example, a CPU 27a, a connector 27b, USB I/Fs 27c and 27d, and a WiFi module 27e. Note that USB is an abbreviation for Universal Serial Bus, and both USB and WiFi are registered trademarks (the same applies below). Note that the control box 61 can be directly connected to the CPU 27a without being provided with various I/Fs and connectors.

As shown in FIG. 4, the control box 61 and the sensor box 63 are connected by an interface exemplified by the connector 27b and the USB I/F 27c, and the connection line is housed in the cable K1. A part of the cable K1 passes through the inside of the bridge 62 (and the outer housing 69). The control box 61 and the human detection/identification box 68 are connected by an interface exemplified by the connector 27b and the USB I/F 27d, and the connection line is housed in the cable K2.

In this way, the information processing device 60 is equipped with circuit boards and cables connecting them in each box and bridge 62. By adopting the divided box configuration described above and optimizing the orientation of the connectors that connect the cables, the information processing device 60 can reduce the space required for the excess cable length and reduce the size of the circuit board and mounted components. Also, in this embodiment, the placement locations of each box are separated, making it possible to reduce the width of each structure.

The sensor box 63 can house, for example, a distance sensor 26a that functions as a seating sensor to detect when the toilet seat 11 is sat on, and a first camera 26b that captures images of excrement.

Next, the components within each box will be described.
The CPU 27a is an example of a main control unit of the information processing device 60, and controls the entire information processing device 60. The connector 27b connects the human presence sensor 24 and the distance sensor 26a to the CPU 27a. The USB I/F 27c connects the first camera 26b to the CPU 27a, and the USB I/F 27d connects the second camera 25 to the CPU 27a.

The distance sensor 26a is a sensor that measures the distance to an object (the buttocks of the user of the toilet 10) and detects that the user has sat on the toilet seat 11, and detects that the object has sat on the toilet seat 11 when a certain amount of time has elapsed after the threshold value is exceeded. In addition, the distance sensor 26a detects that the user has left the toilet seat 11 if the distance to the object changes after sitting down.

The distance sensor 26a may be, for example, an infrared sensor, an ultrasonic sensor, an optical sensor, or the like. When an optical sensor is used for the distance sensor 26a, a transmitting/receiving element may be arranged so that light (not limited to visible light) can be transmitted and received through a hole provided in the sensor box 63. The transmitting/receiving element may be configured as a transmitting element and a receiving element separately, or may be integrated. The distance sensor 26a is connected to the CPU 27a via the connector 27b, and is capable of transmitting the detection result to the CPU 27a. Based on the detection result, the CPU 27a can obtain seating/departure data indicating information on whether a person is seated or not, and can also transmit the data to the server 40 via the WiFi module 27e.

The first camera 26b is an example of a camera that captures image data that is the basis for obtaining excretion information, and can be an optical camera with a lens portion disposed in a hole provided in the sensor box 63. The first camera 26b is installed so that the excrement excretion range in the toilet bowl 10 is included in the imaging range. The first camera 26b is connected to the CPU 27a via the USB I/F 27c, and transmits the imaging data to the CPU 27a.

The human presence sensor 24 is an example of a sensor 1a, and is a sensor that detects the presence of a person (entry or exit) in a specific area (measurement area range of the human presence sensor 24) that is part of the toilet room, and can be called an entry/exit sensor. The human presence sensor 24 can employ any detection method, such as an infrared sensor, ultrasonic sensor, or optical sensor. The human presence sensor 24 is connected to the CPU 27a via the connector 27b, and when it detects a person in the specific area, it transmits the detection result to the CPU 27a. This detection result can also be transmitted by the CPU 27a to the server 40 via the WiFi module 27e.

The second camera 25 is an example of an imaging device 1b, and can be an optical camera, and is an example of a face recognition camera that captures a facial image of a user, etc., to obtain face image data in order to identify the user of the toilet. The second camera 25 can be installed in the toilet room (inside the toilet) in which the toilet bowl 10 is installed. The CPU 27a performs face recognition processing using the face image data captured by the second camera 25. In this system, when the human presence sensor 24 detects a person, the second camera 25 can be processed to capture an image of the subject who has entered the toilet, and thus face recognition can be performed only when a person is detected.

The WiFi module 27e is an example of a communication device that transmits various acquired data to the server 40, and can be replaced with a module that employs other communication standards. The data to be transmitted can be transmitted to the server 40 by the CPU 27a via the WiFi module 27e. Note that facial image data acquired by the second camera 25 is not generally transmitted in order to reduce the amount of data and to consider privacy.

The CPU 27a provisionally determines whether the person who has entered the room is a care recipient as a user of the room or not by determining whether the person is wearing a mask based on image data obtained by the second camera 25. A provisional determination that the person is not a care recipient means that the person is provisionally determined to be a caregiver.

Furthermore, if the result of the provisional determination is that the person is a care recipient, i.e., that the person is not wearing a mask, the CPU 27a executes face recognition processing based on the facial image data of the care recipient in the image data. If the image data includes facial image areas of multiple people, the face recognition processing may be executed based on the data of the facial image areas of the care recipients among all the people. Here, the face recognition processing is executed by comparing the facial image data with feature data indicating facial features stored in advance for each care recipient and each caregiver who may enter the room accompanying the care recipient. Therefore, in the face recognition processing, if the result of the provisional determination is that the person is a caregiver, i.e., that the person is wearing a mask, the face recognition processing is not executed and the subsequent processing is terminated.

The feature data may be data indicating facial feature points, for example. The feature data may be data generated from facial image data acquired by a terminal device such as the terminal device 50 and stored in an internal storage device of the information processing device 60. This internal storage device may be connected to the CPU 27a, for example (not shown), or may be provided inside the CPU 27a. This comparison may also be performed, for example, by inputting facial image data of the care recipient who has entered the room into a learning model (trained model) that has been machine-trained using learning data including the feature data and the image data (facial image data) on which the feature data is based.

This facial recognition process makes it possible to identify the person who entered the target room from the facial image data of the person who has been provisionally determined to be a care recipient, that is, to identify (identify) the individual who entered the target room. The person identified here is the result of the above-mentioned comparison, and may be provisionally determined to be a caregiver, in which case the facial recognition result may not be output. In addition, it is possible for the user of the room to be someone who does not fall into either the category of a care recipient or a caregiver (for example, a family member who has come to visit), and such a person may be identified as an unknown person as a result of the facial recognition process failing to be recognized, and a notification of this may be sent. In consideration of privacy, it is preferable not to store the facial image data captured by the second camera 25 after the facial recognition process.

The CPU 27a and the WiFi module 27e can be an example of the output unit 1e in FIG. 1. The CPU 27a outputs the result of the face recognition process to the notification destination via the WiFi module 27e. If the CPU 27a authenticates a caregiver in the face recognition process, it can also output the result of the face recognition process to the notification destination, excluding the result of the face recognition process for the caregiver. If the result of the face recognition process is an unknown person, the CPU 27a may or may not notify that an unknown person has entered the room, and it is sufficient to determine in advance which process to execute.

In this way, the CPU 27a can be responsible for the main processing of detecting the detection event indicated by the results of the face recognition processing, excretion information, and seating/leaving information. The notification destination can be the terminal device 50 held by one or more of the caregivers, and the caregiver to be the notification destination can be included in the notification conditions. The caregiver can be notified by displaying and/or outputting sound on the terminal device 50. In addition, the terminal device 50 has been described assuming that it is a smart device, but it may also be a device that simply emits sound when a notification is received, or an electronic bulletin board that displays the entry and personal name. In this way, the notification destination may be, in addition to or instead of the terminal device 50, a notification device of a nurse call system, another terminal device (e.g., PHS: Personal Handy-phone System) held by the caregiver other than the terminal device 50, an intercom, etc. In addition, details of the notification of excretion information and seating/leaving information are omitted, but both can be treated as one of the detection events.

The CPU 27a, the USB I/F 27c, the WiFi module 27e, and the server 40 obtain the above-mentioned excretion information based on the imaging data captured by the first camera 26b. In this case, the main process of obtaining the excretion information from the imaging data can be performed by the server 40. However, the CPU 27a can also be configured to perform this process. In order to obtain the excretion information indicating the content of the excretion based on the imaging data, the server 40 or the CPU 27a includes an analysis unit that analyzes the excretion information based on the imaging data obtained by the first camera 26b. This analysis unit performs analysis of the excretion information when the care recipient can be authenticated as a result of the above-mentioned face recognition process, that is, when the entrant can be identified and is the care recipient. Imaging by the first camera 26b can also be performed only when such a result of the face recognition process is obtained.

In the example of FIG. 2, the server 40 inputs the image data into the trained model, for example, to obtain excretion information. Preferably, the server 40 obtains information indicating whether the image data includes a foreign object other than feces and urine as a subject other than the toilet bowl and its cleaning liquid, as part of the excretion information. The foreign object may be called a "other object" and may be liquid or solid other than feces and urine, and may include, for example, one or more of vomit, bloody stool, vomited blood (vomited blood), diapers, urine pads, toilet paper tubes, etc. In addition, the server 40 may be configured to obtain the shape, color, and amount of the excretion as excretion information when the excretion is not a foreign object but excretion. The CPU 27a and the WiFi module 27e, or the server 40, may also notify the notification destination of the analysis results in the analysis unit. In this case, the notification destination may be different from the notification destination of the result of the face recognition process, and the notification destination may be stored, for example, as a notification condition, in the storage device or storage unit 42 in the information processing device 60.

In addition, the server 40 can store information such as excretion information and information on leaving and sitting, for example, in the memory unit 42, and the server 40 can output it in response to access from the terminal device 50. In particular, it is preferable that the server 40 has a function for generating an excretion diary. In this case, if the generated excretion diary is stored in the memory unit 42, the caregiver who is the user of the terminal device 50 can view the excretion diary from the terminal device 50 whenever he or she wishes. The excretion diary will include the date and time of the excretion act, and this date and time can be obtained from any of the following: the image capture date and time of the image data of the excretion, the identification date and time of the subject, the entry and exit date and time, the sitting date and time, the leaving date and time, the intermediate date and time between sitting and leaving, etc.

Next, an example of the process of generating and registering facial feature data will be described with reference to FIG. 5. FIG. 5 is a schematic diagram for explaining an example of the process flow of generating and registering facial model data in the information processing system of FIG. 2.

As shown in FIG. 5, the facial feature data (hereinafter, face model data) is registered by generating face model data and transmitting the face model data to the information processing device 60. First, the face photo used to generate the face model data can be imaging data obtained by a caregiver or the person being cared for using a terminal such as the terminal device 50. This terminal transmits the obtained imaging data to a server 40 on the cloud, and the server 40 generates face model data for face recognition from the imaging data. Regardless of the type of face model data, it is sufficient that the authentication accuracy in the face recognition process is high, but it is preferable to use face model data that has a light processing load in the face recognition process. The face model data can be data including information extracted from a face photo, such as the eyes, mouth, nose, and contour. In addition, the care recipient can take a face photo of himself/herself using a terminal equivalent to the terminal device 50, transmit the imaging data to the server 40, and the server 40 can generate the face model data.

Then, the server 40 transmits the generated face model data to the information processing device 60, and the information processing device 60 registers, i.e. installs, the face model data so that the face model data can be referenced during face recognition processing, and retains the face model data. When transmitting data from the server 40 to the information processing device 60, it is possible to transmit only the modeled data in this way, without transmitting image data such as a facial photograph itself.

The generation of the facial model data can also be performed by a server other than server 40. In addition, when the facial model data can be generated by a terminal such as terminal device 50 that captured the facial photograph, it is also possible to transmit the data generated by the terminal directly to information processing device 60 without using a server such as server 40.

Next, the functions corresponding to the determination unit 1c and face authentication unit 1d, that is, face detection, face identification (face authentication), and mask detection, will be described with reference to Figures 6 and 7. Figure 6 is a schematic diagram for explaining an example of face authentication processing in the information processing device 60 when the caregiver is not wearing a mask. Figure 7 is a schematic diagram for explaining an example of face authentication processing in the information processing device 60 when the caregiver is wearing a mask.

Facial detection, facial identification (facial authentication), and mask detection can be performed using neural network-based algorithms using machine learning. Figure 6 shows an overview of the facial authentication function when the caregiver is not wearing a mask. As shown in Figure 6, when the CPU 27a receives the detection result from the human sensor 24 and detects entry into the toilet, it starts the second camera 25 for facial authentication, scans the imaging range of the second camera 25, and when a face is detected, surrounds the detected face with a rectangle. As a result, as shown in image FD in Figure 6, the facial areas of the care recipient and the caregiver in the imaging data are surrounded by rectangles fa1 and fa2, respectively.

Next, the CPU 27a performs face recognition processing by comparing the detected face areas fa1 and fa2 with face model data, as shown in image FR1 of FIG. 6. If the face recognition processing results in the determination that the person is a caregiver (in this example, the face area fa2 is determined to be the face of the caregiver), the CPU 27a excludes the face area fa2 from the face recognition target, as shown in image FR2 of FIG. 6, even if the caregiver is within the imaging range of the second camera 25. The determination that the person is a caregiver can be made, for example, by first detecting a certain feature point in the face model data of the caregiver. This feature point can be registered in advance as a feature point such as a hat, hairpin, hairstyle, etc. Finally, the CPU 27a can identify that this face is care recipient A by performing face recognition processing on face area fa1 in image FR2.

As shown in Figure 7, an overview of the face recognition function when the caregiver is wearing a mask is shown. First, when the CPU 27a receives the detection result from the human presence sensor 24 and detects entry into the toilet, it activates the second camera 25 for face recognition, scans the imaging range of the second camera 25, and when a face is detected, it surrounds the detected face with a rectangle. As a result, as shown in image FD in Figure 7, the facial areas of the care recipient and the caregiver in the imaging data are surrounded by rectangles fa1 and fa2, respectively.

When there is a mask in the rectangular facial areas fa1 and fa2, the CPU 27a determines that a mask is being worn by surrounding the mask with a rectangle (in this case, rectangular area ma2) as shown in image MD in Figure 7. If the CPU 27a determines that the person is wearing a mask, i.e., a caregiver, as a result of the mask wearing determination, the CPU 27a excludes facial area fa2 from facial recognition as shown in image FR in Figure 7, and does not perform face matching, even if the caregiver is within the imaging range of the second camera 25. The CPU 27a executes facial recognition processing to match the facial area fa1 not wearing a mask in image FR in Figure 7 with face model data, and can ultimately identify that this face is care recipient A.

Note that in Figures 6 and 7, a rectangular area is actually enclosed, but it is sufficient if a rectangular area can be detected and that area can be specified. Also, the shape of the face can be other than rectangular, such as an ellipse. Also, the mask can be other than rectangular, such as a polygon with five or more sides, or an ellipse.

Next, an example of displaying notification information on the terminal device 50 will be described with reference to FIG. 8. FIG. 8 is a diagram for explaining an example of notification on a terminal device in the information processing system of FIG. 2.

As shown in FIG. 8, the terminal device 50 can notify notification information 52 to 54 in that order on its display screen 51. Notification information 52 is an example of notification information that notifies the entry of a care recipient into the toilet when a person is detected by the human presence sensor 24 and the person is provisionally determined to be a care recipient. As in this example, a notification may also be sent to the notification destination when the entry of a care recipient, i.e., the entry of a person not wearing a mask, is detected as a result of the provisional determination.

After notification of notification information 52, notification information 53 is notified as a result of the facial recognition process. This is an example of notification information notifying that the person who has entered the room is the user (in this example, the care recipient PA) as a result of identifying the user through facial recognition processing. Note that it is possible to know that the person is PA if a name is associated with the facial model data.

In addition, when it is detected that PA has left the room, notification information 54 indicating this can be displayed. Furthermore, after notification information 52 is sent, if facial recognition processing detects that the person entering the room is a caregiver, the name of the individual caregiver (e.g., C) can be sent, or no notification need be sent. As an alternative process, in this case, a notification can be sent that excludes the results of facial recognition processing, that is, a notification can simply be sent that the person entering the room was a caregiver.

Next, with reference to FIG. 9, a flow of face recognition processing and notification in this system, which is triggered by the entry of a person requiring care into a room, will be described. FIG. 9 is a flow diagram for explaining an example of face recognition processing in the information processing device 60. Note that the processing in the information processing device 60 is mainly executed by the CPU 27a.

First, the information processing device 60 monitors whether or not there has been a reaction from the human presence sensor 24, which functions as an entrance sensor (step S1), and when there is a reaction (step S1 is YES), it activates the second camera 25, which is an optical camera (step S2). When step S1 is NO, that is, when there is no reaction from the human presence sensor 24, it waits until there is a reaction.

In this way, when a person requiring care enters the toilet, step S1 becomes YES and the second camera 25 is activated. The activated second camera captures an image of the person who has entered, and the CPU 27a or the second camera 25 body performs a process to detect a face within the imaging range and determines whether or not a face has been detected (step S3). In step S3, as illustrated in Figures 6 and 7, when the second camera 25 detects a face, the detected face is surrounded by a rectangle. The determination in step S3 is performed while changing the imaging range of the second camera 25 as necessary and capturing images until a face is detected. Step S3 is executed until a face is detected, but the process can also be terminated after a predetermined time has elapsed, for example.

The CPU 27a performs mask detection by determining whether or not a mask is being worn for the face area detected in step S3 (step S4). If step S4 returns YES, i.e., if it detects that the person is wearing a mask, the CPU 27a determines that the person is a caregiver, terminates the face recognition process, and does not perform any further recognition process on the face image of the person. On the other hand, if step S4 returns NO, i.e., if the person is not wearing a mask, the CPU 27a compares the face area (image data of the face area) with face model data (step S5). When comparing, it is also possible to generate face model data for the image data of the face area in the same way as when the face model data was generated, and compare the two.

As a result of the comparison in step S5, the CPU 27a determines whether the image data of the face area matches or corresponds to the registered face model data (also simply referred to as face data) (step S6). If the answer is YES in step S6, the CPU 27a determines whether the user is a care recipient or a caregiver (step S7). If the CPU 27a determines in step S7 that the user is a caregiver, the process ends. On the other hand, if the CPU 27a determines in step S7 that the user is a user, the CPU 27a works with the notification function and excrement analysis function to notify the user of the person entering the room and analyze the excrement, and processes the notification of excrement information (step S8), and ends the process. The cooperation in step S8 can also include cooperation with the server 40.

Also, if step S6 is NO, that is, if the image data of the face area is unregistered or has been registered but cannot be matched, the CPU 27a marks the occupant as unknown (step S9), links with the notification function and excrement analysis function (step S8), and ends the process. Note that after the process of step S9, the process can also end without such linkage. In this way, when the identified occupant is a user, the information processing device 60 can link with processes such as analysis and notification of excrement indicating the user's toilet usage status (analysis of the user's status and notification to the caregiver).

The above description of this system assumes that there is only one toilet 10 (only one toilet room). However, in this system, it is preferable to obtain face recognition results for detecting detection events for multiple toilets. This makes it possible to apply this system even in cases where a certain person being assisted may use two or more toilets.

In addition, server 40 can be installed within a facility such as a hospital, and in the case of personal use, it can be installed in a private home or an apartment building. In either case, server 40 can be a cloud server as described above.

The program of the terminal device 50 can be executable and embedded in the terminal device 50 as nursing software including a notification function of notification information. In addition to notification information, the terminal device 50 can also directly (or directly and automatically) receive and store information obtained on the toilet side such as the information processing device 60, and similarly, can receive and store various information recorded in the server 40.

Of course, it is also possible to configure the information processing device 60 by itself to notify the terminal device 50 by providing the functions described as the functions of the server 40 without using the server 40. In other words, it is also possible to adopt a configuration in which the information processing device 60 directly notifies the terminal device 50 held by the caregiver of the information detected by the information processing device 60. In this case, the notification can be made directly using, for example, Bluetooth (registered trademark) or ZigBee (registered trademark), or can be made using a communication network using, for example, Wi-Fi or LTE.

As described above, in this system, facial recognition processing can be performed based on information acquired by the information processing device 60, which corresponds to the so-called edge of the communication network, in a care facility or home, to monitor those in need of care. For example, the information processing device 60 is installed in a toilet where entry detection is desired. As a result, the information processing device 60 can detect entry using the human presence sensor 24, perform mask determination processing and facial recognition processing, and, if it is a care recipient, notify the terminal device 50 of information indicating the entry and the care recipient identified as a result of the facial recognition.

Here, in this system, the face model data used for face recognition is stored in a server on the cloud or the like, and the information processing device 60 acquires and retains the face model data in advance using a communication function. The information processing device 60 detects people using the human presence sensor 24, which serves as an entry/exit sensor, and detects entry and exit into the toilet. After the human presence sensor 24 detects a person, the information processing device 60 performs face recognition processing to identify the person who has entered the toilet by photographing the person who has entered the toilet using the second camera 25 for face recognition and comparing the photograph with the face model data stored in the device and extracting the person who most closely resembles the person. The photographed photograph data is not stored in consideration of privacy. The information processing device 60 can also be configured to acquire information such as the location of the toilet entered, the user's name, the time of entry and exit, and whether the user has sat on or left the toilet seat using the distance sensor 26a, the first camera 26b, the human presence sensor 24, and the second camera 25 for face recognition. In this case, the information processing device 60 can transmit the acquired information to the server 40 via the WiFi module 27e, and the information can be notified to the terminal device 50 held by the caregiver via the server 40. At this time, image data captured by the first camera 26b for excrement and the second camera 25 for face recognition can be avoided from being transmitted to the server 40.

With the above-mentioned configuration, this system can solve the two problems described as (1) and (2) in the following way.

First, problem (1) above can be solved as follows. In this embodiment, facial model data of the user and caregiver are pre-loaded into the information processing device 60. The information processing device 60 then compares the facial model data with the face captured by the facial recognition camera, and if the person who has entered the room is identified as a caregiver, that person is subsequently excluded from facial recognition if they are within the imaging range of the facial recognition camera. This allows only the user to be the subject of facial recognition processing, making it possible to improve the accuracy of person identification without reducing the processing power of facial recognition.

However, even if face model data of caregivers is incorporated, if one of the caregivers wears a mask, the recognition accuracy of face recognition deteriorates, as explained in problem (2) above. Therefore, problem (2) above is solved as follows. In this embodiment, by utilizing the fact that caregivers sometimes wear masks while users do not wear masks inside the facility, erroneous recognition of face recognition is reduced by adding mask detection processing to face recognition processing. Specifically, when the information processing device 60 detects that a person has entered the restroom, it performs face detection processing of the person and performs mask detection processing on the detected face. By performing mask detection processing, it is possible to exclude caregivers wearing masks from the target of face recognition processing, and face recognition can be performed only on faces not wearing masks.

Even if machine learning is used, determining whether a mask is being worn is faster than face matching, so it is possible to improve real-time performance even in edge devices equipped with space-saving and power-saving CPUs. In particular, identifying a user using face recognition in a bathroom requires coordination with a notification function to a caregiver, such as assisting the user in the bathroom or rushing to prevent accidents, so not only identification accuracy but also real-time performance is required, but this system enables real-time processing. In fact, the recognition process for a face wearing a mask is prone to misrecognition because it matches with little facial feature information other than the mask. In contrast, in this system, if the mask is detected during the mask detection process, it can be excluded from the face recognition process, so it is possible to prevent a decrease in the performance of the face recognition process and misrecognition of the user even if the user enters the room accompanied by a caregiver.

In this embodiment, by incorporating the face model data of the caregiver and mask detection processing as described above, even if the user and the caregiver are within the imaging range of the face recognition camera, face recognition is performed only on the user, so that it is possible to improve the accuracy of person identification without reducing the processing power of face recognition. In other words, even if the CPU 27a is a weak information processing device 60 with a space-saving and power-saving CPU, face recognition of the user is possible without reducing the processing performance and authentication accuracy. As a result, in this embodiment, it is possible to reduce the burden on the caregiver and provide thorough support to the user. In addition, in this embodiment, since the caregiver can be excluded from the face recognition, it is also possible to exclude cases such as toilet cleaning by the caregiver. In addition, for excretion management, which is a heavy burden on the caregiver, the burden of assistance work can be reduced by linking the toilet entry and exit management using face recognition processing and the excretion analysis function.

As described above, the present system enables real-time user identification with high accuracy and minimal performance degradation, reducing the burden on caregivers and providing thorough support to users. In other words, the present system achieves the effects described in embodiment 1, and in particular achieves the following effects:

The first benefit is that by registering the caregiver in advance, it is possible to reduce the number of times the caregiver is mistakenly detected as the user, and by adding a mask detection process, it is possible to obtain an even greater improvement in accuracy if the caregiver is wearing a mask.

The second advantage is that because mask detection processing is a lighter process than face recognition processing, even when multiple caregivers are assisting a single user, the user can be identified through face recognition without a significant loss of performance.

The third effect is that caregivers wearing masks are excluded from face recognition due to the mask detection process, so they can be excluded from using the toilet when cleaning the toilet, etc.

<Embodiment 3>
In the second embodiment, a toilet is described as an example of a room, but other types of rooms such as a bathroom can also be used. In this embodiment, an example of use in a bathroom will be described with reference to Fig. 10, but the various examples described in the first and second embodiments can be applied. Fig. 10 is a schematic diagram showing an example of the configuration of an information processing device according to a third embodiment.

As shown in FIG. 10, the information processing device according to this embodiment does not require the components related to excretion information in the information processing device 60 in embodiment 2. The information processing device according to this embodiment includes a control box 61 and a human detection/identification box 68 connected to the control box 61 via a cable K2. Note that the server 40 and the terminal device 50 are omitted in FIG. 10.

The human detection/identification box 68 can be installed inside the bathroom, or can be placed outside the bathroom in a position where it can detect a person entering through the door 71. Figure 10 shows an example in which the human detection/identification box 68 is installed at the back of the bathtub 70 in the bathroom, facing the door 71. The control box 61 can also be installed inside the bathroom, but it can also be installed outside the bathroom.

In this embodiment, too, monitoring of people requiring care in nursing facilities or homes can be performed by an information processing device that corresponds to the so-called edge of a communication network. In this embodiment, when the information processing device is installed in a location in a bathroom where entry detection and face recognition are desired, it is possible to detect entry by the human presence sensor 24 and perform face recognition processing based on image data acquired by the second camera 25.

Although a bathroom has been used as an example, the human detection/identification box 68 can be installed in rooms other than the toilet or bathroom, such as the facility entrance, bedroom, or dining room, to monitor the care recipient. In addition to nursing facilities, the system can also be used in hospitals, food companies, kindergartens, schools, and other facilities where many people wear masks and there is concern about leaking information over networks. For example, by installing a face recognition camera in a dining room, visitors to the dining room can be identified through face recognition processing, and meals can be prepared that are tailored to the user. Alternatively, by installing a face recognition camera at the entrance to the facility, it is possible to prevent the person to be authenticated from going out alone by notifying others or sounding an alarm on the device of the person to be authenticated. In any of the installation examples, operation can be performed by linking the identification of facility users with records of other tasks such as nursing care.

<Other embodiments>
[a]
In each embodiment, the functions of the information processing system and each device included in the system have been described, but each device is not limited to the configuration example shown in the figure, and it is sufficient for each device to be able to realize these functions, and additional functions may also be provided.

[b]
Each of the devices described in the first to third embodiments may have the following hardware configuration. Fig. 11 is a diagram showing an example of the hardware configuration of the device. The same applies to the other embodiment [a] described above.

The device 100 shown in FIG. 11 may have a processor 101, a memory 102, and a communication interface (I/F) 103. The processor 101 may be, for example, a microprocessor, a microprocessor unit (MPU), or a CPU. The processor 101 may include multiple processors. The memory 102 is, for example, configured from a combination of a volatile memory and a non-volatile memory. The functions of each device described in embodiments 1 to 3 are realized by the processor 101 reading and executing a program stored in the memory 102. At this time, information can be sent and received with other devices via the communication interface 103 or an input/output interface (not shown).

The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

Note that this disclosure is not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, this disclosure can be implemented by combining the respective embodiments as appropriate.

Some or all of the above embodiments may be described as follows, but are not limited to the following:

(Appendix 1)
A sensor that detects a person entering the room;
an imaging device that captures an image of a person entering the room and obtains image data when the sensor detects the person entering the room;
a determination unit that determines whether or not a person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and thereby provisionally determines whether or not the person who has entered the room is a person to be authenticated as a user of the room;
a face authentication unit that executes face authentication processing based on face image data of the person to be authenticated in the image data when the result of the provisional determination by the determination unit indicates that the person is the person to be authenticated;
An output unit that outputs a result of the face recognition processing to a notification destination;
Equipped with
The face authentication unit executes the face authentication process by comparing the face image data with feature data indicating facial features pre-stored for each of the authentication subjects and for each of the attendants who may accompany the authentication subjects and enter the room.
Information processing device.
(Appendix 2)
When the attendant is authenticated as a result of the face authentication processing, the output unit outputs the result of the face authentication processing excluding the result of the attendant.
2. The information processing device according to claim 1.
(Appendix 3)
The feature data is generated from face image data acquired by a terminal device and stored in the information processing device.
3. The information processing device according to claim 1 or 2.
(Appendix 4)
the room is a toilet, and the imaging device is installed inside the toilet;
The information processing device includes an analysis unit that analyzes excretion information indicating the content of excretion based on imaging data captured by another imaging device that is installed so as to include an excretion range in the toilet bowl of the toilet within an imaging range,
the analysis unit performs an analysis of the excretion information when the person to be authenticated is authenticated as a result of the face authentication process;
The output unit outputs the analysis result of the analysis unit to a notification destination.
4. The information processing device according to claim 1 .
(Appendix 5)
The sensor detects when someone enters the room,
When the sensor detects the entry of a person into the room, an imaging device captures an image of the person who entered the room to obtain image data;
determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated;
If the result of the provisional determination is that the person is the person to be authenticated, a face authentication process is executed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter a room, based on the face image data of the person to be authenticated in the image data;
outputting a result of the face recognition processing to a notification destination;
Information processing methods.
(Appendix 6)
When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant.
6. The information processing method according to claim 5.
(Appendix 7)
The feature data is data generated from face image data acquired by a terminal device and stored.
7. The information processing method according to claim 5 or 6.
(Appendix 8)
the room is a toilet, and the imaging device is installed inside the toilet;
If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on imaging data captured by another imaging device installed so as to include in its imaging range the excretion range in the toilet bowl of the toilet;
outputting the analysis result of the analysis process to a notification destination;
8. An information processing method according to any one of appendix 5 to 7.
(Appendix 9)
On the computer,
The sensor detects when someone enters the room, and inputs the result.
When the sensor detects the entry of a person into the room, an image capturing device captures an image of the person who entered the room to obtain image data;
determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated;
If the result of the provisional determination is that the person is the person to be authenticated, a face authentication process is executed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter a room, based on the face image data of the person to be authenticated in the image data;
outputting a result of the face recognition processing to a notification destination;
A program for executing information processing.
(Appendix 10)
When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant.
10. The program according to claim 9.
(Appendix 11)
The feature data is generated from face image data acquired by a terminal device and stored in the computer.
11. The program according to claim 9 or 10.
(Appendix 12)
the room is a toilet, and the imaging device is installed inside the toilet;
The information processing includes:
If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on imaging data captured by another imaging device installed so as to include in its imaging range the excretion range in the toilet bowl of the toilet;
outputting the analysis result of the analysis process to a notification destination;
The program according to any one of claims 9 to 11.

1, 60 Information processing device 1a Sensor 1b Imaging device 1c Determination unit 1d Face authentication unit 1e Output unit 10 Toilet bowl 10a Side surface 10b Planar portion 10c Edge portion 11 Toilet seat 11a Toilet seat body 11b Toilet seat cover 24 Human presence sensor 25 Imaging device (second camera)
26 Information collecting unit 26a Distance sensor 26b First camera 27a CPU
27b connector 27c, 27d USB I/F
27e WiFi module 40 Server 41 Control unit 42 Storage unit 50 Terminal device 51 Display screen 52, 53, 54 Notification information 61 Storage housing (control box)
62 Bridge section
63 Inner housing (sensor box)
67 Control unit 68 Separate housing (human detection/identification box)
69 Outer housing 70 Bathtub 71 Door 100 Device 101 Processor 102 Memory 103 Communication interface

Claims (7)

A sensor that detects a person entering a room that is a toilet ;
an imaging device that is installed in the toilet room and captures an image of a person entering the room to obtain image data when the sensor detects the person entering the room;
a determination unit that determines whether or not a person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and thereby provisionally determines whether or not the person who has entered the room is a person to be authenticated as a user of the room;
a face authentication unit that executes face authentication processing based on face image data of the person to be authenticated in the image data only when the person to be authenticated is determined to be the person to be authenticated as a result of the provisional determination by the determination unit;
An output unit that outputs a result of the face recognition processing to a notification destination;
an analysis unit that analyzes excretion information indicating the content of the excretion based on imaging data captured by another imaging device that is installed so as to include the excretion area in the toilet bowl in an imaging range;
Equipped with
the face authentication unit performs the face authentication process by comparing the face image data with feature data indicating facial features stored in advance for each of the authentication subjects and for each of the attendants who may accompany the authentication subjects and enter the room ;
the analysis unit performs an analysis of the excretion information when the person to be authenticated is authenticated as a result of the face authentication process;
The output unit outputs the analysis result of the analysis unit to a notification destination.
Information processing device. When the attendant is authenticated as a result of the face authentication processing, the output unit outputs the result of the face authentication processing excluding the result of the attendant.
The information processing device according to claim 1 . The feature data is generated from face image data acquired by a terminal device and stored in the information processing device.
3. The information processing device according to claim 1 or 2. The sensor detects when someone enters the room, which is the toilet .
When the sensor detects the entry of a person into the toilet, an imaging device installed in the toilet captures an image of the person entering the toilet and obtains image data;
determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated;
only when the result of the provisional determination indicates that the person is the person to be authenticated, a face authentication process is executed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter a room,
outputting a result of the face recognition process to a notification destination;
If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include an excretion range in the toilet bowl of the toilet in an image capture range,
outputting the analysis result of the analysis process to a notification destination;
Information processing methods. When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant.
The information processing method according to claim 4 . On the computer,
The sensor detects when a person enters a room, which is a toilet , and inputs the result.
When the sensor detects the entry of a person into the toilet room, an image capturing device installed in the toilet room captures an image of the person who entered the toilet room to obtain image data;
determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated;
only when the result of the provisional determination indicates that the person is the person to be authenticated, a face authentication process is executed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter a room,
outputting a result of the face recognition process to a notification destination;
If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on imaging data captured by another imaging device installed so as to include in its imaging range the excretion range in the toilet bowl of the toilet;
outputting the analysis result of the analysis process to a notification destination;
A program for executing information processing. When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant.
The program according to claim 6 .

Images