JP7757681B2 - Information processing method, information processing program, and information processing device - Google Patents

Description

The present invention relates to an information processing method for outputting abnormality information.

Elderly care monitoring systems have been proposed. For example, Patent Document 1 proposes a medical system that can grasp the living conditions of those being monitored, such as elderly people, at an early stage, take appropriate measures, prevent deterioration, encourage independence, maintain quality of life, and reduce costs for service providers.

Japanese Patent Application Laid-Open No. 2020-201851

However, conventional systems such as the one proposed in Patent Document 1 require the installation of multiple sensors in the elderly person's home. This requires time and effort for installation and maintenance. The present invention was made in light of this situation. Its purpose is to provide an information processing method that detects abnormalities in a person being monitored using an easily installed terminal and outputs abnormality information.

In one aspect of the present application, an information processing method includes a computer executing a process in which facial image data of a user is acquired through an interactive robot, the acquired facial image data is input into a first learning model trained to output a first abnormal state related to the user's abnormality when the facial image data is input, and first abnormality information is acquired by inputting the acquired facial image data into a first learning model trained to output a first abnormality state related to the user's abnormality when the facial image data is input, dialogue data exchanged between the user and the interactive robot is acquired, the acquired dialogue data is input into a second learning model trained to output second abnormality information related to the user's abnormality when the dialogue data is input, and second abnormality information is acquired, and if both the first abnormality information and the second abnormality information are output within a predetermined time, the computer outputs the abnormality information to another terminal device.

In one aspect of the present application, an easily installed terminal can detect abnormalities in the person being monitored and output abnormality information.

FIG. 1 is an explanatory diagram illustrating an example of the configuration of a monitoring system. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a management server. FIG. 2 is a block diagram showing an example of the hardware configuration of a monitoring terminal. FIG. 2 is a block diagram showing an example of the hardware configuration of a family terminal. FIG. 2 is a block diagram showing an example of the hardware configuration of an expert terminal. FIG. 10 is an explanatory diagram showing an example of a subject DB. FIG. 2 is an explanatory diagram illustrating an example of a scenario DB. FIG. 10 is an explanatory diagram illustrating an example of a determination result DB. FIG. 10 is an explanatory diagram illustrating an example of a notification destination DB. FIG. 10 is an explanatory diagram illustrating an example of a diagnostic model. FIG. 10 is an explanatory diagram illustrating an example of an analytical model. 10 is a flowchart illustrating an example of a procedure for interactive processing. 10 is a flowchart illustrating an example of a procedure for interactive processing. 10 is a flowchart illustrating an example of a procedure for an abnormality handling process. 10 is a flowchart illustrating an example of a procedure for remote handling processing. FIG. 2 is a block diagram illustrating an example of a functional configuration of a management server.

(Embodiment 1)
An embodiment will be described below with reference to the drawings. Fig. 1 is an explanatory diagram showing an example of the configuration of a monitoring system. The monitoring system 100 includes a management server 1, a monitoring terminal 2, a family terminal 3, and an expert terminal 4. The management server 1, the monitoring terminal 2, the family terminal 3, and the expert terminal 4 are communicably connected via a network N. Although Fig. 1 shows one monitoring terminal 2, one family terminal 3, and one expert terminal 4, two or more of each may be used.

The management server 1 is composed of a server computer, workstation, PC (Personal Computer), etc. The management server 1 analyzes the images and audio of the person being monitored collected by the monitoring terminal 2 and determines the health condition of the person being monitored. Depending on the determination result, the management server 1 notifies the family terminal 3 and the expert terminal 4. The management server 1 may also be composed of a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. The functions of the management server 1 may also be realized by a cloud service. The person being monitored is assumed to be an elderly person living alone. In the following explanation, the "person being monitored" may also be simply referred to as the "target person."

The monitoring terminal 2 is composed of a pet robot, a communication robot, a mobile robot, a smart speaker with a camera, etc. The monitoring terminal 2 is installed in the home of the person being monitored. The monitoring terminal 2 engages in voice dialogue with the person being monitored. The monitoring terminal 2 also acquires facial images of the person being monitored.

The family terminal 3 is composed of a smartphone, tablet computer, notebook PC, etc. The family terminal 3 is used by family members who live separately from the person being monitored. The family terminal 3 communicates with the management server 1 and displays the health status of the person being monitored.

The expert terminal 4 is composed of a smartphone, tablet computer, laptop PC, etc. The expert terminal 4 is used by experts who work with the person being monitored, such as doctors, nurses, care managers, and caregivers. If the management server 1 determines that an abnormality has occurred in the person being monitored, the expert terminal 4 receives a notification of this from the management server 1 and prompts the expert to take action. The expert terminal 4 used by a doctor is an example of a doctor terminal.

Figure 2 is a block diagram showing an example hardware configuration of a management server. The management server 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 15, and a reading unit 16. The control unit 11, the main memory unit 12, the auxiliary memory unit 13, the communication unit 15, and the reading unit 16 are connected by a bus B.

The control unit 11 has one or more central processing units (CPUs), microprocessing units (MPUs), graphics processing units (GPUs), and other processing devices. The control unit 11 reads and executes the control program 1P (information processing program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, and the like related to the management server 1 and realizing various functional units.

The main memory unit 12 is a static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, etc. The main memory unit 12 mainly temporarily stores data required by the control unit 11 to perform arithmetic processing.

The auxiliary memory unit 13 is a hard disk or SSD (Solid State Drive), etc., and stores the control program 1P and various DBs (Databases) required for the control unit 11 to execute processing. The auxiliary memory unit 13 stores a subject DB 131, a scenario DB 132, a judgment result DB 133, and a notification destination DB 134. The auxiliary memory unit 13 also stores a diagnostic model 141 and an analytical model 142. The auxiliary memory unit 13 may be a storage device separate from and externally connected to the management server 1. The various DBs, etc. stored in the auxiliary memory unit 13 may also be stored in a database server or cloud storage other than the management server 1.

The communication unit 15 communicates with the monitoring terminal 2, family terminal 3, expert terminal 4, etc. via the network N. In addition, the control unit 11 may use the communication unit 15 to download the control program 1P from another computer via the network N, etc., and store it in the auxiliary memory unit 13.

The reading unit 16 reads portable storage media 1a, including CD (Compact Disc)-ROM and DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the auxiliary storage unit 13. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.

Figure 3 is a block diagram showing an example of the hardware configuration of a monitoring terminal. The monitoring terminal 2 includes a control unit 21, a main memory unit 22, an auxiliary memory unit 23, a communication unit 24, an input unit 25, a display unit 26, an imaging unit 27, a detection unit 28, and an audio input/output unit 29. Each component is connected by a bus B.

The control unit 21 has one or more processing units such as a CPU, MPU, GPU, etc. The control unit 21 provides various functions by reading and executing a control program 2P (program, program product) stored in the auxiliary storage unit 23.

The main memory unit 22 is an SRAM, DRAM, flash memory, etc. The main memory unit 22 mainly temporarily stores data required by the control unit 21 to perform arithmetic processing.

The auxiliary memory unit 23 is a hard disk or SSD, etc., and stores various data necessary for the control unit 21 to execute processing. The auxiliary memory unit 23 may also be an external storage device connected to the monitoring terminal 2. The various DBs, etc. stored in the auxiliary memory unit 23 may also be stored on a database server or cloud storage.

The communication unit 24 communicates with the management server 1 via the network N. The control unit 21 may also use the communication unit 24 to download the control program 2P from another computer via the network N, etc., and store it in the auxiliary storage unit 23.

The input unit 25 is a keyboard and a mouse. The display unit 26 includes an LCD panel or the like. The display unit 26 displays messages output by the management server 1. The input unit 25 and the display unit 26 may also be integrated to form a touch panel display. Note that the input unit 25 and the display unit 26 are not essential components of the monitoring terminal 2. If the monitoring terminal 2 is configured as a smart speaker, the input unit 25 and the display unit 26 may not be provided.

The imaging unit 27 includes an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and acquires image data such as a facial image of the subject by photoelectrically converting an optical signal input via the image sensor. The imaging unit 27 may be equipped with a zoom lens capable of optical zoom. The imaging unit 27 may also be equipped with a pan/tilt mechanism.

The detection unit 28 detects the environment where the monitoring terminal 2 is installed. The detection unit 28 also detects the presence of a target person within a predetermined range from the monitoring terminal 2. The detection unit 28 is equipped with, for example, an illuminance sensor, and detects the brightness of the location where it is installed. The detection unit 28 is equipped with, for example, an infrared sensor, and detects the presence of a person (presumably the target person) within a predetermined range. Note that the detection unit 28 is not an essential component of the monitoring terminal 2, and may not be included.

The audio input/output unit 29 includes a microphone and a speaker. The audio input/output unit 29 acquires the target person's voice and other information via the microphone. The audio input/output unit 29 outputs an audio message to the target person from the speaker.

Figure 4 is a block diagram showing an example of the hardware configuration of a family terminal. The family terminal 3 includes a control unit 31, a main memory unit 32, an auxiliary memory unit 33, a communication unit 34, an input unit 35, and a display unit 36. Each component is connected by a bus B.

The control unit 31 has one or more processing units such as a CPU, MPU, GPU, etc. The control unit 31 provides various functions by reading and executing the control program 3P stored in the auxiliary storage unit 33.

The main memory unit 32 is an SRAM, DRAM, flash memory, etc. The main memory unit 32 mainly temporarily stores data required by the control unit 31 to perform arithmetic processing.

The auxiliary storage unit 33 is a hard disk or SSD, etc., and stores various data necessary for the control unit 31 to execute processing. The various DBs, etc. stored in the auxiliary storage unit 33 may be stored on a database server or cloud storage.

The communication unit 34 communicates with the management server 1 via the network N. The control unit 31 may also use the communication unit 34 to download the control program 3P from another computer via the network N, etc., and store it in the auxiliary memory unit 33.

The input unit 35 is a keyboard and mouse. The display unit 36 includes an LCD panel or the like. The display unit 36 displays messages output by the management server 1. The input unit 35 and display unit 36 may also be integrated to form a touch panel display.

Furthermore, the family terminal 3 may be equipped with an imaging unit and an audio input/output unit, just like the monitoring terminal 2. If the family terminal 3 is configured as a smartphone, the family terminal 3 will be equipped with an imaging unit and an audio input/output unit.

Figure 5 is a block diagram showing an example of the hardware configuration of an expert terminal. The expert terminal 4 includes a control unit 41, a main memory unit 42, an auxiliary memory unit 43, a communication unit 44, an input unit 45, a display unit 46, an imaging unit 47, and an audio input/output unit 48. Each component is connected via a bus B. The control unit 41, the main memory unit 42, the auxiliary memory unit 43, the communication unit 44, the input unit 45, the display unit 46, the imaging unit 47, and the audio input/output unit 48 are similar to the control unit 21, the main memory unit 22, the auxiliary memory unit 23, the communication unit 24, the input unit 25, the display unit 26, the imaging unit 27, and the audio input/output unit 29 of the monitoring terminal 2, and therefore their description will be omitted.

Figure 6 is an explanatory diagram showing an example of a subject DB. Subject DB 131 stores information about subjects. Subject DB 131 includes a subject ID column, a name column, an age column, a gender column, a height column, a weight column, an activity time column, and a health condition column. The subject ID column stores a subject ID that uniquely identifies a subject. The name column stores the name of the subject. The age column stores the age of the subject. The gender column stores the gender of the subject. The height column stores the height of the subject. The weight column stores the weight of the subject. The activity time column stores the activity time of the subject. The activity time indicates the time from waking up to going to bed. The health condition column stores the health condition of the subject. It is desirable to update the values of the height column, weight column, activity time column, and health condition column regularly or irregularly in accordance with fluctuations.

Figure 7 is an explanatory diagram showing an example of a scenario DB. Scenario DB 132 stores scenarios for the monitoring terminal 2 to converse with the subject. Scenario DB 132 includes an ID sequence, an utterance sequence, a response sequence, and a transition ID sequence. The ID sequence stores an ID that identifies each utterance uttered by the monitoring terminal 2. The utterance sequence stores the content of the utterance. The response sequence stores the content of the subject's response to the utterance. Since multiple responses are expected from the subject to each utterance, there is a one-to-many relationship between utterances and responses. The transition ID sequence stores the ID of the next utterance uttered by the monitoring terminal 2 after receiving a response. In the transition ID sequence, "None" indicates the end of the dialogue. A value indicating the end of the dialogue may be other than "None".

Figure 8 is an explanatory diagram showing an example of a judgment result DB. The judgment result DB 133 stores the judgment results of the subject's current condition obtained through dialogue, etc. The judgment result DB 133 includes a subject ID column, a date and time column, a health column, and a dialogue column. The subject ID column stores the subject ID. The date and time column stores the date and time when the judgment was made. The health column stores the judgment results of the subject's health condition. The dialogue column stores the classification results of the dialogue content. In Figure 8, the judgment results of the health condition are shown as numbers, but the possible values can also be divided into multiple intervals and a word indicating the condition can be assigned to each interval. Furthermore, the classification results of the dialogue content can be strings indicating the classification rather than numbers.

Figure 9 is an explanatory diagram showing an example of a notification destination DB. The notification destination DB 134 stores information on who to notify when an abnormal situation occurs for a target person. Notification destinations include family members living separately, a primary care doctor, and the responsible care manager. The notification destination DB 134 includes a target person ID column, a number column, a name column, a relationship column, an email address column, and a phone number column. The target person ID column stores the target person ID. The number column stores the order of notifications, priority, etc. The name column stores the name of the person to be notified. The relationship column stores the relationship between the notification destination and the target person. The email address column stores the email address of the notification destination. The phone number column stores the phone number of the notification destination.

Figure 10 is an explanatory diagram showing an example of a diagnostic model. When facial image data of a subject is input, diagnostic model 141 (first learning model) outputs an evaluation score indicating the subject's level of health. The evaluation score is, for example, a value between 0 and 1, with the closer to 1 the healthier the score and the closer to 0 the unhealthier (abnormal) the score. In the following explanation, diagnostic model 141 is described as a CNN (Convolutional Neural Network), but diagnostic model 141 is not limited to CNN and may be a model constructed using other learning algorithms, such as neural networks other than CNN, Bayesian networks, or decision trees. The evaluation score is an example of first abnormality information.

The management server 1 inputs facial images contained in the training data into the input layer, undergoes calculation processing in the intermediate layer, and obtains evaluation scores from the output layer. The management server 1 compares the evaluation scores output from the output layer with the labels contained in the training data, i.e., the correct values, and optimizes the parameters used in the calculation processing in the intermediate layer so that the output values of the output nodes approach the correct values. These parameters include, for example, the weights (coupling coefficients) between neurons and the coefficients of the activation functions used in each neuron. There are no particular limitations on the method for optimizing the parameters, but for example, the management server 1 optimizes various parameters using the backpropagation method. The management server 1 performs the above learning process using all of the training data and generates a learned diagnostic model 141.

Figure 11 is an explanatory diagram showing an example of an analysis model. When dialogue data between the subject and the monitoring terminal 2 is input, the analysis model 142 (second learning model) determines which of multiple categories the content of the dialogue falls into and outputs the determined category. The analysis model 142 includes a speech recognition model 1421 and a classification model 1422. The speech recognition model 1421 is configured, for example, by a GMM-HMM, a DNN-HMM, or an end-to-end model. GMM is a Gaussian Mixture Model (a mixed normal distribution model). HMM is a Hidden Markov Model (a hidden Markov model). DNN is a Deep Neural Network (a deep neural network). The classification model 1422 is configured, for example, by a recurrent neural network (RNN), a long short-term memory (LSTM), or a naive Bayes classifier.

When voice data is input, the voice recognition model 1421 performs voice recognition and outputs text. When text is input, the classification model 1422 classifies the content of the text. Examples of classifications include normal, slurred speech, incomplete conversation, angry, and forgetful. The classification model 1422 outputs a classification value (classification information) indicating the classification. The classification value is an example of second abnormality information. Note that the classification may also include olfactory abnormalities such as not being able to smell anything. To detect olfactory abnormalities, the monitoring terminal 2 asks questions in the dialogue, such as "Can you smell coffee?" or "Can you smell miso soup?" Alternatively, the monitoring terminal 2 may be equipped with an aroma spray that periodically emits a scent and asks whether the user can smell it.

Training data for classification model 1422 is prepared by associating dialogue text with content classification values. The dialogue text that constitutes the training data is input into classification model 1422, and parameters are adjusted so that the classification value output by classification model 1422 matches the classification value (= correct value) that constitutes the training data.

Next, the information processing performed by the monitoring system 100 will be described. Figures 12 and 13 are flowcharts showing an example of the procedure for dialogue processing. The control unit 21 of the monitoring terminal 2 determines whether or not a target person has been detected (step S1). The target person is detected, for example, by the detection unit 28. If the control unit 21 determines that a target person has not been detected (NO in step S1), it repeats step S1. If the control unit 21 determines that a target person has been detected (YES in step S1), it determines whether or not to call out to the target person (step S2). For example, if the control unit 21 detects a target person for the first time in a certain day, it determines that it will call out to the target person. If it is within a time period that has been set in advance for dialogue, the control unit 21 determines that it will call out to the target person. However, if a predetermined time (e.g., three hours) has not passed since the last time it called out to the target person, the control unit 21 may determine that it will not call out to the target person. If the control unit 21 determines that it will not call out to the target person (NO in step S2), it ends the dialogue processing. If the control unit 21 determines that a call should be made (YES in step S2), it selects a message to be output as a call (step S3). The selected message may be, for example, a greeting, and an appropriate message is selected depending on the time of day and the season. The message is stored in advance in the auxiliary storage unit 23. The control unit 21 outputs the message as audio from the audio input/output unit 29 (step S4). The control unit 21 captures an image of the subject's face using the imaging unit 27 (step S5). If the control unit 21 determines that capturing an image of the subject's face is difficult, for example, because the subject is far from the monitoring terminal 2, it outputs a message from the audio input/output unit 29 encouraging the subject to approach the monitoring terminal 2 and turn their face toward it. This message is also stored in advance in the auxiliary storage unit 23. The control unit 21 transmits a facial image of the subject to the management server 1 (step S6). The control unit 11 of the management server 1 receives the facial image (step S7). The control unit 11 inputs the facial image into the diagnostic model 141 to obtain an evaluation score, and stores the facial image and the evaluation score (step S8). The control unit 11 selects a message based on the evaluation score (step S9). The control unit 11 transmits the message to the watching terminal 2 (step S10). The control unit 21 of the watching terminal 2 receives the message (step S11). The control unit 21 outputs the message as voice from the voice input/output unit 29 (step S12). The control unit 21 collects the target person's response to the message (step S13). For example, the voice input/output unit 29 collects the target person's response voice. The control unit 21 determines whether the target person has responded (step S14). For example, the control unit 21 analyzes the waveform of the collected response voice to determine whether the target person has responded. If the control unit 21 determines that the target person has responded (YES in step S14), it transmits the collected response voice to the management server 1 (step S15). The control unit 11 of the management server 1 receives the response voice (step S16). The control unit 11 inputs the response voice into the analysis model 142 to analyze the content of the dialogue and stores the resulting classification value (step S17). The classification value is stored in the judgment result DB 133. The control unit 11 determines whether the subject is normal or not based on the classification value (step S18). If the control unit 11 determines that the subject is normal (YES in step S18), it selects a message to be output next (step S19). The control unit 11 transmits the message to the watching terminal 2 (step S20). The control unit 21 of the watching terminal 2 receives the message (step S21). The control unit 21 outputs the message as voice from the voice input/output unit 29 (step S22). The control unit 21 determines whether to end the processing (step S23). If the output message is a message to end the dialogue, it determines that the processing should be ended. If the control unit 21 determines not to end the processing (NO in step S23), it returns the processing to step S13. If the control unit 21 determines that the processing should be ended (YES in step S23), it ends the processing. If the control unit 11 of the management server 1 determines that the subject is not normal (NO in step S18), it performs an abnormality handling process (step S26) and terminates the process. The abnormality handling process will be described later. If the control unit 21 of the monitoring terminal 2 determines that the subject is not responding (NO in step S14), it transmits a message indicating that there is no response to the management server 1 (step S24). The control unit 11 of the management server 1 receives the message indicating that there is no response (step S25). The control unit 11 performs an abnormality handling process (step S26) and terminates the process. It is assumed that the dialogue process will be repeated several times a day. If the imaging unit 27 is equipped with a zoom lens or pan/tilt mechanism, it may capture images other than facial images. For example, it may follow the subject as they move around the room and capture their body movements as they move. It may also capture basic daily activities such as eating and changing clothes. By capturing more images other than facial images, more data is available for analyzing the subject's condition, improving the accuracy of abnormality detection and increasing the types of abnormalities that can be detected.

Figure 14 is a flowchart showing an example of the abnormality handling process. The control unit 11 of the management server 1 transmits an operational instruction to the monitoring terminal 2 (step S41). For example, if it is determined that an abnormality may have occurred in the target person, the control unit 11 instructs the monitoring terminal 2 to call out to the target person again. For example, the control unit 11 instructs the monitoring terminal 2 to output messages such as "Are you OK?", "Can you hear me?", and "Can you speak?" as the content of the message. Furthermore, to determine whether the target person did not respond to the message due to an abnormality, the control unit 11 instructs the monitoring terminal 2 to perform an analysis using sensing. The control unit 21 of the monitoring terminal 2 receives the instruction (step S42). The control unit 21 determines whether the instruction is a call (step S43). If the control unit 21 determines that the instruction is not a call (NO in step S43), it collects environmental information (step S44). The control unit 21 performs sensing using the detection unit 28, collecting information on the room brightness and the amount of infrared light using the infrared sensor. The control unit 21 performs analysis based on the collected information (step S45). Based on the analysis results, the control unit 21 determines whether the subject is normal (step S46). For example, if the subject is expected to be sleeping during a certain time period, the room is dark, the infrared center predicts that the subject is not within a certain range, and the installation location is set to a location other than the bedroom, the control unit 21 determines that the subject is normal. Also, if the subject is expected to be sleeping during a certain time period, the room is bright, the infrared center predicts that the subject is within a certain range, and the installation location is set to the living room, the control unit 21 determines that the subject is abnormal. The installation location of the monitoring terminal 2 is stored in advance in the auxiliary storage unit 13. If the control unit 21 determines that the subject is abnormal (NO in step S46), the control unit 21 changes the mode of the monitoring terminal 2 (step S47). For example, the control unit 21 transitions from normal mode to abnormal mode. For example, in abnormal mode, video calls are enabled by external control. The control unit 21 notifies the management server 1 that the subject is abnormal (step S48). After receiving the notification, the control unit 11 of the management server 1 notifies the family terminal 3 or the expert terminal 4 (step S59). The notification may include details of the detected abnormality and an image captured by the monitoring terminal 2. This notification is an example of abnormality information. The control unit 11 determines the notification destination by referencing the notification destination DB 134. The control unit 11 returns the process to the caller. If the control unit 21 determines that the target person is normal (YES in step S46), it transmits a message indicating that the target person is normal to the management server 1 (step S57). The control unit 11 of the management server 1 receives the message indicating that the target person is normal (step S58). The control unit 11 returns the process to the caller. If the control unit 21 determines that the instruction is a call (YES in step S43), it outputs a call message and then collects the target person's response to the message (step S49). For example, the voice input/output unit 29 collects the target person's response voice. The control unit 21 determines whether there is a response from the target person (step S50). If the control unit 21 determines that there is no response from the target person (NO in step S50), the control unit 21 proceeds to step S44.

If the control unit 21 determines that there has been a response from the subject (YES in step S50), it transmits the collected response voice to the management server 1 (step S51). The control unit 11 of the management server 1 receives the response voice (step S52). The control unit 11 inputs the response voice into the analysis model 142 to analyze the dialogue content and stores the resulting classification value (step S53). The control unit 11 determines whether the subject is normal or not based on the classification value (step S54). If the control unit 11 determines that the subject is normal (YES in step S54), the control unit 11 returns the process to the caller. If the control unit 11 determines that the subject is not normal (NO in step S54), it transmits a mode change instruction to the watching terminal 2 (step S55). The control unit 21 of the watching terminal 2 receives the mode change instruction (step S56). The control unit 21 proceeds to step S47.

Figure 15 is a flowchart showing an example of the procedure for remote response processing. The remote response processing is triggered when the expert terminal 4 receives a notification in step 59 of the abnormality handling processing shown in Figure 14. The control unit 41 of the expert terminal 4 receives the notification from the management server 1 and displays it on the display unit 46 (step S71). After confirming the notification, the expert instructs the expert terminal 4 to connect to the monitoring terminal 2. The control unit 41 sends a connection request to the monitoring terminal 2 (step S72). The connection request may include the expert's ID and password. The control unit 21 of the monitoring terminal 2 receives the connection request (step S73). The control unit 21 determines whether to permit the connection (step S74). For example, if the mode is abnormal mode, the control unit 21 determines to permit the connection. If the connection request includes an ID and password, the control unit 21 performs authentication and determines whether to permit the connection. The ID and password combination for permitting the connection is stored in advance in the auxiliary memory unit 23. If the control unit 21 determines not to permit the connection (NO in step S74), it transmits a connection denial to the expert terminal 4 (step S75). The control unit 41 of the expert terminal 4 receives the connection denial (step S76). The remote support process ends. If the control unit 21 determines to permit the connection (YES in step S74), it transmits a connection permission to the expert terminal 4 (step S77). The control unit 41 of the expert terminal 4 receives the connection permission (step S78). The expert terminal 4 and the monitoring terminal 2 transmit and receive video and audio (steps S79 and S80) to provide a video call. The expert grasps the condition of the subject through the video call. The control unit 41 of the expert terminal 4 determines whether to end (step S81). If the control unit 41 determines not to end (NO in step S81), the process returns to step S79. If the control unit 41 determines to end (YES in step S81), it transmits an end command to the monitoring terminal 2 (step S82). The control unit 41 determines that the video call has ended, for example, when the expert instructs the expert to end the video call. The control unit 21 of the monitoring terminal 2 receives the end command (step S83). The control unit 21 disconnects communication with the expert terminal 4 (step S84). The control unit 41 of the expert terminal 4 detects the disconnection of communication (step S85). The control unit 41 transmits a message indicating that the video call has ended to the management server 1 (step S86). At this time, it is desirable to transmit the subject's condition and diagnosis results created by the expert. The control unit 11 of the management server 1 receives the message indicating that the video call has ended (step S87). The control unit 11 stores the fact that the video call was held as history in the auxiliary storage unit 13 (step S88). The control unit 11 transmits a message indicating receipt to the expert terminal 4 (step S89). The control unit 41 of the expert terminal 4 receives the message indicating receipt (step S90) and ends the process. Here, a case where the expert terminal 4 executes the remote response process has been described, but the same remote response process is performed when the family terminal 3 receives a notification. Online consultations are also possible when a doctor uses the specialist terminal 4 to video chat with the patient.

In the above-described embodiment, monitoring is possible by installing a monitoring terminal 2 in the home of the person being monitored. Abnormalities in the person being monitored can be detected based on the person's facial expressions and responses during conversations. If an abnormality is detected, a notification is sent to the family terminal 3 or the expert terminal 4. The family member or expert who receives the notification can check on the person being monitored via a video call using the family terminal 3 or the expert terminal 4 and the monitoring terminal 2. If necessary, emergency measures, such as requesting an ambulance, can be taken. Furthermore, because the person being monitored can interact with the monitoring terminal 2, a soothing effect similar to interacting with a pet can be expected. Even if the person being monitored lives alone, this can help reduce feelings of loneliness and may be effective in preventing dementia and mental illness. Furthermore, by configuring the monitoring terminal 2 with a pet robot or other device with a tracking function, it is possible to capture the movements of the person being monitored as they move. Additionally, during conversations, the monitoring terminal 2 can react by combining speech and movement, providing an even more soothing effect to the person being monitored compared to voice-only communication.

Figure 16 is a block diagram showing an example of the functional configuration of the management server. The management server 1 includes a first data acquisition unit 11a, a first abnormality acquisition unit 11b, a second data acquisition unit 11c, a second abnormality acquisition unit 11d, and an abnormality output unit 11e. When the control unit 11 executes the control program 1P, the management server 1 operates as follows:

The first data acquisition unit 11a acquires facial image data of the user obtained through the interactive robot. The first abnormality acquisition unit 11b acquires first abnormality information by inputting the acquired facial image data into a first learning model that has been trained to output a first abnormal state related to the user's abnormality when the facial image data is input. The second data acquisition unit 11c acquires dialogue data exchanged with the user through the interactive robot. The second abnormality acquisition unit 11d acquires second abnormality information by inputting the acquired dialogue data into a second learning model that has been trained to output second abnormality information related to the user's abnormality when the dialogue data is input. The abnormality output unit 11e outputs the abnormality information to another terminal device if both the first abnormality information and the second abnormality information are output within a predetermined time.

The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed.
The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

REFERENCE SIGNS LIST 100 Monitoring system 1 Management server 11 Control unit 11a First data acquisition unit 11b First abnormality acquisition unit 11c Second data acquisition unit 11d Second abnormality acquisition unit 11e Abnormality output unit 12 Main memory unit 13 Auxiliary memory unit 131 Target person DB
132 Scenario DB
133 Judgment result DB
134 Notification destination DB
141 Diagnostic model 142 Analysis model 1421 Speech recognition model 1422 Classification model 15 Communication unit 16 Reading unit 1P Control program 1a Portable storage medium 1b Semiconductor memory 2 Monitoring terminal 2P Control program 3 Family terminal 3P Control program 4 Expert terminal B Bus N Network

Claims (8)

Acquire face image data of the user acquired through the interactive robot;
inputting the acquired facial image data into a first learning model that has been trained to output first abnormality information regarding an abnormality of a user when the facial image data is input, and acquiring first abnormality information;
making a speech in response to the first abnormality information through the interactive robot;
Acquire conversation data between the user and the interactive robot through the interactive robot;
inputting the acquired dialogue data into a second learning model that has been trained to output second anomaly information related to an anomaly of the user when dialogue data is input, and acquiring second anomaly information;
anomaly information is output to a terminal device in response to the second anomaly information ,
The second abnormality information has classification information including slurred speech, inability to hold a conversation, anger, or forgetfulness,
If the second abnormality information is classified as slurred speech or incomplete conversation, the interactive robot calls out to the user again.
An information processing method characterized in that the processing is executed by a computer. 2. The information processing method according to claim 1 , wherein, in a situation where the interactive robot detects the user using a sensor, if the interactive robot does not receive a response from the user when the interactive robot speaks to the user again, the interactive robot executes a notification process. When the interactive robot detects the user through a sensor, the interactive robot outputs a greeting voice;
acquiring face image data of the user responding to the output voice via an imaging unit of the interactive robot;
The information processing method according to claim 1 or 2 , further comprising inputting the acquired face image data into the first learning model. 4. The information processing method according to claim 1, wherein, when the first learning model outputs the first anomaly information, the dialogue data between the first learning model and the user in response to the first anomaly information is acquired. The classification information includes olfactory abnormality,
The information processing method according to claim 1 , wherein , when the second abnormality information is classified as an olfactory abnormality, the abnormality information is output to the terminal device. outputting the abnormality information to a doctor's terminal;
The information processing method according to any one of claims 1 to 5 , further comprising the steps of: connecting the doctor terminal and the interactive robot so that they can communicate with each other, and starting an online consultation. Acquire face image data of the user acquired through the interactive robot;
inputting the acquired facial image data into a first learning model that has been trained to output first abnormality information regarding an abnormality of a user when the facial image data is input, and acquiring first abnormality information;
making a speech in response to the first abnormality information through the interactive robot;
Acquire conversation data between the user and the interactive robot through the interactive robot;
inputting the acquired dialogue data into a second learning model that has been trained to output second anomaly information related to an anomaly of the user when dialogue data is input, and acquiring second anomaly information;
anomaly information is output to a terminal device in response to the second anomaly information ,
The second abnormality information has classification information including slurred speech, inability to hold a conversation, anger, or forgetfulness,
If the second abnormality information is classified as slurred speech or incomplete conversation, the interactive robot calls out to the user again.
An information processing program that causes a computer to execute a process. In an information processing device including a control unit,
The control unit
Acquire face image data of the user acquired through the interactive robot;
inputting the acquired facial image data into a first learning model that has been trained to output first abnormality information regarding an abnormality of a user when the facial image data is input, and acquiring first abnormality information;
making a speech in response to the first abnormality information through the interactive robot;
Acquire conversation data between the user and the interactive robot through the interactive robot;
inputting the acquired dialogue data into a second learning model that has been trained to output second anomaly information related to an anomaly of the user when dialogue data is input, and acquiring second anomaly information;
anomaly information is output to a terminal device in response to the second anomaly information ,
The second abnormality information has classification information including slurred speech, inability to hold a conversation, anger, or forgetfulness,
The control unit
If the second abnormality information is classified as slurred speech or incomplete conversation, the interactive robot calls out to the user again.
1. An information processing device comprising: