JP7689324B2 - Situation estimation device, situation estimation system, situation estimation method using the same, and situation estimation program - Google Patents

Description

The present disclosure relates to a situation estimation device, a server device, a situation estimation system, and a situation estimation method that estimate a user's situation based on the user's biometric data acquired by a wearable device.

Conventionally, technologies have been developed that acquire biometric data (heart rate, pulse rate, body temperature, etc.) of workers working in factories or outdoors, and estimate the deterioration of the worker's physical condition based on the biometric data (see Patent Documents 1 and 2). These conventional technologies can ensure the safety of workers by detecting abnormalities in the worker at an early stage and issuing a warning to the worker.

On the other hand, in the above-mentioned conventional technology, changes in biometric data are judged based on a predetermined standard (e.g., past normal values), but changes in biometric data judged based on simple standards may not be able to accurately estimate the physical condition of the worker.

In response to this, a technique is known in which an estimation function table is used to estimate the worker's physical condition from biometric data, and the estimation function table is updated appropriately based on the estimated results of the worker's physical condition and the worker's subjective symptoms (see Patent Document 3). According to this conventional technique, the worker's physical condition is estimated using an estimation function table that is appropriately updated, so if the accuracy of the estimation function table is improved, the worker's physical condition can be estimated more accurately.

JP 2015-154920 A JP 2010-218126 A JP 2017-173899 A

In recent years, AI (artificial intelligence) technology has been developing rapidly, and it is believed that AI technology such as machine learning can be applied to the estimation of the worker's condition (including the worker's physical condition, posture, movement state, etc.) as described in the above-mentioned Patent Documents 1-3.

However, in AI technology, generating a trained model for accurately estimating a worker's situation usually requires a large amount of training data on many workers, which increases the time and cost required for training. In addition, because the characteristics of human movements vary greatly from person to person, it can be difficult to improve the accuracy of estimating a worker's situation with a general-purpose trained model, even if a large amount of training data is used.

Therefore, the main objective of the present disclosure is to provide a situation estimation device, a server device, a situation estimation system, and a situation estimation method that can improve the accuracy of estimating a user's situation based on the user's biometric data acquired by a wearable device.

The situation estimation system disclosed herein is a situation estimation system that estimates a user's situation, and includes a wearable device that is worn by the user and acquires biometric data of the user, and a situation estimation device that estimates the user's situation using a specific trained model that is individually provided to the user based on the biometric data. The situation estimation device includes a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that executes re-learning of the specific trained model, a model update unit that updates the specific trained model based on the result of the re-learning executed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model update unit. When a specified situation occurs for the user, the model re-learning unit executes the re-learning using the biometric data acquired in the specified situation as learning data.

The situation estimation device disclosed herein is a situation estimation device that estimates the situation of a user using a specific trained model provided individually to the user based on the user's biometric data acquired by a wearable device, and includes a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that executes re-learning of the specific trained model, a model update unit that updates the specific trained model based on the result of the re-learning executed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model update unit, and is configured such that when a specified situation occurs for the user, the model re-learning unit executes the re-learning using the biometric data acquired in the specified situation as learning data.

The server device of the present disclosure is a server device constituting a situation estimation device in the situation estimation system, and is configured to include a model update unit that updates the specific trained model based on the result of the re-learning performed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model update unit.

The server device of the present disclosure is a server device that performs re-learning of a specific trained model that is individually provided to a user based on the user's biometric data acquired by a wearable device, and has a candidate model storage unit that stores multiple pre-trained candidate models, a model selection unit that selects the specific trained model from the candidate models, and a model re-learning unit that performs re-learning of the specific trained model, and is configured such that when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as training data.

The situation estimation method disclosed herein is a situation estimation method executed by at least one computer to estimate the situation of a user using a specific trained model individually provided to the user based on the user's biometric data acquired by a wearable device, and is configured to select the specific trained model from a plurality of pre-trained candidate models, perform re-learning of the specific trained model, update the specific trained model based on the results of the re-learning, estimate the user's situation based on the latest biometric data using the updated specific trained model, and, in the re-learning of the specific trained model, when a specified situation occurs for the user, perform the re-learning using the biometric data acquired in the specified situation as learning data.

According to the present disclosure, it is possible to improve the accuracy of estimation when estimating a user's condition based on the user's biometric data acquired by a wearable device.

1 is a diagram showing the overall configuration of a situation estimation system according to an embodiment of the present invention; FIG. 2 is a functional block diagram showing the configuration of the user device shown in FIG. 1; FIG. 2 is a functional block diagram showing the configuration of the situation estimation device shown in FIG. 1 . FIG. 4 is an explanatory diagram of a candidate model stored in the candidate model storage unit shown in FIG. 3; FIG. 4 is an explanatory diagram of a trained model set stored in the specific model storage unit shown in FIG. 3. Flow diagram showing authentication processing when putting on and taking off a wearable device An explanatory diagram showing the flow of the initial setup process A diagram showing the process flow for retraining a specific trained model FIG. 13 is an explanatory diagram showing an example of information related to biometric data; FIG. 13 is an explanatory diagram showing an example of an inquiry screen displayed on a user terminal. FIG. 4 is a diagram showing a modified example of the functional block diagram shown in FIG. 3.

The first invention made to solve the above problem is a situation estimation system that estimates a user's situation, comprising: a wearable device that is worn by the user and acquires biometric data of the user; and a situation estimation device that estimates the user's situation using a specific trained model individually provided to the user based on the biometric data. The situation estimation device has a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models; a model re-learning unit that executes re-learning of the specific trained model; a model update unit that updates the specific trained model based on the result of the re-learning executed by the model re-learning unit; and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model update unit. When a specified situation occurs for the user, the model re-learning unit executes the re-learning using the biometric data acquired in the specified situation as learning data.

With this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

The second invention is also configured such that the plurality of candidate models includes a plurality of trained models that have been generated using biometric data previously acquired from a plurality of other users as individual training data.

By using multiple trained models generated using the biometric data of multiple other users individually, it is possible to reduce the time and cost required to obtain a specific trained model to be provided to a target user.

The third invention is also configured such that the specified situation includes at least one of the user's walking, the user's body unsteadiness, the user's fall, the user's stumbling, and a specified posture of the user.

This allows re-learning to be performed using biometric data acquired in specific situations (user walking, user body unsteadiness, user falling, user stumbling, or a specified user posture) whose characteristics vary significantly from person to person as learning data, thereby making it possible to more reliably improve the accuracy of estimating the target user's situation.

In addition, the fourth invention is configured such that the model selection unit selects the specific trained model based on the estimation result of the user's movement by each of the candidate models, which is based on the biometric data acquired when the user performs a predetermined movement.

As a result, a specific trained model is selected from the estimation results of the user's movements by each candidate model based on biometric data when the user performs a specific movement, thereby maintaining good estimation accuracy of the user's situation.

The fifth invention is configured such that the wearable device further has a voice output unit that outputs a guide voice to prompt the user to perform the predetermined action, and the situation estimation device further has a voice control unit that causes the voice output unit to output the guide voice.

This makes it possible to more reliably obtain biometric data when the user performs a specified action.

The sixth invention is configured such that the predetermined motion includes a walking motion of the user.

As a result, by having the user perform walking movements that are suitable for selecting a specific trained model, it is possible to more appropriately select a specific trained model.

In addition, the seventh invention is configured such that the wearable device further includes a voice output unit that outputs a voice inquiry to inquire of the user as to whether the specified situation has occurred, and a voice input unit that inputs a response voice of the user that responds to the voice inquiry, and the situation estimation device further includes a voice control unit that causes the voice output unit to output the voice inquiry, and whether the specified situation has occurred is determined based on the voice response.

This makes it possible to more reliably obtain biometric data when a specified situation occurs for a user based on the user's voice response to a voice inquiry.

The eighth invention further includes a user terminal used by the user, the user terminal further includes a display unit that displays a query screen for querying the user as to whether the specified situation has occurred, and an input unit that accepts an input operation by the user in response to the query screen, and the situation estimation device further includes a display control unit that causes the query screen to be displayed on the display unit.

This makes it possible to obtain a more detailed understanding of the specific situation that the user is facing based on the user's input operations on the inquiry screen, thereby improving the accuracy of the biometric data used for re-learning.

The ninth invention is configured such that the wearable device is an earphone-type device that is worn on the user's ear.

This makes it possible to reduce the impact on biometric data caused by differences in the movement characteristics of each individual's body parts.

The tenth invention is a situation estimation device that estimates the situation of a user using a specific trained model provided individually to the user based on the user's biometric data acquired by a wearable device, and includes a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that executes re-learning of the specific trained model, a model update unit that updates the specific trained model based on the result of the re-learning executed by the model re-learning unit, and a situation estimation unit that estimates the situation of the user based on the latest biometric data using the specific trained model updated by the model update unit, and is configured such that, when a specified situation occurs for the user, the model re-learning unit executes the re-learning using the biometric data acquired in the specified situation as learning data.

With this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

The eleventh invention is a server device constituting a situation estimation device in the situation estimation system, comprising: a model update unit that updates the specific trained model based on the result of the re-learning performed by the model re-learning unit; and a situation estimation unit that uses the specific trained model updated by the model update unit to estimate the user's situation based on the latest biometric data.

As a result, when estimating a user's situation based on the user's biometric data acquired by a wearable device, it is possible to execute the estimation process of the user's situation using relatively few computational resources.

The twelfth invention is a server device that performs re-learning of a specific trained model that is individually provided to a user based on the user's biometric data acquired by a wearable device, and includes a candidate model storage unit that stores a plurality of pre-trained candidate models, a model selection unit that selects the specific trained model from the candidate models, and a model re-learning unit that performs re-learning of the specific trained model, and is configured such that, when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as training data.

With this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

The thirteenth invention is a situation estimation method executed by at least one computer to estimate the situation of a user using a specific trained model individually provided to the user based on the user's biometric data acquired by a wearable device, the method comprising the steps of: selecting the specific trained model from a plurality of pre-trained candidate models; re-learning the specific trained model; updating the specific trained model based on the results of the re-learning; estimating the situation of the user based on the latest biometric data using the updated specific trained model; and, in the re-learning of the specific trained model, when a specified situation occurs for the user, performing the re-learning using the biometric data acquired in the specified situation as learning data.

With this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

The following describes an embodiment of the present disclosure with reference to the drawings.

Figure 1 is an overall configuration diagram of a situation estimation system 1 according to this embodiment. As shown in Figure 1, the situation estimation system 1 includes a wearable device 5 and a user terminal 7 as user-side devices 4 that are worn or carried by a user 3. The situation estimation system 1 also includes a user management server 9 and a learning server 11 as situation estimation devices 8 that execute a process to estimate the situation of the user 3 (including physical condition, posture, motion state, etc.).

The user 3 is, for example, a worker working in a factory or outdoors, and is the subject of situation estimation by the situation estimation device 8. However, the user 3 is not limited to a worker, and may be any person whose situation needs to be confirmed (estimated). For convenience, in this embodiment, only one user 3 is described, but in practice, multiple users similar to the user 3 are the subject of situation estimation. When there are multiple users, each user wears or carries a wearable device 5 and a user terminal 7.

The wearable device 5 is an earphone-type device that is worn on the ear of the user 3. The wearable device 5 has a function of acquiring biometric data of the user 3. The acquired biometric data includes data related to the body of the user 3, such as the heart rate, pulse, and body temperature of the user 3, as well as data related to the behavior of the user 3, such as the movement, position, and posture of the user 3. The wearable device 5 is not necessarily limited to an earphone-type device, and may be, for example, a glasses-type or wristwatch-type device. However, by using an earphone-type device as the wearable device 5 (i.e., acquiring biometric data from the head of the user 3), there is an advantage in that the influence on the acquired biometric data can be suppressed even if the movement characteristics of each part of the body other than the head (e.g., the arms) differ greatly from person to person.

The user terminal 7 is an information device with wireless communication capabilities, such as a smartphone or tablet terminal carried by the user 3. The user terminal 7 can communicate with the wearable device 5, for example, by short-range wireless communication conforming to the Bluetooth (registered trademark) standard. The user terminal 7 is used to supplement functions (such as long-range communication functions and display functions) that are lacking in the wearable device 5, which has a relatively simple configuration. Therefore, if the wearable device 5 is capable of performing the same functions as the user terminal 7, the user terminal 7 can be omitted.

The user management server 9 manages the activities of the user 3 and the safety of the user 3 during those activities. To ensure the safety of the user 3, the user management server 9 estimates the situation of the user 3 using a trained model (hereinafter referred to as a specific trained model) that is individually provided to the user 3 based on the biometric data of the user 3. As such a specific trained model, a neural network that learns by deep learning or a statistical machine learning model such as an SVM (support vector machine) can be used.

The user management server 9 is placed in the same LAN (Local Area Network) so as to be able to communicate with the user terminal 7. This allows the user management server 9 to sequentially acquire the biometric data of the user 3 acquired by the wearable device 5 via the user terminal 7. However, the user management server 9 may also be communicatively connected to the user terminal 7 via a wider network (for example, the network 13 described below).

The learning server 11 generates multiple pre-trained candidate models that are candidates for the specific trained model used by the user management server 9. Such candidate models include multiple trained models (hereinafter referred to as individual trained models) that are generated using biometric data previously acquired from multiple other users as individual training data. An individual trained model is a trained model specialized for estimating the situation of a specific user (i.e., the user from whom the biometric data used as training data was acquired).

In addition, when a specified situation occurs for user 3, the learning server 11 uses the biometric data acquired in the specified situation as learning data to re-learn the specific trained model. Such specified situations include at least one of the following: user 3 walking, user 3 body unsteadiness, user 3 falling, user 3 tripping, and a specified posture of user 3.

The learning server 11 can communicate with the user management server 9 via a network 13 such as the Internet. However, the learning server 11 may also be connected to the user management server 9 via a LAN or a dedicated line so as to be able to communicate with it.

Figure 2 is a functional block diagram showing the configuration of the user device 4 shown in Figure 1.

The wearable device 5 includes a proximity sensor 21, a pulse wave sensor 22, a motion sensor 23, and a GPS sensor 24 as sensors for acquiring biometric data of the user 3. These sensors 21-24 can be publicly known sensors. The proximity sensor 21 detects the state of proximity to the body of the user 3. The pulse wave sensor 22 detects the pulse wave of the user 3. The motion sensor 23 includes a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor, and detects the movement and posture of the user 3. The GPS sensor 24 detects the position of the user 3 based on a signal from a satellite. Note that, when the user 3 is active indoors, the wearable device 5 may detect the position of the user 3 by including a beacon signal receiving unit that receives a signal from a beacon transmitter placed at an appropriate location. When the user terminal 7 includes a GPS sensor or a beacon signal receiving unit, the position of the user 3 may be detected using the GPS sensor or the beacon signal receiving unit of the user terminal 7.

The wearable device 5 also includes a speaker (audio output unit) 27 that outputs audio, and a microphone (audio input unit) 28 that inputs the voice of the user 3 and surrounding sounds.

In the wearable device 5, the communication unit 31 communicates with other devices (here, the user terminal 7) in accordance with a known wireless communication standard. The communication unit 31 has known hardware (not shown) such as an antenna, modem, and wireless communication circuit for communicating with other devices. However, the wearable device 5 may be connected to the user terminal 7 so as to enable wired communication according to the USB standard or the like. As will be described in detail later, the wearable device 5 transmits biometric data detected by the sensors 21-24, voice input to the microphone 28, and the like to the user terminal 7. These data are further transmitted (transferred) from the user terminal 7 to the user management server 9.

In addition, in the wearable device 5, the control unit 33 controls each unit of the wearable device 5 in an integrated manner. The control unit 33 has a user authentication unit 34 that performs a process of authenticating the user 3 who wears the wearable device 5. The control unit 33 includes one or more processors as hardware. The functions of the control unit 33, including the processing of the user authentication unit 34, are realized by the processor executing a program stored in a memory (not shown).

The user terminal 7 includes a display unit 41 that displays information including characters and figures to the user 3, and an input unit 42 that accepts input operations of data and information by the user 3. The display unit 41 and the input unit 42 are realized, for example, by a known liquid crystal panel and touch panel.

In the user terminal 7, the communication unit 45 communicates with other devices (here, the wearable device 5 and the user management server 9) in accordance with a known wireless communication standard. The communication unit 45 has known hardware (not shown) such as an antenna, modem, and wireless communication circuit for communicating with other devices. As will be described in detail later, the user terminal 7 transmits (transfers) a control command (audio output command) from the user management server 9 to the wearable device 5 in order to output a specified sound from the speaker 27.

In addition, in the user terminal 7, the control unit 47 performs overall control of each unit of the wearable device 5. Furthermore, the control unit 47 executes a process of estimating the situation of the user 3 in cooperation with the user management server 9 using a pre-installed user management application 49. The control unit 47 includes one or more processors as hardware. The functions of the control unit 47, including the user management application 49, are realized by the processor executing a program stored in a memory (not shown).

Figure 3 is a functional block diagram showing the configuration of the situation estimation device 8 shown in Figure 1. Figure 4 is an explanatory diagram of the candidate models stored in the candidate model storage unit 55 shown in Figure 3. Figure 5 is an explanatory diagram of the trained model set stored in the specific model storage unit 57 shown in Figure 3. Note that Figure 11 shows a modified version of the functional block diagram shown in Figure 3.

The user management server 9 is a server device having a known hardware configuration. The user management server 9 includes a communication unit 51, a control unit 53, a candidate model storage unit 55, a specific model storage unit 57, and a biometric data storage unit 59.

The communication unit 51 communicates with other devices (here, the user terminal 7 and the learning server 11) in accordance with a known communication standard. The communication unit 51 has known hardware (not shown) such as an antenna, modem, and wireless communication circuit for communicating with other devices. As described in detail later, the user management server 9 transmits learning data for generating an individual trained model and learning data for re-training a specific trained model to the learning server 11.

The control unit 53 comprehensively controls each unit of the user management server 9. Furthermore, the control unit 53 has a model selection unit 61, a learning data extraction unit 62, a model update unit 63, a situation estimation unit 64, a voice control unit 65, and a display control unit 66.

The model selection unit 61 selects a specific trained model to be individually provided to the user 3 from among multiple candidate models prepared in advance. The candidate model storage unit 55 stores multiple candidate models in advance (see pre-trained models (1), (2), (3), ... in Figure 4). The multiple candidate models are composed of individual trained models.

The specific trained model selected by the model selection unit 61 is associated with a specific user (here, user 3) and sequentially stored in the specific model storage unit 57. Typically, each specific trained model is used only to estimate the situation of the associated single user.

However, multiple specific learned models may be selected for one user 3. In that case, for example, as shown in FIG. 5, the specific model storage unit 57 stores a set of learned models as specific learned models for estimating the situation of, for example, user A. The set of learned models includes, for example, a learned model 67 including user A's walking detection pattern as a parameter, a learned model 68 including user A's body sway detection pattern as a parameter, and a learned model 69 including user A's fall detection pattern as a parameter (the same applies to other users B, C, etc.).

The biometric data of user 3 acquired by wearable device 5 is stored sequentially in biometric data storage unit 59. The learning data extraction unit 62 extracts a portion of the biometric data of user 3 stored in biometric data storage unit 59 as learning data for re-learning a specific trained model. More specifically, when the learning data extraction unit 62 determines that a specified situation has occurred for user 3 (such as unsteadiness of the user 3's body), it extracts a portion of the stored biometric data acquired in the specified situation as learning data. In addition, the learning data extraction unit 62 can also extract a portion of the biometric data of user 3 as learning data for generating an individual trained model in the learning server 11.

The model update unit 63 appropriately updates the specific trained model stored in the specific model memory unit 57 based on the results of re-learning of the specific trained model executed by the learning server 11.

The situation estimation unit 64 uses the specific trained model to estimate the situation of user 3 based on biometric data obtained in real time from user 3 (i.e., the latest biometric data). When the specific trained model is updated by the model update unit 63, the situation estimation unit 64 estimates the situation of user 3 using the updated specific trained model. However, when starting to newly estimate the user's situation, the situation estimation unit 64 estimates the situation of user 3 using the candidate model selected by the model selection unit 61 (i.e., a specific trained model that has not been re-trained).

The audio control unit 65 transmits an audio output command to the user terminal 7 to output a predetermined audio from the speaker 27 of the wearable device 5. The audio output from the speaker 27 includes a guide audio for causing the user 3 to perform a predetermined action when the model selection unit 61 selects a specific trained model. The audio output from the speaker 27 also includes an inquiry audio for inquiring of the user 3 as to whether or not a specified situation (such as the user 3's body unsteadiness) has occurred.

The display control unit 66 transmits a screen display command to the user terminal 7 to display a predetermined screen on the display unit 41 of the user terminal 7. Such a screen includes an inquiry screen for the user management server 9 to inquire about the situation of the user 3. If the wearable device 5 has a display unit (such as a liquid crystal display or projector), the display control unit 66 may display a predetermined screen on the display unit of the wearable device 5 instead of the display unit 41 of the user terminal 7 (or together with the display unit 41 of the user terminal 7).

The learning server 11 is a server device having a known hardware configuration. The user management server 9 includes a communication unit 71, a control unit 73, a pre-trained model storage unit 75, a re-trained model storage unit 77, and a learning data storage unit 79. The learning server 11 appropriately provides the user management server 9 with candidate models to be used by the user management server 9 and specific re-trained trained models.

The communication unit 71 communicates with other devices (here, the user management server 9) in accordance with a known communication standard. The communication unit 71 has known hardware, similar to the communication unit 51 of the user management server 9.

The control unit 73 comprehensively controls each part of the learning server 11. Furthermore, the control unit 73 includes a model generation unit 81, a model re-learning unit 83, and a model distribution unit 85.

The model generation unit 81 generates multiple individual trained models using biometric data acquired from multiple users other than user 3 as individual training data. The trained models generated by the model generation unit 81 are stored sequentially in the pre-trained model storage unit 75. However, the pre-trained model storage unit 75 may also include a general-purpose trained model. The general-purpose trained model may be generated, for example, using multiple biometric data corresponding to specified situations that have been collected in advance as sample data depending on the application as training data.

The model re-learning unit 83 executes re-learning of the corresponding specific trained model based on the training data extracted by the user management server 9 (training data extraction unit 62) and transmitted to the training server 11. The re-learned models are stored sequentially in the re-learned model storage unit 77. The re-learned model storage unit 77 may store the specific trained model to be re-learned (i.e., before re-learning). The model re-learning unit 83 can execute re-learning of the same specific trained model (i.e., associated with the same user) every time new training data is acquired. As another variation, the training data received by the training server 11 can be stored in the training data storage unit 79 in the training server 11 and used as appropriate as data for re-learning executed by the model re-learning unit 83. The timing of transmission of the learning data extracted by the learning data extraction unit 62 may be manually input/instructed by the user at any timing using an application (not shown) of the user management server, or the learning data extracted by the learning data extraction unit 62 may be temporarily stored in a recording area of the user management server 9, and the temporarily stored learning data may be automatically transmitted at a predetermined interval.

The model distribution unit 85 appropriately transmits the initial trained models (individual trained models, general-purpose trained models) stored in the pre-trained model storage unit 75 to the user management server 9 as candidate models. In addition, when re-learning of a specific trained model is completed, the model distribution unit 85 appropriately transmits the re-trained specific trained model to the user management server 9 as a fixed trained model for update.

In addition, in the situation estimation system 1, if the user management server 9 has sufficient computational resources, it is possible to configure it to have at least some of the functions of the learning server 11, and conversely, if the computational resources of the user management server 9 are insufficient, it is also possible to configure it so that the learning server 11 has some of the functions. For example, as shown in FIG. 11, instead of the user management server 9, the learning server 11 may have the functions of the candidate model storage unit 55 and the model selection unit 61. In some cases, the situation estimation device 8 may be realized as a single device that has the functions of both the user management server 9 and the learning server 11. Furthermore, in addition to the user management server 9 and the learning server 11, the situation estimation device 8 may also be configured to further include other devices (such as a server or a PC) that can execute some of the functions. The situation estimation device 8 is not limited to a server device and may be configured by at least one computer having similar functions. In addition, it is also possible to improve security against cyber attacks on the network by configuring the processing of the situation estimation unit 64 and the model re-learning unit 83 to be secretly shared by multiple servers. These functional layouts can be appropriately designed and implemented depending on the resources (CPU power, memory, storage capacity, etc.) of the user management server 9 and the learning server 11, the network communication speed and environment between the user management server 9 and the learning server 11, the security level required for the situation assessment system, etc.

Figure 6 is a flow diagram showing the authentication process when the wearable device 5 is put on and taken off.

The wearable device 5 acquires biometric data only of the specific user with which it is associated. Therefore, when the wearable device 5 is worn and removed, the wearer is subjected to authentication processing as shown in FIG. 6.

First, when user 3 puts on wearable device 5 to start an activity, the proximity sensor 21 of wearable device 5 detects that it is being worn by user 3 (ST101: Yes).

Next, the wearable device 5 makes an identity authentication request to the user 3 (ST102). In the identity authentication request, for example, a voice is output from the speaker 27 to prompt the user 3 to speak. Then, the user authentication unit 34 executes a voiceprint authentication process on the user's voice input to the microphone 28 in response to that voice.

After that, when user 3 is successfully authenticated (ST103: Yes), wearable device 5 enters a continuous authentication state (ST104). As a result, wearable device 5 starts acquiring biometric data from user 3.

At the same time, the wearable device 5 transmits an authentication completion notification to the user terminal 7 to inform the user 3 that the user 3 has been successfully authenticated (ST105). The user terminal 7 that has received the authentication completion notification displays an authenticated screen on the display unit 41 indicating that the user 3 has been successfully authenticated. The wearable device 5 may output a sound from the speaker 27 to inform the user 3 that the user 3 has been successfully authenticated.

After that, when the user 3 finishes his/her activity and removes the wearable device 5, the wearable device 5 detects by the proximity sensor 21 that the wearable device 5 has been removed from the user 3 (ST106: Yes). This causes the wearable device 5 to enter a de-authentication state (ST107) and stops acquiring biometric data from the user 3.

In addition, in the above step ST101, the wearable device 5 may use the motion sensor 23 instead of (or together with) the proximity sensor 21 to detect that the wearable device 5 is worn by the user 3. In addition, in the above step ST102, the wearable device 5 may authenticate the user 3 using a method other than voiceprint authentication (for example, biometric authentication processing based on the characteristics of the movements of the user 3 detected by the motion sensor 23, or keyword authentication processing by voice utterance and voice recognition using a pre-agreed password, etc.).

Figure 7 is an explanatory diagram showing the flow of the initial setup process when user 3 starts using the situation estimation system 1. The initial setup process is a process for selecting the most suitable candidate model for user 3 from multiple candidate models as the specific trained model.

The user management server 9 starts the initial setup process in response to a setup request from the user terminal 7 (user management application 49) based on an input operation by the user 3 (1001). Then, the user management server 9 transmits an action execution command to the wearable device 5 via the user terminal 7 to cause the user 3 to execute a predetermined action (1003).

The wearable device 5, which has received the action execution command, outputs a guide voice from the speaker 27 to cause the user 3 to execute a predetermined action (1005). Data for the guide voice is either stored in the wearable device 5 in advance, or is included in the action execution command sent from the user management server 9 to the wearable device 5. The guide voice includes, for example, voices such as "Turn right and walk five steps," and "Go up the stairs." As a result, the user 3 executes the predetermined action (for example, turn right and walk five steps) in accordance with the guide voice.

The wearable device 5 acquires biometric data of the user 3 performing the predetermined action (1007). Furthermore, the wearable device 5 transmits the acquired biometric data to the user management server 9 via the user terminal 7 (1009). In this way, the user management server 9 can more reliably acquire biometric data when the user 3 performs the predetermined action. Note that by including a characteristic action (e.g., a jumping action) at the start and end of the predetermined action in response to the action execution command, the user management server 9 can easily recognize the start and end of the predetermined action in the biometric data acquired as time-series data.

When the user management server 9 acquires the biometric data of the user 3 from the wearable device 5 when the user 3 performs a specific movement, the user management server 9 executes a process of selecting a specific trained model based on the biometric data (1011). More specifically, the model selection unit 61 uses multiple candidate models stored in the candidate model storage unit 55 to calculate the estimation accuracy (%) of the user 3 performing a specific movement (e.g., a walking movement of turning right and taking five steps) from the biometric data of the user 3, and selects the candidate model with the highest estimation accuracy as the specific trained model.

Then, the user management server 9 notifies the user terminal 7 of the completion of the initial setup, including information on the selected specific trained model (1013). In the user terminal 7, the information on the selected specific trained model is registered by the user management application 49 (1015). Note that the information on the selected specific trained model is not registered in the user terminal 7, and the user management server 9 manages the association between the information on the selected specific trained model and the ID of user 3 (or the ID of user terminal 7), making it possible to apply the selected specific trained model to user 3.

Figure 8 is an explanatory diagram showing the flow of the process of re-learning a specific trained model in the situation estimation system 1. Figure 9 is an explanatory diagram showing an example of information related to biometric data. Figure 10 is an explanatory diagram showing an example of an inquiry screen displayed on the user terminal 7.

As shown in FIG. 8, after the initial setup is complete, the wearable device 5 continuously acquires biometric data of the user 3 during his/her activity at a predetermined period (which may be the sampling period of the pulse wave sensor 22, motion sensor 23, etc., or may be an appropriately thinned sampling period to reduce the battery power of the wearable device) (2001). The acquired biometric data is transmitted from the wearable device 5 to the user management server 9 via the user terminal 7 (2003). The user management server 9 sequentially stores information related to the biometric data received from the wearable device 5 in the biometric data storage unit 59 (2005).

As shown in FIG. 9, the information related to the biometric data stored in the biometric data storage unit 59 includes, for example, "sensor data" which is the sensor detection value as the biometric data, "location information" which indicates the location of the user 3 when the data was detected, as well as related information such as a "user ID" for identifying the user 3, the "date" and "time" when the sensor data was detected, and a "specified situation detection result" which indicates that a specified situation has occurred with respect to the sensor data. The sensor data may include not only data detected at a specified time, but also data detected during a specified period (e.g., waveform data). The "specified situation detection result" is not included in the initial biometric data information stored in the biometric data storage unit 59, and is added as appropriate when a specified situation occurs.

The user management server 9 estimates the status of the user 3 using the specific trained model based on the biometric data received from the wearable device 5 (2007). If the user management server 9 determines as a result of the estimation that the status of the user 3 does not correspond to the specified status (or that it is unlikely to correspond), it waits to receive the next biometric data from the wearable device 5.

On the other hand, when the user management server 9 determines from the estimation result of the user 3's situation that a specified situation (such as the user 3's body unsteadiness) has occurred, it transmits a voice output command to the wearable device 5 via the user terminal 7 (2009).

The wearable device 5, which has received the voice output command, outputs a voice inquiry about the situation of the user 3 from the speaker 27 (2011). The voice inquiry includes, for example, voice inquiring about the situation of the user 3, such as "Are you OK?" or "Is something wrong?". The voice inquiry may also include advice about managing the user 3's physical condition, such as "It's hot, so please take a break and drink some fluids."

When user 3 responds to the voice inquiry by speaking, the response voice of user 3 is input to microphone 28, and the response voice data is further transmitted to user management server 9 via user terminal 7 (2013). The voice inquiry includes, for example, voices such as "I'm feeling a bit dizzy" and "I'm fine."

The control unit 53 of the user management server 9 receives the data of the response voice and estimates the content of the response voice through voice recognition processing (2015). Next, when the user management server 9 determines from the estimation result that a specified situation has occurred for user 3, it transmits a screen display command to display a query screen on the display unit 41 of the user terminal 7 (2017). At this time, the user management server 9 adds the result of estimating the content of the response voice (e.g., that user 3 has become unsteady) to the information on the corresponding biometric data (see FIG. 9) as a specified situation detection result. In this way, the user management server 9 can more reliably acquire biometric data when a specified situation occurs for user 3, based on the response voice of user 3 to the query voice.

Based on a screen display command from the user management server 9, the user terminal 7 displays an inquiry screen 91 for checking the status of the user 3 in more detail (2019). The inquiry screen displays items to be checked related to the movement (here, body unsteadiness) estimated based on the response voice of the user 3, as shown in FIG. 10, for example. Here, the items to be checked related to the body unsteadiness of the user 3 include options such as unsteadiness during "normal walking", unsteadiness during "brisk walking", unsteadiness while climbing "stairs", "fall" after unsteadiness, unsteadiness due to "stumbling", and "other". The result of the input operation (e.g., selection of an option) by the user 3 on the inquiry screen 91 is transmitted to the user management server 9 as response data (2021).

The user management server 9, which has received the response data from the user terminal 7, determines the situation of the user 3 based on the response data, and adds the situation of the user 3 (for example, that the user 3 tripped while climbing stairs) to the corresponding biometric data information (see FIG. 9) as a specified situation detection result (2023). In this way, the user management server 9 can grasp the specified situation that occurred to the user 3 in more detail based on the input operation of the user 3 on the inquiry screen, thereby improving the accuracy of the biometric data used for re-learning.

The user management server 9 can add the specified situation detection result of the corresponding biometric data information (see FIG. 9) based on at least one of the response voice from the wearable device 5 and the response data from the user terminal 7. In other words, at least one of the process for acquiring the response voice from the wearable device 5 and the process for acquiring the response data from the user terminal 7 may be omitted.

The user management server 9 appropriately extracts the biometric data with the specified situation detection result as learning data and transmits it to the learning server 11 (2025). The learning server 11 re-learns the specific trained model based on the received learning data (2027). At this time, the specified situation detection result is used as training data.

The learning server 11 transmits the specific trained model for which re-learning has been completed to the user management server 9 as information regarding the results of the re-learning (2029). The user management server 9 updates the currently used specific trained model with the re-learned specific trained model (2031).

When the specific trained model includes multiple trained models (see FIG. 5), the re-learning of the specific trained model may be performed for each trained model. The re-learning in the training server 11 may be performed every time training data is received from the user management server 9. Alternatively, the frequency of occurrence of cases where the estimation result of the situation of user 3 by the specific trained model applied to user 3 differs from the specified situation detection result (the user's actual situation determined by the response data of user 3) (estimation error) is measured, and the re-learning process is performed when the occurrence frequency exceeds a predetermined criterion, thereby eliminating the execution of unnecessary re-learning processes and performing re-learning only when the accuracy of the situation estimation has deteriorated. In this case, the same effect can be obtained by determining the estimation error on the user management server 9 side, transmitting training data to the training server 11 when the occurrence frequency exceeds a predetermined criterion, and performing re-learning on the training server 11 side every time training data is received.

In this way, the situation estimation system 1 selects a specific trained model for estimating the situation of the target user 3 from among multiple pre-trained candidate models, and re-trains the specific trained model using the biometric data of the user 3 in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the situation of the user 3.

In addition, the situation estimation system 1 can reduce the time and cost required to obtain a specific trained model to be provided to the target user 3 by using multiple trained models generated individually using the biometric data of multiple other users as candidate models.

The present disclosure has been described above based on specific embodiments, but these embodiments are merely examples and the present disclosure is not limited to these embodiments.

For example, the situation estimation system 1 can be configured to omit the user management server 9 and have the user terminal 7 function as the user management server 9. Furthermore, the situation estimation system 1 can not only manage the safety of the active user 3, but also manage the work of the user 3, for example, by estimating the walking status of the user 3.

The components of the situation estimation device, server device, situation estimation system, and situation estimation method according to the present disclosure shown in the above embodiments are not necessarily all essential, and can be selected as appropriate at least as long as they do not deviate from the scope of the present invention.

1: Situation estimation system 3: User 4: User device 5: Wearable device 7: User terminal 8: Situation estimation device 9 : User management server 11: Learning server 13: Network 21: Proximity sensor 22: Pulse wave sensor 23: Motion sensor 24: GPS sensor 27: Speaker 28: Microphone 31: Communication unit 33: Control unit 34: User authentication unit 41: Display unit 42: Input unit 45: Communication unit 47: Control unit 49: User management application 51: Communication unit 53: Control unit 55: Candidate model storage unit 57: Specific model storage unit 59: Biometric data storage unit 61: Model selection unit 62: Learning data extraction unit 63: Model update unit 64: Situation estimation unit 65: Voice control unit 66: Display control unit 71: Communication unit 73: Control unit 75: Pre-trained model storage unit 77: Re-trained model storage unit 79: Learning data storage unit 81: Model generation unit 83: Model re-learning unit 85: Model distribution unit 91: Inquiry screen

Claims (10)

A situation estimation system for estimating a user's situation, comprising:
A device for acquiring biometric information of the user;
a situation estimation device that estimates a situation of the user by using a specific trained model that is individually provided to the user based on the biometric information,
The situation estimation device includes:
A model selection unit that selects the specific trained model from a plurality of pre-trained candidate models;
A model re-learning unit that performs re-learning of the specific trained model;
A model update unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit;
A situation estimation unit that estimates a situation of the user based on the latest biometric information by using the specific trained model updated by the model update unit.
Situation estimation system. The device comprises:
At least one of an earphone-type wearable device, a glasses-type wearable device, and a wristwatch-type wearable device.
The situation estimation system according to claim 1 . The device is worn or carried by the user.
The situation estimation system according to claim 1 . the model re-learning unit, when a specified situation occurs for the user, executes the re-learning by using the biometric information acquired in the specified situation as learning data.
The situation estimation system according to claim 1 . The device comprises:
a display unit that displays an inquiry screen for inquiring of the user as to whether or not a specified situation has occurred;
an input unit that accepts an input operation by the user in response to the inquiry screen;
Further comprising:
The situation estimation device includes:
Further comprising a display control unit that causes the display unit to display the inquiry screen.
The situation estimation system according to claim 1 . a display unit that displays an inquiry screen for inquiring of the user as to whether or not a specified situation has occurred;
an input unit that accepts an input operation by the user in response to the inquiry screen;
A display device having
The situation estimation system according to claim 1 . A situation estimation method for a situation estimation system that estimates a user's situation, comprising:
The situation estimation system includes a device and a situation estimation apparatus,
The device acquires biometric information of the user,
the situation estimation device estimates the user's situation based on the biometric information by using a specific trained model individually provided to the user;
Selecting the specific trained model from among a plurality of pre-trained candidate models;
Retraining the specific trained model;
updating the specific trained model based on the results of the re-training performed;
Using the updated specific trained model, a state of the user is estimated based on the latest biometric information.
How to estimate the situation. A situation estimation device that estimates a situation of a user by using a specific trained model individually provided to the user based on biometric information of the user,
A model selection unit that selects the specific trained model from a plurality of pre-trained candidate models;
A model re-learning unit that performs re-learning of the specific trained model;
A model update unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit;
a situation estimation unit that estimates a situation of the user based on the latest biometric information by using the specific trained model updated by the model update unit;
having
Situation estimation device. A situation estimation method for estimating a situation of a user by a situation estimation device, the situation estimation device estimating a situation of the user by using a specific trained model individually provided to the user based on biometric information of the user,
Selecting the specific trained model from among a plurality of pre-trained candidate models;
Retraining the specific trained model;
updating the specific trained model based on the results of the re-training;
Using the updated specific trained model, a state of the user is estimated based on the latest biometric information.
How to estimate the situation. A situation estimation program that causes a computer to execute information processing to estimate a situation of each user using a specific trained model individually provided to the user based on biometric information of the user,
The information processing includes:
The computer,
Selecting the specific trained model from among a plurality of pre-trained candidate models;
Retraining the specific trained model;
updating the specific trained model based on the results of the re-training;
A situation estimation program comprising a step of estimating the user's situation based on the latest biometric information using the updated specific trained model.

Images