JP6969177B2 - Specific behavior detection program, specific behavior detection method and specific behavior detection system - Google Patents

Description

The present invention relates to a specific behavior detection program, a specific behavior detection method, and a specific behavior detection system.

There is known a technique for determining a user's eating behavior by using a sensor that detects the user's heartbeat or a sensor that detects the movement of the user's arm. For example, an acceleration sensor detects accelerations of at least three axes that are orthogonal to each other attached to the lower arms of both arms of the user, and the detected sensor signals are compared with the stored eating behavior criteria to determine the eating behavior of the user. A technique for determining the presence or absence is known.

Japanese Unexamined Patent Publication No. 2011-115508

Raul I. Ramos-Garcia, Adam W. Hoover "A Study of Temporal Action Sequencing During Consumption of a Meal" ACM-BCB 2013

However, with the above technique, there is a limit to the accuracy of behavior detection. That is, in the above technique, the meal is detected by raising and lowering the arm, but the raising and lowering of the arm is only a part of a series of movements performed by the meal. For this reason, if it is identified as a meal just because the arm is raised and lowered, it becomes difficult to suppress false detection when the arm is raised and lowered in activities of daily life other than meals.

In one aspect, it is an object of the present invention to provide a specific behavior detection program, a specific behavior detection method, and a specific behavior detection system that can improve the detection accuracy of a specific behavior such as a meal.

In one embodiment, a plurality of sensors having a first feature from the sensor data are referred to by referring to the sensor data of a predetermined section acquired from the sensor for measuring the activity of the subject, and based on the sensor data and the first condition. The first section and the plurality of second sections having the second feature are extracted, and the first feature amount for at least one parameter is calculated for the plurality of first sections, and the plurality of said. The second feature amount for the at least one parameter is calculated for the second section of the above, and based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. , The computer is made to perform a process of detecting whether the activity of the subject in the predetermined section is a specific action indicated by the determination model.

According to one aspect, it is possible to improve the detection accuracy of a specific behavior such as eating.

FIG. 1 is a diagram showing an example of a functional configuration of each device included in the health care support system according to the first embodiment. FIG. 2 is a diagram showing an example of a behavior detection method. FIG. 3 is a diagram showing an example of a section feature amount. FIG. 4 is a flowchart showing the procedure of the specific action detection process according to the first embodiment. FIG. 5 is a diagram showing an example of sensor data. FIG. 6 is a diagram showing an example of a phase section. FIG. 7 is a diagram showing an example of a section feature amount. FIG. 8 is a diagram showing an example of phase features. FIG. 9 is a flowchart showing the procedure of the learning process of the model according to the first embodiment. FIG. 10 is a diagram showing an example of phase features. FIG. 11 is a diagram showing an example of a functional configuration of each device included in the health care support system according to the second embodiment. FIG. 12 is a diagram showing an example of a phase type. FIG. 13 is a diagram showing an example of the maximum value of acceleration in the operating section and the stationary section. FIG. 14 is a diagram showing an example of phase features. FIG. 15 is a diagram showing an example of the appearance of the wearable terminal 50. FIG. 16 is a diagram showing a hardware configuration example of a computer that executes the specific behavior detection program according to the first to third embodiments.

The specific behavior detection program, the specific behavior detection method, and the specific behavior detection system according to the present application will be described below with reference to the attached drawings. It should be noted that this embodiment does not limit the disclosed technology. Then, each embodiment can be appropriately combined as long as the processing contents do not contradict each other.

[System configuration]
FIG. 1 is a diagram showing an example of a functional configuration of each device included in the health care support system according to the first embodiment. The healthcare support system 1 shown in FIG. 1 provides various healthcare support services. For example, as an example of a healthcare support service, a service for recording a specific action of a target person using sensor data acquired by a wearable terminal 10, for example, an action related to daily life, and a derivative service utilizing the record, etc. Can be mentioned.

The "specific activities" referred to here include ADL (Activities of Daily Living) and IADL (Instrumental Activities of Daily Living). Of these, ADL includes activities of daily living such as eating, excreting, changing clothes, moving, dressing, and bathing. In addition, IADL includes higher-level living functions than ADL, such as how to use the telephone, shopping, housework, moving, going out, managing medication, and managing money.

As shown in FIG. 1, the healthcare support system 1 includes a wearable terminal 10, an IoT (Internet of Things) device 20, and a server device 30. Although FIG. 1 shows a case where the IoT device 20 accommodates one wearable terminal 10, the IoT device 20 may accommodate a plurality of wearable terminals 10. Further, although FIG. 1 illustrates a case where the server device 30 accommodates one IoT device 20, the server device 30 may accommodate a plurality of IoT devices 20.

Of these, data is exchanged between the wearable terminal 10 and the IoT device 20 via short-range wireless communication such as BLE (Bluetooth (registered trademark) Low Energy) as an example. This is just an example, and the wearable terminal 10 and the IoT device 20 may be connected by wire. Further, the IoT device 20 and the server device 30 can be connected via any network regardless of whether they are wired or wireless. As an example of such a network, in addition to a mobile network compatible with 3G (Generation), LTE (Long Term Evolution), 4G, 5G, etc., a private communication network such as LAN (Local Area Network) or VPN (Virtual Private Network) , Internet, etc.

The wearable terminal 10 is a computer worn by a target person who receives a healthcare support service.

For example, the wearable terminal 10 is equipped with a sensor 11 that measures the activity of the target person. As an example of the activity of the subject, the case where the movement of the arm is measured is taken as an example. For example, a wristband type gadget can be adopted for the wearable terminal 10. In this case, the wearable terminal 10 is attached to the wrist or the like of the dominant arm as an example. Further, the sensor 11 mounted on the wearable terminal 10 can be equipped with a 3-axis accelerometer or the like. The time-series data related to the movement of the arm measured by the sensor 11 is output to the IoT device 20 as an example of the sensor data. Here, as an example of the sensor 11, the case where the acceleration sensor is used is illustrated, but the angular velocity sensor can be used together with the acceleration sensor, or the angular velocity sensor can be used alone instead of the acceleration sensor.

Although FIG. 1 illustrates the case where the sensor 11 is mounted on the wearable terminal 10, the sensor 11 may not necessarily be mounted on the wearable terminal 10. For example, the sensor 11 may be installed in an environment in which the subject lives in daily life. In this case, a camera, a motion sensor, or the like is installed as an environmental device. For example, when a camera is used, sensor data can be obtained by tracking the movement of the arm detected from the image captured by the camera. In addition, when using a motion sensor installed on a flat surface of the desk, the vertical movement of the wrist is detected depending on whether the wrist is in contact with or close to the flat surface of the desk, and the movement of the wrist on the desk is tracked. Can detect movement in the front-back direction.

The IoT device 20 is a device having a function of connecting a wearable terminal 10 to a network as one aspect. As an example, the IoT device 20 is compatible with mobile terminal devices such as slate terminals and tablet terminals, including smartphones and mobile phones used by the target person, PHS (Personal Handyphone System), and the like. For example, the IoT device 20 transfers the sensing data notified from the wearable terminal 10 via the BLE communication or the like to the server device 30 via the network.

As an example, a sensor event such as acceleration data transferred from the wearable terminal 10 to the server device 30 via the IoT device 20 can be transferred as an MQTT (Message Queuing Telemetry Transport) message. At this time, the sensor data may be transmitted in real time each time the acceleration is measured, or may be transmitted after being accumulated for a predetermined period, for example, 12 hours, 1 day, 1 week, 1 month, or the like. It doesn't matter.

The server device 30 is a computer that provides the above-mentioned healthcare support service. For example, the server device 30 can be implemented as packaged software or online software by installing a health care support program that realizes a function corresponding to the above-mentioned health care support service on a desired computer. For example, the server device 30 may be implemented as a Web server that provides the above-mentioned healthcare support service, or may be implemented as a cloud that provides the above-mentioned healthcare support service by outsourcing.

Here, in the server device 30 according to the present embodiment, the characteristics of the distribution regarding the section feature amount obtained for each phase section in which the sensor data is divided into the action phases are obtained for each phase type, and the distribution for each phase type is obtained. A specific behavior detection function that detects a specific behavior based on the feature and the judgment model is implemented.

That is, human behavior is often a repetition of multiple phases. For example, taking a meal as an example of a specific behavior, the movement of the subject during a meal includes Phase A, in which the food to be ingested is grabbed in one bite, Phase B, in which the food is brought back to the mouth, and chewing and swallowing. It is a repetition of three types of phases with Phase C, in which movement stops during (or conversation).

Under the assumption that the phase of the action to be detected is repeated by the sensor data, the server device 30 divides the sensor data into a plurality of sections to which the type of the phase included in the action is assigned. Hereinafter, the section divided from the sensor data may be referred to as a “phase section”.

FIG. 2 is a diagram showing an example of a behavior detection method. FIG. 2 shows an example of detecting a meal as an example of a specific behavior. As shown in the second from the top of FIG. 2, the sensor data shown first from the top of FIG. 2 is divided into a plurality of phase sections to which the type of the phase including the meal is given. For example, when the wearable terminal 10 is attached to the wrist of the dominant arm, each phase section is given a phase type according to the following conditions. That is, when the movement of the wrist in the vertical direction, that is, in the direction of gravity is detected from the sensor data measured in the phase section, the phase section is labeled with the phase type D. When the movement of the wrist in the front-back direction is detected from the sensor data measured in the phase section, the phase section is labeled with the phase type E.

After that, as shown in the third position from the top of FIG. 2, a feature amount representing a meal-likeness is calculated from the sensor data of the phase section for each phase section. Since the "feature amount" referred to here is a feature amount calculated in units of phase sections as one aspect, it may be described as "section feature amount" below.

After such a section feature amount is calculated, as shown in the fourth from the top of FIG. 2, a distribution of the section feature amount belonging to the phase type is generated for each phase type. Then, the feature amount of the distribution is further calculated from the distribution of the section feature amount generated for each phase type. In the example shown in FIG. 2, the histogram H1 is generated by counting the frequency of the class to which the section feature amount calculated from the phase section belongs for each phase section labeled with the phase type D. Further, the histogram H2 is generated by counting the frequency of the class to which the section feature amount calculated from the phase section belongs for each phase section labeled with the phase type E. A statistical value of distribution for each of the histograms H1 and H2, for example, an average is calculated as a feature amount. As one aspect, the feature amount calculated for each phase type has the aspect of the feature amount representing the distribution of the section feature amount belonging to each phase type. Therefore, in the following, it is described as "phase feature amount". There is.

Then, as shown in the fifth from the top of FIG. 2, the phase feature amount calculated for each phase type is input to the model learned by machine learning, so that the classification result of the behavior type, that is, F (meal) Or, whether or not it is notF (non-meal) is output.

Here, the phase interval divided from the sensor data does not necessarily have to coincide with the actual meal phase, and may have an accuracy that overlaps with the actual meal phase over a certain period of time. This is because, even if the features corresponding to the phase types do not clearly appear in the individual section features calculated in each phase section, the statistical values of the distribution of the section features for each phase type follow the law of large numbers. This is because the features corresponding to are clearly shown. In this way, it is possible to detect an action by capturing the characteristics of each phase type that are repeatedly included in the action. Therefore, it is possible to improve the detection accuracy of a specific action such as a meal, as compared with the case where the action detection is performed only by a part of the action as in the technique described in the background technique section.

Further, in addition to the technique described in the column of background technique, there is also a technique such as Non-Patent Document 1 in which an individual phase section is estimated and a meal is determined by a transition method between the phase types. However, in order to learn a phase by machine learning, it takes time and effort to label a large amount of sensor data with a phase type that is finer than the action type of meal. For example, in the case of Non-Patent Document 1, it is troublesome to attach a phase type label such as "cut", "chew", "break", and "drink" to a large amount of sensor data in the phase section. As described above, the non-patent document 1 has a weak point that the hurdle to implementation is high.

On the other hand, in the server device 30 according to the present embodiment, machine learning can be applied to the determination of action detection by assigning a label with a particle size of an action type that is coarser than the particle size of the phase type. Therefore, the server device 30 according to the present embodiment can lower the hurdle to mounting. In addition, in the server device 30 according to the present embodiment, the phase feature amount for each phase type can be applied to machine learning. Therefore, by learning the model by machine learning, it is possible to optimally approach the judgment conditions for behavior detection, such as selection of phase types and setting of judgment threshold values. Therefore, according to the server device 30 according to the present embodiment, it is possible to realize a behavior detection that is robust to the measurement environment of the sensor data and the individual difference of the target person. From this aspect as well, the accuracy of behavior detection can be improved.

After the specific action is detected in this way, the server device 30 can record the action time when the action related to daily life is performed, and also various kinds of health care related actions based on the action time recorded so far. Recommendations can be output. For example, when a meal is detected as an example of a specific action, the server device 30 records the meal time, generates a list of meal time zones over a predetermined period, for example, one week, and outputs the list. It is also possible to output various recommendations after analyzing the eating habits or diet from the meal times recorded so far. As an example, such a recommendation can be output to the IoT device 20 used by the subject, and is also used by a person related to the subject, such as a relative, a medical person such as an industrial physician, or a care person. It can also be output to the terminal device. As a result, the above-mentioned healthcare support service is realized.

Here, as an example, a case where the server device 30 acquires a sensor data and then executes a series of healthcare support services including detection of a specific action, action accumulation, and recommendation output is illustrated, but not all of them are servers. It may not be executed by the device 30. For example, by operating the specific behavior detection program extracted from the specific behavior detection function on the server device 30, the server is focused on the front-end portion from the acquisition of the sensor data to the detection of the specific behavior. It can also be executed by the device 30. In this case, the server device 30 can be entrusted to a service or application that executes functions such as action storage and recommendation output on the back end, AI (Artificial Intelligence), another server device outside the data center, and the like.

[Functional configuration of server device 30]
Next, the functional configuration of the server device 30 according to this embodiment will be described. As shown in FIG. 1, the server device 30 includes an acquisition unit 31, a division unit 32, a section feature amount calculation unit 33, a first calculation unit 34, a second calculation unit 35, and a third calculation. It has a unit 36 and a detection unit 37. In addition to the functional unit shown in FIG. 1, the server device 30 may have various functional units of a known computer, for example, a communication interface for controlling communication with other devices.

Functional units such as the acquisition unit 31, the division unit 32, the section feature amount calculation unit 33, the first calculation unit 34, the second calculation unit 35, the third calculation unit 36, and the detection unit 37 shown in FIG. 1 are CPUs. It is virtually realized by a hardware processor such as (Central Processing Unit) or MPU (Micro Processing Unit). That is, the processor is a storage device (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), or an OS (Operating System), as well as a healthcare support program or a specific action detection program, which are not shown. Read the program. Then, the processor executes a healthcare support program or a specific action detection program to develop a process corresponding to the above-mentioned functional unit on a memory such as a RAM (Random Access Memory). As a result, the above functional unit is virtually realized as a process. Here, a CPU and an MPU are exemplified as an example of a processor, but the above-mentioned functional unit may be realized by an arbitrary processor regardless of a general-purpose type or a specialized type. In addition, the above-mentioned functional unit may be realized by hard-wired logic such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The acquisition unit 31 is a processing unit that acquires sensor data from the sensor 11. As one embodiment, the acquisition unit 31 refers to the sensor data of a predetermined section acquired from the sensor 11 that measures the activity of the target person. As an example, the function of the acquisition unit 31 that acquires the sensor event from the wearable terminal 10 can be implemented by executing the program for the MQTT server. For example, the acquisition unit 31 can input the sensor data acquired from the sensor 11 to the division unit 32 in real time, and can also acquire the sensor data by reading the history of the sensor data stored in the internal storage (not shown). .. The acquisition unit 31 is an example of a reference unit.

The division unit 32 is a processing unit that divides into a plurality of first sections having the first feature and a plurality of second sections having the second feature based on the sensor data and the first condition. The divided portion is an example of an extraction portion.

As one embodiment, the division unit 32 divides the sensor data input from the acquisition unit 31 at equal intervals. For example, the division unit 32 divides the sensor data into phase sections each time the time length of the sensor data input from the acquisition unit 31 reaches a predetermined time length, for example, 5 seconds. Subsequently, the division unit 32 determines whether or not the phase section satisfies the first condition. As an example of such a first condition, in the division portion 32, whether or not the vertical displacement observed in the sensor data of the phase section, that is, the difference between the maximum value and the minimum value of the height is equal to or more than a predetermined threshold value. Is determined. Here, when the difference between the maximum value and the minimum value of the height is equal to or more than the threshold value, it can be identified that there is a high possibility that the food is in the phase of carrying food because there is a height difference in the wrist in the phase section. In this case, the division unit 32 classifies the phase section into the phase type "movement in the vertical direction". The phase type "movement in the vertical direction" corresponds to an example of the first feature, and the phase section to which the phase type "movement in the vertical direction" is given corresponds to an example of the "first section".

On the other hand, when the first condition is not satisfied, that is, when the difference between the maximum value and the minimum value of the height is not equal to or more than the threshold value, the division unit 32 further determines whether or not the phase interval satisfies the second condition. As an example of such a second condition, the division unit 32 determines whether or not the sum of squares of the front-back and left-right accelerations observed in the sensor data of the phase section is equal to or higher than a predetermined threshold value. At this time, when the sum of squares of the front-back and left-right accelerations is equal to or greater than the threshold value, it can be identified that there is a high possibility that the phase is the phase of cutting food because there is a horizontal plane movement of the wrist in the phase section. In this case, the division unit 32 classifies the phase section into the phase type “horizontal movement”. On the other hand, if the second condition is not satisfied, it can be identified that there is a high possibility that the patient is chewing or swallowing. In this case, the division unit 32 classifies the phase section into the phase type “stationary”. The phase type "horizontal movement" corresponds to an example of the second feature, and the phase section to which the phase type "horizontal movement" is given corresponds to an example of the "second section".

Although the example in which the sensor data is divided into equal intervals has been described here as an example, the time length of the phase section may be variable, and the phase section and the phase section in which the sensor data is divided may be variable. It is also possible to overlap a part between them.

The section feature amount calculation unit 33 is a processing unit for calculating the section feature amount of the phase section.

As one embodiment, when a new phase section is generated by the division by the division unit 32, the section feature amount calculation unit 33 calculates the section feature amount from the sensor data measured in the phase section for each phase section. FIG. 3 is a diagram showing an example of a section feature amount. FIG. 3 shows section features related to meals. FIG. 3 shows sensor data measured in a situation where a series of meal phases such as carving food, carrying food to the mouth, and chewing are repeated twice. Further, FIG. 3 shows two waveforms, that is, the time-series data of the height in the vertical direction and the time-series data of the acceleration in the horizontal direction as an example of the sensor data. As shown in FIG. 3, when the meal is separated, the characteristic tends to appear in the fluctuation range of the frequency and the amplitude of the horizontal acceleration. Therefore, the fluctuation range of the frequency and the amplitude of the acceleration in the horizontal direction can be calculated as the interval feature amount. Also, when food is brought to the mouth, the feature tends to appear at the maximum height in the vertical direction. Therefore, the maximum height in the vertical direction can be calculated as the section feature amount. Furthermore, when chewing, the characteristics tend to appear in the section length itself. Therefore, the section length of the height in the vertical direction can be calculated as the section feature amount.

Each time the section feature amount is calculated in this way, the section feature amount calculation unit 33 stores each type of section feature amount in an internal memory (not shown) in association with the phase section. In this internal memory, a section feature amount of a phase section for a predetermined period, for example, one minute, is left retroactively from the new phase section. Although FIG. 3 shows the section feature amounts for which features are likely to appear for each phase type, all types of section feature amounts are calculated for each phase section regardless of the phase type to which the phase section belongs.

The first calculation unit 34 is a processing unit that calculates a first phase feature amount for at least one section feature amount for a plurality of first sections. The "section feature amount" referred to here corresponds to an example of the parameter, and the "first phase feature amount" corresponds to an example of the first feature amount.

As one embodiment, the first calculation unit 34 starts the process when the section feature amount of the new phase section is calculated by the section feature amount calculation unit 33. That is, the first calculation unit 34 is the section feature amount stored in the internal memory for a predetermined period, for example, one minute, retroactively from the new phase section, and the phase type is ". Read out the section features that satisfy the condition of "movement in the vertical direction". Then, the first calculation unit 34 belongs to the phase type "movement in the vertical direction" by counting up the frequency of the class to which the section feature amount belongs for each section feature amount read from the internal memory. Generate a distribution of interval features. After that, the first calculation unit 34 calculates the statistical value of the distribution of the section feature amount as the first phase feature amount. As an example, the "statistical value" referred to here includes a mean, a standard deviation, a median value, a maximum value, a minimum value, and the like.

The second calculation unit 35 is a processing unit that calculates a second phase feature amount for at least one section feature amount for a plurality of second sections. The "section feature amount" referred to here corresponds to an example of the parameter, and the "second phase feature amount" corresponds to an example of the second feature amount.

As one embodiment, the second calculation unit 35 starts the process when the section feature amount is calculated by the section feature amount calculation unit 33. That is, the second calculation unit 35 is the section feature amount stored in the internal memory for a predetermined period, for example, one minute retroactively from the new phase section, and the phase type is ". Read out the interval features that satisfy the condition of "horizontal movement". Then, the second calculation unit 35 belongs to the phase type "horizontal movement" by counting up the frequency of the class to which the section feature amount belongs for each section feature amount read from the internal memory. Generate a distribution of interval features. After that, the second calculation unit 35 calculates the statistical value of the distribution of the section feature amount as the second phase feature amount.

The third calculation unit 36 is a processing unit that calculates a third phase feature amount for at least one section feature amount for a plurality of third sections.

As one embodiment, the third calculation unit 36 activates the process when the section feature amount is calculated by the section feature amount calculation unit 33. That is, the third calculation unit 36 is a phase section stored in a predetermined period, for example, one minute retroactively from the new phase section among the section feature quantities stored in the internal memory, and the phase type is "stationary". The interval feature amount that satisfies the condition of "" is read out. Then, the third calculation unit 36 counts up the frequency of the class to which the section feature amount belongs for each section feature amount read from the internal memory, so that the section feature amount belonging to the phase type "stationary" is counted up. Generate a distribution of. After that, the third calculation unit 36 calculates the statistical value of the distribution of the section feature amount as the third phase feature amount.

In this way, the first calculation unit 34 to the third calculation unit 36 generate the distribution of the section feature amount belonging to the phase type for each phase type, and the statistical value of the distribution is used for the first phase feature amount to the first phase type. Calculated as the three-phase feature quantity. Further, when calculating the first phase feature amount to the third phase feature amount, the first calculation unit 34 to the third calculation unit 36 can also calculate after deleting the end portion of the distribution. For example, in the first calculation unit 34 to the third calculation unit 36, the first phase feature amount to the third phase are obtained from the distribution of the section feature amount in which the portions corresponding to the upper 10% and the lower 10% are deleted. The feature amount can be calculated. Further, when generating a distribution of interval features, a given interval feature can be given as a weight. For example, when a phase section with a variable time length is divided, the interval feature amount of the phase section with a longer time length is distributed by generating a distribution of the section feature amount with the time length of the phase section as a weight. It does not matter if the effect is large. When there are a plurality of types of section features, the first calculation unit 34 to the third calculation unit 36 generate a distribution of the section features for each type of the section features, and statistics on the distribution. Needless to say, the value is calculated as the first phase feature amount to the third phase feature amount.

The detection unit 37 is a processing unit that detects the behavior of a new phase section by using the phase feature amount and the model learned by machine learning.

As one embodiment, the detection unit 37 determines the activity of the subject in a certain section based on the first determination model related to at least one section feature amount, the first phase feature amount, and the second phase feature amount. However, it detects a specific behavior indicated by the judgment model. The "judgment model" here refers to a procedure such as a condition for judging various phase features or a combination of a plurality of conditions, and as an example, a discriminator whose phase features are learned by machine learning or a discriminator. Behavior judgment models such as classifiers are included.

For example, the detection unit 37 may input a feature amount vector in which the first phase feature amount, the second phase feature amount, and the third phase feature amount are vectorized for each type of the interval feature amount to express the likeness of eating. Using the behavior judgment model that outputs the degree, the presence or absence of meal in the new phase section is detected. For example, when the behavior determination model outputs a value from -1 to +1 and the output value from the behavior determination model is a predetermined value, for example, "0" or more, the detection unit 37 sets the new phase section. Classify into meals. The behavior judgment model is an example of the judgment model.

For machine learning of this behavior determination model, algorithms such as Deep Learning, SVM (Support Vector Machine), decision tree, Random Forest, and nearest neighbor can be used as an example. This machine learning accepts inputs such as a meal period, that is, a meal start time and an end time, or a meal start time and a meal required time. As a result, the behavior type label "meal" is given to the first phase feature amount to the third phase feature amount included in the meal period, while the first phase feature amount included outside the meal period to the third phase feature amount. The behavior type label "non-meal" can be given to the three phase features. Furthermore, by using the first phase feature amount to the third phase feature amount obtained from the sensor data measured with the chopsticks held in the dominant hand as the teacher data for machine learning, an action judgment model for chopsticks is used. For knives by using the first phase features to the third phase features obtained from the sensor data measured with the knife in the dominant hand as the teacher data for machine learning. It is possible to generate an action judgment model.

Further, the detection unit 37 does not necessarily have to detect the behavior by using machine learning. That is, the detection unit 37 can also detect a specific action by executing the condition determination according to the following procedure. For example, in the detection unit 37, the first phase feature amount "average value of the maximum height in the vertical direction" regarding the phase type "movement in the vertical direction" is equal to or more than a predetermined threshold value, and the phase type "movement in the horizontal direction" is used. It is determined whether or not the second phase feature amount "dispersion of the fluctuation width of the amplitude of the horizontal acceleration" is equal to or more than a predetermined threshold value. Then, the detection unit 37 may detect the new phase section as a meal when both are equal to or higher than the threshold value. Further, as an important matter, the detection unit 37 may further determine whether or not the third phase feature amount “average of section lengths” regarding the phase type “stationary” is equal to or less than a predetermined threshold value. In this case, the detection unit 37 detects a new phase section as a meal when the first phase feature amount and the second phase feature amount are equal to or more than the threshold value and the third phase feature amount is equal to or less than the threshold value. It doesn't matter what you do. The above condition determination procedure is an example of a determination model.

Further, when the first phase feature amount, the second phase feature amount, or the third phase feature amount satisfy a predetermined condition, the detection unit 37 may use the conditions used by the detection unit 37 for determination, the type of machine learning, or the machine. It is also possible to change the learning model. For example, when a meal is detected as an example of a specific behavior, the detection unit 37 has a predetermined threshold value of the second phase feature amount “average of the magnitude of movement in the front-back direction” regarding the phase type “horizontal movement”. It is determined whether or not it is the above. At this time, when the second phase feature amount "average of the magnitude of the movement in the front-back direction" regarding the phase type "movement in the horizontal direction" is equal to or more than the threshold value, the wrist of the dominant arm is greatly moving in the front-back direction, so that the dominant hand is used. You can estimate the situation where you have a knife. In this case, as an example, the detection unit 37 can change the behavior determination model used by the detection unit 37 for behavior detection from the behavior determination model for chopsticks to the behavior determination model for knives. In addition, when the second phase feature amount "average of the maximum heights of vertical movement" related to the phase type "movement in the vertical direction" is equal to or greater than the threshold value, the wrist of the dominant arm is greatly moving in the vertical direction, so that the dominant hand is used. You can estimate the situation where you have chopsticks or forks. In this case, as an example, the detection unit 37 can change the behavior determination model used by the detection unit 37 for behavior detection from the behavior determination model for knives to the behavior determination model for chopsticks.

Further, the detection unit 37 can change a part of the detection results when the detection result for each phase section, that is, the arrangement of the meal or non-meal classification results satisfies a predetermined condition. For example, a series of phases included in a meal is repeated a certain number of times even if the amount of food and drink is small. Therefore, if only a small number of phase sections are isolated and detected as a meal, the possibility of false detection increases. From this, the detection unit 37 classifies the phase section classified as the action type "meal" when the number of times the phase section is continuously classified into the action type "meal" is less than a predetermined number of times, for example, three times. The result can be updated to the action type "non-meal". Similarly, if the previous and next phase sections are classified as the action type "meal" and only the middle phase section is classified as the action type "non-meal", the middle phase section can be erroneously detected. Sex increases. Therefore, the detection unit 37 can update the classification result of the phase section of the action type "non-meal" sandwiched between the phase sections in which the action type before and after is "meal" to the action type "meal".

After that, the detection unit 37 outputs the classification result of the action type detected for each phase section to a computer, AI, or a computer outside the server device 30 that executes functions such as action accumulation and recommendation output on the back end. .. As a result, the above-mentioned healthcare support service is realized.

(1) Specific behavior detection process FIG. 4 is a flowchart showing a procedure of the specific behavior detection process according to the first embodiment. As an example, this process is repeatedly executed every time sensor data is acquired from the sensor 11 mounted on the wearable terminal 10. As shown in FIG. 4, the acquisition unit 31 acquires sensor data from the sensor 11 mounted on the wearable terminal 10 (step S101).

[Example of sensor data]
FIG. 5 is a diagram showing an example of sensor data. FIG. 5 shows an example of the sensor data acquired in step S101. As shown in FIG. 5, as the sensor data, data in which items such as acceleration and height are associated with each measurement date and time can be adopted. The "measurement date and time" here refers to the date and time when the sensor data was measured. Further, the "acceleration" refers to the acceleration measured by the sensor 11, and for example, the acceleration in the left-right direction, the front-back direction, and the up-down direction is measured. Further, the "height" refers to the displacement from the height at the start of measurement, and can be obtained, for example, by integrating the difference between the acceleration in the gravitational direction and the gravitational acceleration twice. As shown in FIG. 5, it can be seen that the sensor data sampled at a cycle of 1 second is acquired by the sensor 11. The sensor data acquired in this way can be stored in an internal storage (not shown) as a history.

Subsequently, the division unit 32 divides the sensor data acquired in step S101 into phase sections to which the phase type is assigned (step S102).

[Example of phase section]
FIG. 6 is a diagram showing an example of a phase section. FIG. 6 shows an example in which the time length of the phase interval divided from the sensor data is variable. As shown in FIG. 6, the phase section can be defined by items such as a section ID (IDentification), a section start date and time, a section end date and time, and a phase type. The "section ID" referred to here refers to identification information that identifies a phase section, and for example, serial numbers of natural numbers are numbered in ascending order in the order of occurrence of the phase section. Further, the "section start date and time" refers to the date and time when the phase section starts, and the "section end date and time" refers to the date and time when the phase section ends. The "phase type" refers to the type of movement included in the action. For example, in the case of a meal, the "horizontal" phase in which the food to be ingested is grabbed in one bite, and the "up and down" in which the food is brought back to the mouth. There is a "direction" phase, a "stationary" phase in which movement stops during chewing or swallowing (or conversation), and the like.

When a new phase section occurs in step S102 (step S103Yes), the section feature amount calculation unit 33 determines the horizontal frequency, the horizontal size, and the maximum height in the vertical direction from the sensor data measured in the new phase section. The section feature amount such as the height and the section length is calculated (step S104). If a new phase section does not occur (step S103 No), the process returns to step S101.

[Example of section features]
FIG. 7 is a diagram showing an example of a section feature amount. As shown in FIG. 7, the section ID and the phase type are associated with the section features such as the horizontal frequency, the horizontal size, the maximum height in the vertical direction, and the section length calculated by the section feature calculation unit 33. In this state, it is added to the internal memory and saved as a new record of the section feature amount history 41.

After that, the first calculation unit 34 to the third calculation unit 36 of the section feature amount history 41 stored in the internal memory, trace back from the new phase section to a predetermined period, for example, a 1-minute phase section. Read the feature amount record (step S105).

Then, in the first calculation unit 34, among the section feature amounts read out in step S105, the section feature amount of the type belongs to each type of the section feature amount whose phase type is "movement in the vertical direction". By counting up the frequency of the classes, the distribution of the interval features is generated (step S106). After that, the first calculation unit 34 calculates the statistical value of the distribution of the interval feature amount calculated in step S106 as the first phase feature amount (step S107).

In parallel with the first calculation unit 34, the second calculation unit 35 corresponds to each type of section feature amount whose phase type is "horizontal movement" among the section feature amounts read in step S105. By counting up the frequency of the class to which the section features of the type belong, the distribution of the section features is generated (step S106). After that, the second calculation unit 35 calculates the statistical value of the distribution of the interval feature amount calculated in step S106 as the second phase feature amount (step S107).

Further, in parallel with the second calculation unit 35, the third calculation unit 36 performs the type of the section feature amount read in step S105 for each type of the section feature amount whose phase type is “stationary”. By counting up the frequency of the class to which the section feature amount belongs, the distribution of the section feature amount is generated (step S106). After that, the third calculation unit 36 calculates the statistical value of the distribution of the interval feature amount calculated in step S106 as the third phase feature amount (step S107).

[Example of phase features]
As described in steps S105 to S107, the phase feature amount for each type of section feature amount is calculated for each phase type. FIG. 8 is a diagram showing an example of phase features. As shown in FIG. 8, the phase feature amount is calculated for each phase type, section feature amount type, and statistical value type. The "ID of the latest section" referred to here refers to the ID of the phase section that was the latest when the phase feature amount was calculated. For example, when the phase section of the section ID "102" occurs as a new phase section and the section feature amount of the phase section is calculated, it goes back from the section ID "102" to a predetermined period, for example, a phase section of 1 minute. The record of the section feature amount of is read from the section feature amount history 41 in the internal memory. Using the section feature amount read out in this way, there are three phase types, "horizontal", "vertical" and "stationary", "horizontal frequency", "horizontal magnitude", and "upper and lower". Phase features are calculated for each of the four types of section features, "maximum height in direction" and "section length," and for each of the two types of statistics, "average" and "dispersion." For example, when the phase features related to the phase type "horizontal", the section feature type "horizontal frequency", and the statistical value type "average" are calculated, the section feature history 41 stored in the internal memory is used. After the records of the section features having the value of "horizontal direction" in the phase type are extracted from the records of the section features read out, the horizontal frequencies are averaged among the extracted records. , The phase feature amount is calculated. Note that FIG. 8 shows excerpts of the phase features related to the "horizontal direction" among the three phase types, but the phase features related to the "vertical direction" and "stationary" are also stored after calculation. NS.

After that, the detection unit 37 inputs the feature amount vector in which the first phase feature amount, the second phase feature amount, and the third phase feature amount are vectorized for each type of the interval feature amount, and determines the certainty of the action. Using the action determination model 42 that outputs the represented likelihood, the presence or absence of an action in the new phase section is detected (step S108), and the process is terminated.

(2) Model learning process FIG. 9 is a flowchart showing a procedure for learning process of the model according to the first embodiment. This process is started, for example, when the sensor data to the extent that the learning accuracy can be sufficiently exhibited is accumulated in the internal storage.

As shown in FIG. 9, the acquisition unit 31 acquires the sensor data history 43 stored in the internal storage (step S201). Subsequently, the division unit 32 divides the sensor data acquired in step S201 into phase sections to which the phase type is assigned (step S202).

Then, when a new phase section occurs (step S203Yes), the section feature amount calculation unit 33 determines the horizontal frequency, the horizontal size, and the maximum height in the vertical direction from the sensor data measured in the new phase section. And the section feature amount such as the section length is calculated (step S204). The section feature amount calculated in this way is added to and saved in the section feature amount history 41 stored in the internal memory. If a new phase section does not occur (step S203 No), the process proceeds to step S209.

After that, the first calculation unit 34 to the third calculation unit 36 of the section feature amount history 41 stored in the internal memory, trace back from the new phase section to a predetermined period, for example, a 1-minute phase section. Read the feature amount record (step S205).

Then, in the first calculation unit 34, among the section feature amounts read out in step S205, the section feature amount of the type belongs to each type of the section feature amount whose phase type is "movement in the vertical direction". By counting up the frequency of the classes, the distribution of the interval features is generated (step S206). After that, the first calculation unit 34 calculates the statistical value of the distribution of the interval feature amount calculated in step S206 as the first phase feature amount (step S207).

In parallel with the first calculation unit 34, the second calculation unit 35 corresponds to each type of section feature amount whose phase type is "horizontal movement" among the section feature amounts read in step S205. By counting up the frequency of the class to which the section features of the type belong, the distribution of the section features is generated (step S206). After that, the second calculation unit 35 calculates the statistical value of the distribution of the interval feature amount calculated in step S206 as the second phase feature amount (step S207).

Further, in parallel with the second calculation unit 35, the third calculation unit 36 determines the type of the section feature amount read in step S205 for each type of the section feature amount whose phase type is “stationary”. By counting up the frequency of the class to which the section feature amount belongs, the distribution of the section feature amount is generated (step S206). After that, the third calculation unit 36 calculates the statistical value of the distribution of the interval feature amount calculated in step S206 as the third phase feature amount (step S207).

[Example of phase feature + action type history]
As described in steps S205 to S207, the phase feature amount for each type of section feature amount is calculated for each phase type. Each record of the phase feature amount calculated in this way is given a label of the action type according to the action type history 44 in which the external input is performed. As a result, the phase feature amount + action type 45, which is a combination of the phase feature amount record and the action type label, is added and stored in the storage device in the storage device such as the internal memory or the internal storage in record units (as a result). Step S208). For example, the action type history 44 is input data regarding a history of a meal period, that is, a history of a meal such as a meal start time and an end time, or a meal start time and a meal required time. That is, the phase feature amount included in the meal period from the start time of the meal to the end time of the meal specified from the behavior type history 44 is given the behavior type label "meal", while the meal period is covered. Phase feature quantities that are not included are given the behavior type label "non-meal". FIG. 10 is a diagram showing an example of phase features. FIG. 10 shows a phase feature amount labeled with an action type according to the action type history 44. The phase feature amount shown in FIG. 10 is different from the phase feature amount of FIG. 8 used at the time of behavior detection in that a label of action type is given.

After that, the processes from step S201 to step S208 are repeatedly executed until the sensor data stored in the internal storage is completed (step S209No). Then, when the sensor data stored in the internal storage is completed (step S209Yes), the detection unit 37 generates an action determination model 42 by machine learning using the phase feature quantity labeled with the action type shown in FIG. (Step S210), the process is terminated.

[One aspect of the effect]
As described above, the server device 30 according to the present embodiment obtains the characteristics of the distribution regarding the section feature amount obtained for each phase section in which the sensor data is divided into the action phases, and obtains the characteristics of the distribution for each phase type. Specific behavior is detected based on the characteristics of the distribution and the judgment model. As described above, the server device 30 according to the present embodiment repeatedly captures the characteristics of each phase type included in the action and detects the action. Therefore, it is possible to improve the detection accuracy of a specific action such as a meal, as compared with the case where the action detection is performed only by a part of the action as in the technique described in the background technique section.

[Another example of specific behavior]
In Example 1 above, the case where meals are detected is illustrated as an example of behavior, but in addition to meals, daily activities such as excretion, cleaning, changing clothes, movement, dressing, and bathing, and further , How to use the phone, shopping, housework, traveling, going out, managing medication, managing money, etc. can be detected.

As an example other than meals, the case where dishwashing is detected will be illustrated. Even when the dishwashing is detected in this way, the dishwashing detection can be realized by using the wearable terminal 10 in which the sensor 11 such as the acceleration sensor is mounted on the wrist of the dominant arm. That is, the division unit 32 can assign the following phase types to the phase sections divided from the sensor data. For example, the phase of washing dishes with detergent and rinsing with water and the phase of cleaning dishes can be mentioned. In the former phase type, for example, the situation where the wrist is facing downward and the wrist position does not move much is the vertical displacement observed in the sensor data, that is, the maximum and minimum heights. It is determined by whether or not the difference is equal to or less than a predetermined threshold and the height is equal to or less than a predetermined threshold. The latter phase type is, for example, depending on whether or not the sum of squares of the vertical, anteroposterior, and lateral accelerations observed in the sensor data is equal to or more than a predetermined threshold in the situation including the vertical and horizontal movements. , Is determined. In addition, the section feature amount calculation unit 33 uses the section feature amount for dishwashing as the maximum movement distance for a certain period (several seconds), the frequency and magnitude of movement for a certain period (several seconds), the direction of the forearm, and the section length. Etc. can be calculated. Further, the first calculation unit 34 to the third calculation unit 36 indicate the average and variance of the maximum travel distance during a certain period (several seconds), and the frequency of movement during a certain period (several seconds) for each phase type. The average and variance of the size, the average and variance of the orientation of the forearm, and the average and variance of the section length can be calculated as the phase features. With such phase features, it is possible to detect whether or not dishwashing is performed by machine learning or the like.

By the way, in the above-mentioned Example 1, an example of detecting various behaviors has been described, but in this Example, the quantification of the section feature amount and the phase feature amount related to the meal will be described more specifically.

FIG. 11 is a diagram showing an example of a functional configuration of each device included in the health care support system according to the second embodiment. In the health care support system 2 shown in FIG. 11, the wearable terminal 50 is equipped with the first sensor 51 and the second sensor 52. Of these, as the first sensor 51, as an example, a three-axis angular velocity sensor, a so-called gyro sensor, can be adopted. In the following, the time-series data of the angular velocity output from the 3-axis angular velocity sensor will be referred to as "angular velocity data", and the time-series data of the acceleration output from the 3-axis acceleration sensor will be referred to as "acceleration data". May be described as. Further, a 3-axis accelerometer can be adopted as the second sensor 52. These angular velocity sensors and acceleration sensors may be mounted as IC-based motion sensors on one chip. Further, the wearable terminal 50 does not necessarily have to be equipped with the second sensor 52.

Here, the server device 70 according to the present embodiment has an advantage in that the characteristics of the following operations (1) to (8) that repeatedly appear during meals can be quantified. That is, the following movements (1) to (8) are focused on the point that they are likely to occur when some work is performed with the elbow fixed, or the forearm is fixed and moved back and forth.

(1) The number of times a large rotation occurs in a short time in the direction of twisting the wrist of the arm is relatively large compared to the number of times a large rotation occurs in a short time in the rotation directions of the other two axes.
(2) The angle of the arm closest to the axis in the direction of gravity during meals is when food is put into the mouth. The angle is approximately constant each time, and the angle formed by the axis in the direction of gravity is larger than 0 degrees.
(3) The arm is often stationary during chewing and swallowing, and the angle of the stationary arm is approximately constant each time. In particular, when the rest time is close, it becomes almost constant. In addition, the angle is closer to horizontal with respect to the ground than the angle at which the ingested food is put into the mouth.
(4) The duration of the arm moving state and the resting state is often short. (The arm is often stationary during the movement of chewing and swallowing)
(5) In many cases, there is a large change in the angle of the arm before and after the arm shifts from the motion to the stationary state.
(6) The movement of the movement is not constant because the type of food changes and the way of eating changes depending on the amount.
(7) The angle of the forearm with respect to the ground is often the closest to the horizontal while the arm is performing a series of movements.
(8) It is approximately perpendicular to the ground and frequently moves back and forth in the direction of the bones of the arm.

As shown in FIG. 11, the server device 70 includes a first acquisition unit 71, a second acquisition unit 72, a division unit 73, a section feature amount calculation unit 74-1 and a section feature amount calculation unit 74-. 2. It has a phase feature amount calculation unit 75-1, a phase feature amount calculation unit 75-2, and a detection unit 76.

The first acquisition unit 71 is a processing unit for acquiring angular velocity data, and the second acquisition unit 72 is a processing unit for acquiring acceleration data.

The division unit 73 corresponds to the division unit 32 shown in FIG. 1 and has a common point of dividing the sensor data into phase sections, but a part of the processing content is different in the following points. That is, the division unit 73 determines whether or not there is movement in the arm on which the wearable terminal 50 is attached from the value of the angular velocity data acquired by the first acquisition unit 71. This identifies whether the arm is in motion or the arm is stationary.

For example, the division unit 73 determines whether or not the maximum angular velocity of the three axes included in the angular velocity data, for example, 100 msec, is 1 deg / s or more. As a result, when the maximum angular velocity is 1 deg / s or more, the division unit 73 distinguishes the above period from the operation section. Further, when the maximum angular velocity is not 1 deg / s or more, the dividing unit 73 further determines whether or not the maximum angular velocity is 0.5 deg / s or less. At this time, when the maximum angular velocity is 0.5 deg / s or less, the division unit 73 identifies the above period as a stationary section. Further, when the maximum angular velocity is less than 1 deg / s and exceeds 0.5 deg / s, the division unit 73 inherits the result of the determination made immediately before. This makes it possible to identify an operating section or a stationary section. FIG. 12 is a diagram showing an example of a phase type. As shown in FIG. 12, by using the maximum angular velocity in the past predetermined period, it is possible to identify whether the phase type corresponds to the operating section or the stationary section. As a result, it is possible to distinguish between Phase A, in which the food to be ingested is grabbed in one bite, Phase B, in which the food is brought back to the mouth, and Phase C, in which the food is stopped during chewing and swallowing.

At this time, when the division unit 73 is in an environment where the acceleration data can be acquired together with the angular velocity data, the division unit 73 may correct the identification result of the operating section or the stationary section by using the value of the acceleration data. That is, the division unit 73 calculates the absolute value of the value of the acceleration sensor of the axis parallel to the bone of the arm, and obtains the maximum value between the moving section and the stationary section identified from the value of the acceleration sensor. Then, the division unit 73 clusters the maximum values of the operating section and the stationary section included in a predetermined time into two clusters. In addition, the division unit 73 corrects the identification result of the stationary section included in the cluster having the maximum value of the operating section to the operating section, which is not included in the cluster containing a large amount of data of the maximum value of the stationary section. FIG. 13 is a diagram showing an example of the maximum value of acceleration in the operating section and the stationary section. In FIG. 13, the maximum value of the acceleration in the operating section is plotted with a black circle on the time axis, and the maximum value of the acceleration in the stationary section is plotted with a white circle on the time axis. As shown in FIG. 13, during a meal, the maximum angular velocity of 0.5 deg / s or less does not necessarily mean that the arm is stationary on a desk or the like, and the arm is temporarily stationary when chewing or swallowing. May be identified as a stationary section. By the above clustering, the period of temporary rest can be identified as the operating section by correcting the identification result of the stationary section included in the cluster of the maximum acceleration value to the operating section. Further, the division unit 73 obtains the variance of the combined acceleration obtained by synthesizing the accelerations of the three axes from the acceleration data measured in the stationary section, and if the variance is large, the process of correcting the identification result of the stationary section to the operating section. Can also be added.

Further, the division unit 73 can further classify the operation section into two phase types. As shown in FIG. 12, since the motion section includes a phase A for grasping the food to be ingested in one bite and a phase B for bringing the food back to the mouth, the division unit 73 further divides the motion section into the phase A. And phase B. That is, the division unit 73 can identify the phase type of the operation section classified into the cluster having many stationary sections as the phase A even though it is identified as the operation section. Further, the division unit 73 classifies the acceleration of the operating section into one or two. For example, the phase type of the phase section containing the maximum value is identified as phase B, and the phase type of the phase section not including the maximum value is identified as phase A. For example, the division unit 73 determines which of the two clusters the absolute value of the acceleration data of the axis parallel to the axis of the arm bone is closer to among the acceleration data measured in the motion section, and classifies the cluster into which cluster. Depending on what has been done, the time of the operation section can be classified into one or two. In this case, the phase type in the first half of the section classified into the two is identified as phase A, and the phase type in the latter half is identified as phase B. In addition, the phase type of the section that has become one classification can be identified as Phase B. In addition, the hidden Markov model can be used to classify the phase types of Phase A and Phase B.

The section feature amount calculation unit 74-1 and the section feature amount calculation unit 74-2 both correspond to the section feature amount calculation unit 33 and have a common point of calculating the section feature amount, but the processing contents are as follows. Some of them are different. That is, the section feature amount calculation unit 74-1 calculates the section feature amount using the angular velocity, while the section feature amount calculation unit 74-2 calculates the section feature amount using the acceleration.

Further, the phase feature amount calculation unit 75-1 and the phase feature amount calculation unit 75-2 both correspond to the first calculation unit 34 to the third calculation unit 36, and have in common that the phase feature amount is calculated. However, some of the processing contents are different in the following points. That is, the phase feature amount calculation unit 75-1 calculates the phase feature amount using the section feature amount related to the angular velocity, while the phase feature amount calculation unit 75-2 uses the section feature amount related to the acceleration to calculate the phase feature amount. Is calculated.

FIG. 14 is a diagram showing an example of phase features. The section feature amount calculation unit 74-1 and the phase feature amount calculation unit 75-1 calculate the angular velocity feature amount 1 to the angular velocity feature amount 3 among the phase feature amounts shown in FIG. Further, the section feature amount calculation unit 74-2 and the phase feature amount calculation unit 75-2 calculate the acceleration feature amount 1 to the acceleration feature amount 4 among the phase feature amounts shown in FIG.

[Angular velocity feature 1]
The angular velocity feature amount 1 is an index in which "the frequency with which a large arm rotation occurs in a short time on each axis" is quantified. With this angular velocity feature amount 1, the following features can be grasped as numerical values. For example, you can capture the characteristics of various movements such as tilting a cup when drinking a drink, scraping rice with chopsticks, scooping soup with a spoon, and adding sauce to foodstuffs stuck in a fork with a knife. From a different point of view, it is also a feature that captures the aspect that a large amount of rotation cannot be seen in a short time during meals in the axial direction other than twisting the wrist. Therefore, the number of rotations of the other two axes is smaller than the number of rotations in the direction of twisting the wrist.

Such an angular velocity feature amount 1 can be calculated as follows. That is, the section feature amount calculation unit 74-1 takes the absolute value of the angular velocity of each axis of the operation section, and if the integrated value within a predetermined period in the past, for example, 1 second is a predetermined value, for example, 50 deg or more, the arm Is considered to have rotated, and the number of times is calculated as the interval feature amount. Then, the phase feature amount calculation unit 75-1 divides the total number of times calculated as the section feature amount by the predetermined time in a predetermined time, for example, one minute, and divides the value by the predetermined time to obtain the above-mentioned angular velocity feature amount 1. Calculated as. Further, the phase feature amount calculation unit 75-1 uses the angular velocity feature amount 1 calculated for each axis to set the difference of the angular velocity feature amount 1 between the axis in the direction of twisting the wrist and the other two axes, and the angular velocity feature amount 1-. It can be calculated as 2, or the minimum value of the axis in the direction of twisting the wrist and the other two axes can be calculated as the angular velocity feature amount 1-3.

[Angular velocity feature 2]
The angular velocity feature amount 2 is an index in which "typical lengths of each of the moving section and the stationary section" are quantified. With this angular velocity feature amount 2, the following features can be grasped as numerical values. For example, the flow of making a meal sized to fit in the mouth, putting it in the mouth, and keeping the arm stationary while chewing or swallowing occurs repeatedly during the meal, and the time when the arm is moving is also stationary. The time is also short, both of which are characteristic. In the case of behaviors other than eating, it is relatively common for the arms to move all the time.

Such an angular velocity feature amount 2 can be calculated as follows. That is, the section feature amount calculation unit 74-1 calculates the lengths of the operating section and the stationary section classified by the dividing unit 73 within a predetermined time as the section feature amount. Then, the phase feature amount calculation unit 75-1 calculates the median value of the section length as the above-mentioned angular velocity feature amount 2, or 25% from the smallest in the order in which the section length is arranged from the smallest. It is possible to calculate the value of 75% from the smallest value in the order of arrangement from the smallest length of the section or the length of the section as the angular velocity feature amount 2.

[Angular velocity feature 3]
The angular velocity feature amount 3 is an index in which "the degree of similarity between the operating section and the operating section" is quantified. With this angular velocity feature amount 3, the following features can be grasped as numerical values. For example, hand movements during a meal usually change with the progress of the meal. For example, the movement of the hand is different between the appetizer and the main dish, and the movement of the hand changes depending on the amount of one meal. In addition, since drinks are taken between intervals, this also changes. Therefore, it captures the characteristic that the movements seen during meals are not the same movement.

Such an angular velocity feature amount 3 can be calculated as follows. That is, the section feature amount calculation unit 74-1 calculates the DTW (Dynamic Time Warping) distance by combining the two in the section determined as the operation section by the division unit 73 within a predetermined time, and determines the distance as the section feature amount. Calculated as. Then, the phase feature amount calculation unit 75-1 calculates the average value of the section features of all combinations calculated in the section determined as the operation section by the division unit 73 within the predetermined time as the angular velocity feature amount 3. can.

[Acceleration feature 1]
The acceleration feature amount 1 is an index in which "the distribution of the degree of rise in the state where the arm is most raised in the motion section" is quantified. With this acceleration feature amount 1, the following features can be grasped as numerical values. For example, the scene where the arm is raised most (perpendicular to the ground) during a meal is the moment when the meal is eaten. The degree of arm rise at that time does not change significantly with food, but captures the characteristic that it is approximately constant.

Such an acceleration feature amount 1 can be calculated as follows. That is, the section feature amount calculation unit 74-2 takes the absolute value of the axis of the accelerometer parallel to the arm in each section determined to be the operation section by the division unit 73, and calculates the maximum value as the section feature amount. do. Then, the phase feature amount calculation unit 75-2 calculates the histogram of the maximum value within a predetermined time, and calculates the maximum frequency point and the variance as the acceleration feature amount 1.

[Acceleration feature 2]
The acceleration feature amount 2 is an index in which the "distribution of the degree of arm rise in the stationary section" is quantified. In this acceleration feature amount 2, the following features can be grasped as numerical values. For example, the flow of making a meal sized to fit in the mouth, putting it in the mouth, and the arm resting while chewing or swallowing occurs repeatedly during the meal, and the posture of the arm during the resting time is the meal. It captures the characteristic that it is almost constant.

Such an acceleration feature amount 2 can be calculated as follows. That is, the section feature amount calculation unit 74-2 takes the absolute value of the accelerometer axis parallel to the arm in each section determined to be a stationary section by the division unit 73, and calculates the maximum value as the section feature amount. .. Then, the phase feature amount calculation unit 75-2 calculates the histogram of the maximum value within a predetermined time, and calculates the maximum frequency point and the variance as the acceleration feature amount 2.

[Acceleration feature 3]
The acceleration feature amount 3 is an index in which "the frequency with which the vertical movement of the arm changes significantly before and after the transition from the motion section to the stationary section" is quantified. In this acceleration feature amount 3, the following features can be grasped as numerical values. For example, the flow of making a meal sized to fit in the mouth, putting it in the mouth, and resting the arm while chewing or swallowing occurs repeatedly during the meal. Therefore, it captures the characteristic that there is a high probability that an action such as bringing the arm to the mouth and then lowering the arm occurs before the stationary section.

Such an acceleration feature amount 3 can be calculated as follows. That is, the section feature amount calculation unit 74-2 has the maximum and minimum absolute values of acceleration on the same axis as the axis of the arm bone at a predetermined time before and after the transition from the motion section to the stationary section, for example, 2 seconds. The difference is calculated as the interval feature amount. Then, the phase feature amount calculation unit 75-2 generates a histogram of the difference which is the section feature amount, and calculates the ratio of the number of changes of 0.2 g or more as the acceleration feature amount 3.

[Acceleration feature 4]
The acceleration feature amount 4 is an index in which "the difference in the typical arm rise between the acceleration feature amount 1 and the acceleration feature amount 2" is quantified. In this acceleration feature amount 4, the following features can be grasped as numerical values. For example, the flow of making a meal sized to fit in the mouth, putting it in the mouth, and resting the arm while chewing or swallowing occurs repeatedly during the meal. Among them, the posture of the arm when it is put in the mouth and the posture of the arm when it is stationary captures the feature that the posture of the arm when it is stationary is closer to parallel to the ground. ..

Such an acceleration feature amount 4 can be calculated as follows. That is, the section feature amount calculation unit 74-2 calculates the section feature amount in the same manner as the above-mentioned acceleration feature amount 1 and the above-mentioned acceleration feature amount 2. Then, the phase feature amount calculation unit 75-2 calculates the acceleration feature amount 1 and the acceleration feature amount 2, and calculates the difference as the acceleration feature amount 4.

[Robust detection]
In these angular velocity features 1 and acceleration features 1 to 4, absolute values are used in the calculation process. As a result, the acceleration and angular velocity of the three axes are changed only from the features that are invariant to the exchange of the positive and negative of the acceleration and the angular velocity centered on the axis of the bone of the arm, or the exchange of the values of the other two axes. Detection can be performed. FIG. 15 is a diagram showing an example of the appearance of the wearable terminal 50. FIG. 15 shows a wristband type wearable terminal 50 as an example. Even if the wearable terminal 50 shown in FIG. 15 is mounted with the back side of the hand facing the front side, or even if the wearable terminal 50 is mounted facing the back side of the hand, the meal detection can be performed. Further, even if the orientation of the wearable terminal 50 when the wearable terminal 50 shown in FIG. 15 is passed through the arm is different, the meal can be detected. That is, meal detection can be performed regardless of whether the wearable terminal 50 is passed through the arm of the target person from the direction of D1 or the wearable terminal 50 is passed through the arm of the target person from the direction of D2.

The detection unit 76 takes an input of a feature amount vector in which at least one of the above-mentioned angular velocity feature amounts 1 to 3 and the above-mentioned acceleration feature amounts 1 to 4 is vectorized, and outputs an action determination indicating a likelihood of eating. The model is used to detect the presence or absence of food in the new phase section. For example, when the behavior determination model outputs a value from -1 to +1 and the output value from the behavior determination model is a predetermined value, for example, "0" or more, the detection unit 76 determines the new phase section. Classify into meals.

Further, the detection unit 76 does not necessarily have to detect the behavior by using machine learning. For example, when the angular velocity feature amount 1 is the axis in the direction of twisting the wrist, and when the angular velocity feature amount 1 is equal to or more than a predetermined threshold value, a meal can be detected. Further, when the angular velocity feature amount 1 is other than the axis in the direction of twisting the wrist, and when the angular velocity feature amount 1 is equal to or less than a predetermined threshold value, a meal can be detected. Further, when the angular velocity feature amount 2 is a predetermined threshold value, for example, several tens of seconds or less, a meal can be detected. Further, when the angular velocity feature amount 3 is equal to or less than a predetermined threshold value, a meal can be detected. Further, when the variance of the acceleration feature amount 1 is equal to or more than a predetermined threshold value, the meal can be detected. Further, when the variance of the acceleration feature amount 2 is equal to or less than a predetermined threshold value, the meal can be detected. Further, when the acceleration feature amount 3 is equal to or more than a predetermined threshold value, the meal can be detected. Further, in the case of the acceleration feature amount 4, if the acceleration feature amount 1 is larger than the acceleration feature amount 2 by a predetermined value or more, the meal can be detected. By combining these determinations of the angular velocity features 1 to 3 and the acceleration features 1 to 4, meals can be detected under the AND condition.

[One aspect of the effect]
As described above, the server device 70 according to the present embodiment detects the meal based on the angular velocity feature amount and the acceleration feature amount in which the characteristics of the motion repeatedly appearing during the meal are quantified. Therefore, according to the server device 70 according to the present embodiment, the accuracy of detecting meals can be further improved.

Although the embodiments relating to the disclosed apparatus have been described so far, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[Standalone]
In the above-mentioned Examples 1 and 2, the case where the above-mentioned healthcare support systems 1 and 2 are constructed as a client-server system is illustrated, but it may not necessarily be constructed as a client-server system. For example, the specific action detection and learning process shown in FIGS. 4 and 9 executed by the functional unit of the server device 30 or 70 is executed standalone by the IoT device 20 connected to the wearable terminal 10 or 50. It doesn't matter. Further, the wearable terminal 10 or 50 and the IoT device 20 may not necessarily be configured as separate devices. That is, the specific action detection and learning processing shown in FIGS. 4 and 9 executed by the functional unit of the server device 30 or 70 by the sensor 11 or the computer equipped with the first sensor and the second sensor is standalone. It does not matter if it is executed in.

[Distributed and integrated]
Further, each component of each of the illustrated devices does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically distributed in any unit according to various loads and usage conditions. Can be integrated and configured. For example, a network of the acquisition unit 31, the division unit 32, the section feature amount calculation unit 33, the first calculation unit 34, the second calculation unit 35, the third calculation unit 36, or the detection unit 37 as an external device of the server device 30. You may connect via. Further, another device has an acquisition unit 31, a division unit 32, a section feature amount calculation unit 33, a first calculation unit 34, a second calculation unit 35, a third calculation unit 36, or a detection unit 37, respectively. The function of the server device 30 may be realized by being connected to a network and cooperating with each other.

Further, each component of each of the illustrated devices does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured. For example, the first acquisition unit 71, the second acquisition unit 72, the division unit 73, the section feature amount calculation unit 74-1, the section feature amount calculation unit 74-2, the phase feature amount calculation unit 75-1, and the phase feature amount. The calculation unit 75-2 or the detection unit 76 may be connected via a network as an external device of the server device 70. Further, the first acquisition unit 71, the second acquisition unit 72, the division unit 73, the section feature amount calculation unit 74-1, the section feature amount calculation unit 74-2, the phase feature amount calculation unit 75-1, and the phase feature amount. Another device may have the calculation unit 75-2 or the detection unit 76, respectively, and may realize the function of the server device 70 by being connected to a network and cooperating with each other.

[Specific behavior detection program]
Further, the various processes described in the above embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a specific behavior detection program having the same function as that of the above embodiment will be described with reference to FIG.

FIG. 16 is a diagram showing a hardware configuration example of a computer that executes the specific behavior detection program according to the first to third embodiments. As shown in FIG. 16, the computer 100 has an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 has a CPU 150, a ROM 160, an HDD 170, and a RAM 180. Each of these 110 to 180 parts is connected via the bus 140.

As shown in FIG. 16, the HDD 170 includes the acquisition unit 31, the division unit 32, the section feature amount calculation unit 33, the first calculation unit 34, the second calculation unit 35, and the third unit shown in the first embodiment. The specific action detection program 170a that exerts the same function as the calculation unit 36 and the detection unit 37 of the above is stored. Further, in the HDD 170, the first acquisition unit 71, the second acquisition unit 72, the division unit 73, the section feature amount calculation unit 74-1, the section feature amount calculation unit 74-2, and the phase feature amount calculation unit 75-1 , The specific action detection program 170a that exerts the same functions as the phase feature amount calculation unit 75-2 and the detection unit 76 may be stored. The specific behavior detection program 170a may be integrated or separated as in the case of each component of the functional unit shown in FIGS. 1 and 11. That is, not all the data shown in the first embodiment may be stored in the HDD 170, and the data used for processing may be stored in the HDD 170.

Under such an environment, the CPU 150 reads the specific action detection program 170a from the HDD 170 and deploys it to the RAM 180. As a result, the specific behavior detection program 170a functions as the specific behavior detection process 180a as shown in FIG. The specific action detection process 180a expands various data read from the HDD 170 into the area allocated to the specific action detection process 180a in the storage area of the RAM 180, and executes various processes using the expanded various data. .. For example, as an example of the process executed by the specific action detection process 180a, the process shown in FIGS. 4 and 9 is included. In the CPU 150, not all the processing units shown in the first embodiment need to operate, and it is sufficient that the processing units corresponding to the processes to be executed are virtually realized.

The specific action detection program 170a may not necessarily be stored in the HDD 170 or the ROM 160 from the beginning. For example, the specific action detection program 170a is stored in a "portable physical medium" such as a flexible disk inserted into the computer 100, a so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 100 may acquire and execute the specific action detection program 170a from these portable physical media. Further, another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, or the like stores the specific action detection program 170a, and the computer 100 stores the specific action detection program 170a from these. You may get it and execute it.

The following additional notes will be further disclosed with respect to the embodiments including the above embodiments.

(Appendix 1) Refer to the sensor data of the predetermined section acquired from the sensor that measures the activity of the target person.
Based on the sensor data and the first condition, a plurality of first sections having the first feature and a plurality of second sections having the second feature are extracted from the sensor data.
For the plurality of first sections, the first feature amount for at least one parameter is calculated.
For the plurality of second sections, the second feature amount for the at least one parameter is calculated.
Whether the activity of the subject in the predetermined section is a specific behavior indicated by the determination model based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. Detect,
A specific behavior detection program that causes a computer to perform processing.

(Appendix 2) The specific behavior detection program according to Appendix 1, wherein the detection process uses the determination model generated by machine learning to detect the specific behavior.

(Appendix 3) The process referred to is an arm of a predetermined section acquired from the sensor, which is at least one of an acceleration sensor, an angular velocity sensor, or a motion sensor mounted on a wearable terminal mounted on the arm of the subject. Refer to the sensor data regarding the movement of
The extraction process is based on the sensor data and the first condition regarding the movement of the arm, from the sensor data, a plurality of first sections having a first feature and a plurality of second sections having a second feature. Extract the section of
In the process of calculating the first feature amount, the first feature amount relating to the parameter of at least one arm movement is calculated for the plurality of first sections.
The process of calculating the second feature amount is the specific action described in Appendix 1 or 2 for calculating the second feature amount regarding the parameter of the movement of at least one arm for the plurality of first sections. Detection program.

(Appendix 4) The specific behavior detection program according to Appendix 1, 2 or 3 for detecting a meal as the specific behavior.

(Appendix 5) Refer to the sensor data of the predetermined section acquired from the sensor that measures the activity of the target person.
Based on the sensor data and the first condition, a plurality of first sections having the first feature and a plurality of second sections having the second feature are extracted from the sensor data.
For the plurality of first sections, the first feature amount for at least one parameter is calculated.
For the plurality of second sections, the second feature amount for the at least one parameter is calculated.
Whether the activity of the subject in the predetermined section is a specific behavior indicated by the determination model based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. Detect,
A specific behavior detection method in which a computer executes processing.

(Appendix 6) The specific behavior detection method according to Appendix 5, wherein the detection process uses the determination model generated by machine learning to detect the specific behavior.

(Appendix 7) The process referred to is an arm of a predetermined section acquired from the sensor, which is at least one of an acceleration sensor, an angular velocity sensor, or a motion sensor mounted on a wearable terminal mounted on the arm of the subject. Refer to the sensor data regarding the movement of
The extraction process is based on the sensor data and the first condition regarding the movement of the arm, from the sensor data, a plurality of first sections having a first feature and a plurality of second sections having a second feature. Extract the section of
In the process of calculating the first feature amount, the first feature amount relating to the parameter of at least one arm movement is calculated for the plurality of first sections.
The process of calculating the second feature amount is the specific action according to Appendix 5 or 6 for calculating the second feature amount regarding the parameter of the movement of at least one arm for the plurality of first sections. Detection method.

(Appendix 8) The specific behavior detection method according to Appendix 5, 6 or 7, wherein the detection process detects a meal as the specific behavior.

(Appendix 9) Sensors that measure the activity of the subject and
A reference unit that refers to the sensor data of a predetermined section acquired from the sensor, and
An extraction unit that extracts a plurality of first sections having a first feature and a plurality of second sections having a second feature from the sensor data based on the sensor data and the first condition.
A first calculation unit for calculating a first feature amount for at least one parameter for the plurality of first sections, and a first calculation unit.
A second calculation unit that calculates a second feature amount for at least one parameter for the plurality of second sections,
Whether the activity of the subject in the predetermined section is a specific behavior indicated by the determination model based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. And the detector that detects
Specific behavior detection system with.

(Appendix 10) The specific behavior detection system according to Appendix 9, wherein the detection unit detects the specific behavior by using the determination model generated by machine learning.

(Appendix 11) The reference unit is an arm of a predetermined section acquired from the sensor, which is at least one of an acceleration sensor, an angular velocity sensor, or a motion sensor mounted on a wearable terminal mounted on the arm of the subject. Refer to the sensor data regarding movement,
Based on the sensor data and the first condition regarding the movement of the arm, the extraction unit has a plurality of first sections having a first feature and a plurality of second sections having a second feature from the sensor data. Extract the section and
The first calculation unit calculates a first feature amount relating to at least one arm movement parameter for the plurality of first sections.
The specific behavior detection system according to Appendix 9 or 10, wherein the second calculation unit calculates a second feature amount relating to the parameter of at least one arm movement for the plurality of first sections.

(Appendix 12) The specific behavior detection system according to Appendix 9, 10 or 11, wherein the detection unit detects a meal as the specific behavior.

1 Healthcare support system 10 Wearable terminal 11 Sensor 20 IoT device 30 Server device 31 Acquisition unit 32 Division unit 33 Section feature amount calculation unit 34 First calculation unit 35 Second calculation unit 36 Third calculation unit 37 Detection unit

Claims (6)

Refer to the sensor data of the predetermined section acquired from the sensor that measures the activity of the subject, and refer to it.
Based on the sensor data and the first condition, a plurality of first sections having the first feature and a plurality of second sections having the second feature are extracted from the sensor data.
Statistics of the distribution of at least one parameter pair to the plurality of first segment is calculated as the first feature amount,
Statistics of the distribution of said at least one parameter pair to the plurality of second section is calculated as the second feature quantity,
Whether the activity of the subject in the predetermined section is a specific behavior indicated by the determination model based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. Detect,
A specific behavior detection program that causes a computer to perform processing. The specific behavior detection program according to claim 1, wherein the detection process uses the determination model generated by machine learning to detect the specific behavior. The referenced process relates to the movement of the arm in a predetermined section acquired from the sensor, which is at least one of an acceleration sensor, an angular velocity sensor, or a motion sensor mounted on a wearable terminal mounted on the arm of the subject. Refer to the sensor data and
The extraction process is based on the sensor data and the first condition regarding the movement of the arm, from the sensor data, a plurality of first sections having a first feature and a plurality of second sections having a second feature. Extract the section of
It said first processing of calculating a feature amount of calculates the statistics of the distribution of a single arm motion parameters even without least against said plurality of first segment as the first feature amount,
Said second processing for calculating a feature quantity of the claim 1 to calculate the statistical value of the distribution parameter of the previous SL least one arm movements that pair to said plurality of second segment as a second feature quantity or The specific behavior detection program described in 2. The specific behavior detection program according to claim 1, 2 or 3, wherein the detection process detects a meal as the specific behavior. Refer to the sensor data of the predetermined section acquired from the sensor that measures the activity of the subject, and refer to it.
Based on the sensor data and the first condition, a plurality of first sections having the first feature and a plurality of second sections having the second feature are extracted from the sensor data.
Statistics of the distribution of at least one parameter pair to the plurality of first segment is calculated as the first feature amount,
Statistics of the distribution of said at least one parameter pair to the plurality of second section is calculated as the second feature quantity,
Whether the activity of the subject in the predetermined section is a specific behavior indicated by the determination model based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. Detect,
A specific behavior detection method in which a computer executes processing. A sensor that measures the activity of the subject and
A reference unit that refers to the sensor data of a predetermined section acquired from the sensor, and
An extraction unit that extracts a plurality of first sections having a first feature and a plurality of second sections having a second feature from the sensor data based on the sensor data and the first condition.
A first calculation unit for calculating a statistical value of the distribution of at least one parameter that pair to said plurality of first segment as the first feature amount,
A second calculation unit for calculating a statistical value of the distribution of said at least one parameter pair to the plurality of second segment as a second feature quantity,
Whether the activity of the subject in the predetermined section is a specific behavior indicated by the determination model based on the determination model related to the at least one parameter, the first feature amount, and the second feature amount. And the detector that detects
Specific behavior detection system with.

Images