JP7682838B2 - Aspiration risk determination device, method, and program - Google Patents

Description

The present invention relates to a device, method, and program for determining aspiration risk.

For example, Patent Documents 1 to 3 disclose methods for automating monitoring to ensure that a subject is in a safe state.

Patent Document 1 relates to posture determination on bedding such as a bed, and uses a load sensor to determine the sleeping posture of a user on the bedding. Patent Document 2 estimates the posture of the human body using an image sensor consisting of an infrared sensor, and the postures that can be determined are lying on the bed, sitting on the bed, sitting on the edge of the bed, standing, falling, and staying in or being absent from the room. Patent Document 3 relates to a posture detection method, and determines whether the user is in a specified posture based on the size, height, position, and direction of the head image detected by acquiring an image taken from above the user's head, and the positional relationship between the head and the torso.

In Patent Document 1, pressure sensors are arranged in a matrix on the surface of the bedding that supports the human body, while in Patent Document 2, infrared sensors are used, and both require the installation of special equipment, making it difficult to simply automate monitoring. In this regard, in Patent Document 3, posture is determined from images captured by cameras that are commonly used and are standard equipment on modern mobile devices such as smartphones, making it possible to automate monitoring to a certain extent in a simplified manner.

International Publication No. 2015/186182 JP 2015-231517 A International Publication No. 2016/143641

Yasuko Arakaki, Hiromichi Hara, The concept and method of support in seating support devices, Journal of Kyushu Physical Therapists and Occupational Therapists Joint Society, 2005, Vol. 27, Kyushu Physical Therapists and Occupational Therapists Joint Society, p. 119 Takashi Ideo, Shizuka Kusunoki, Hiroto Kamii, A study on the improvement of eating movement in the elderly by focusing on the trunk and lower limbs, Journal of Kyushu Physical Therapists and Occupational Therapists Joint Conference, 2011, Vol. 2011, 33rd Kyushu Physical Therapists and Occupational Therapists Joint Conference, Session ID 227, p. 227 Tomohito Mizuno, Kayo Yamamoto, Yamato Ashiga, Saori Toyama, Aya Kusuyama, Yoko Takagishi, Akifumi Yamada, Takuji Hirao, Rie Hirao, The effect of shoulder girdle position change on swallowing, Physical Therapy Supplement, 2008, Vol. 35 Suppl. No. 2 (Abstracts of the 43rd Annual Meeting of the Japanese Physical Therapy Society) Hiromichi Takeda, Yoshihiro Yamashina, Kazuyuki Tabira, The effect of simulated kyphosis posture on respiratory function and voluntary cough force, Journal of Physical Therapy, 2020, Vol. 47, No. 1, p. 20-26 Papadopoulou, S., Exarchakos, G., Beris, A., & Ploumis, A. (2013). Dysphagia associated with cervical spine and postural disorders. Dysphagia, 28(4), 469-480. Kota Nakaya, Showa University Dental Hospital, Department of Oral Function Rehabilitation, "Home > Column No. 26 > Why is it easier to choke when swallowing while looking up?!", [Retrieved August 12, 2022], Internet <URL: http://www.okuchidetaberu.com/colum/no26.html>

However, conventional technology has not been able to simply address the issue of "not knowing whether one's posture while eating makes one prone to aspiration" as one of the items to be monitored.

In other words, Patent Document 1 determines the sleeping posture, but does not deal with the sitting posture during meals. Similarly, Patent Document 2 does not deal with the sitting posture during meals. Furthermore, Patent Document 3 also determines the posture of the subject to be monitored for falls, trips, etc. in nursing care facilities, but does not deal with the sitting posture during meals. Furthermore, although Patent Document 3 uses a general-purpose camera, it is necessary to install the camera on the ceiling, etc., as it takes pictures from above, which requires installation costs.

Aspiration and eating actions and postures that may cause aspiration are discussed in, for example, non-patent documents 1 to 5.

Elderly people are prone to kyphosis (an excessively forward-curving posture of the thoracic and lumbar spine, hunched back) due to spinal deformation caused by osteoporosis and weakened trunk muscles, which adversely affects their daily lives. When a person hunchbacks while eating, the head tends to lean backward, which causes the neck to bend backward excessively, making it more likely that food or drink will enter the trachea on the anterior wall of the pharynx rather than the esophagus on the posterior wall of the pharynx, leading to aspiration. Anatomically, a tilted neck makes it easier for food or drink to enter the bronchi directly when swallowing, whereas a forward-curved neck when swallowing is considered safe (Non-Patent Document 6).

When elderly people who live alone eat alone, they may not realize that they are in a hunched posture that makes them more likely to aspirate, which can lead to aspiration pneumonia. It is said that many elderly people who live alone, who have been increasing in number in recent years, eat alone. Also, if there is a mealtime attendant or nurse present, they will help with positioning to correct the eating posture, but the shortage of caregivers is expected to accelerate in the nursing care field, and there is a possibility that they will not be able to give sufficient attention to each elderly person.

If aspiration does occur, one thing to keep in mind is aspiration pneumonia. Pneumonia is the leading cause of death in people aged 80 and over in Japan, and 30% of deaths are caused by aspiration pneumonia in people requiring care. Aspiration pneumonia is a dangerous disease that can be life-threatening for elderly people, so preventing aspiration is important.

As described above, preventing aspiration is important, but conventional technology was not able to simply and effectively detect situations in which aspiration may occur.

In view of the problems with the conventional technology described above, the present invention aims to provide an aspiration risk assessment device, method, and program that can simply and effectively detect aspiration risk.

To achieve the above object, the present invention is an aspiration risk assessment device that executes a first process of estimating the head orientation of a user from front images, which are frame images at each time of a frontal video of a user eating captured from the front side, a second process of estimating the user's neck tilt angle as at least linked to the head orientation, and a third process of determining that the user is in a state where aspiration may occur, at least based on a determination that the neck tilt angle has increased. The present invention is also characterized by a method and program corresponding to the device.

According to the present invention, a frontal image that can be captured with a simple camera arrangement is analyzed, and it is determined that the user is in a state where aspiration may occur based at least on a determination that the angle of cervical retroflexion has increased, so that the risk of aspiration can be determined simply and effectively.

FIG. 2 is a functional block diagram of an aspiration risk assessment device according to one embodiment. 13 is a flowchart showing the operation of the aspiration risk assessment device according to one embodiment. 1 is a diagram showing a schematic arrangement of an imaging section of the aspiration risk assessment device according to the present embodiment when imaging a user. FIG. 10 is a diagram showing a schematic diagram of face feature points and head angles obtained by processing of a front image analysis unit. FIG. FIG. 13 is a diagram illustrating the process of the side image analysis unit. 1A to 1C are diagrams showing schematic examples of users in various states that are to be recorded or analyzed for aspiration assessment by the aspiration risk assessment device. 13 is a schematic example of an aspiration risk management table registered for each user. 3 is a flowchart showing a detailed example of step S3 in FIG. 2 . FIG. 1 is a diagram illustrating a hardware configuration of a typical computer.

Figure 1 is a functional block diagram of an aspiration risk assessment device 10 according to one embodiment. As shown in the figure, the aspiration risk assessment device 10 includes an imaging unit 1, a facial feature point extraction unit 2, a neck flexion angle estimation unit 3, an aspiration risk assessment unit 4, an aspiration alert output unit 5, a face-related data extraction unit 31 including a frontal image analysis unit 311 and a lateral image analysis unit 312, a face-related data storage unit 32, a setting value calculation unit 41, and an aspiration risk management table storage unit 42. The aspiration risk assessment device 10 can be realized, for example, as a mobile terminal such as a smartphone equipped with a camera. The imaging unit 1 can be configured as hardware, such as a camera that takes images.

Figure 2 is a flowchart of the operation of the aspiration risk assessment device 10 according to one embodiment. Below, we will explain the details of each functional unit of the aspiration risk assessment device 10 in Figure 1 while explaining each step S1 to S3 in Figure 2.

In step S1, the following processes 1 and 2 are performed as a first pre-registration process related to aspiration of the user who uses the aspiration risk assessment device 10.
(Process 1) The face-related data extraction unit 31 accepts input of the front and side images of the user's face by capturing the front and side images with the photographing unit 1, or by acquiring these images that have been captured separately and prepared.
(Process 2) The face-related data extraction unit 31 analyzes the front image and side image to extract facial feature points and neck angle as the face-related information of the user, and stores the extracted facial feature points and neck angle in the face-related data storage unit 32 as the pre-registered information of the user.

The face-related data storage unit 32 can be realized as hardware in the form of a storage medium, and stores the pre-registered information, and provides the pre-registered information as a reference for processing by the face feature point extraction unit 2 and the neck tilt angle estimation unit 3 when analyzing the video of the user eating, which will be described later.

Figure 3 is a schematic diagram showing the arrangement when a user is photographed by the photographing unit 1 (cameras CAM1 and CAM2 as hardware) of the aspiration risk assessment device 10 in this embodiment, as a schematic example of how to photograph in process 1 in step S1. In Figure 3, the state in which user U is sitting in a chair CH with a table TB in front of him in a sitting position similar to that during eating, photographed by the camera of the photographing unit 1 in step S1 is shown in a front view FV (front view FV is taken from slightly above rather than directly in front of user U horizontally to clearly show the spatial arrangement), a side view SV (side view SV from the right hand side where user U is holding a cup in the figure), and a planar view PV. In each of the views FV, SV, and PV, the three-dimensional space coordinates (x, y, z) are such that the xy plane is a horizontal plane parallel to the floor or table TB, y is the direction toward the front of the user U's face (the direction of depth from the front of the face), the x axis is the direction toward the right hand of the user U holding the cup in the figure, and the z axis is the vertical direction (height direction). (Note that in the following, these three-dimensional space coordinates (x, y, z) will be described as world coordinates.)

As shown in FIG. 3, in step S1, cameras CAM1 and CAM2 are positioned so that the portion BP from above the chest to the face of user U (the portion BP showing user U's bust) is captured in the front and side images. That is, camera CAM1 is positioned in front of user U's face and at the same height as the face (camera CAM1 is positioned, for example, on table TB to be positioned at that height), and a front image is obtained by capturing an image from directly in front of the face in the direction D1 (y-axis direction), and camera CAM2 is positioned on the side of user U's face (either the right or left side is acceptable, but in this example it is the left side) and at the same height as the face (camera CAM2 is positioned on a pedestal PD separate from table TB to be positioned at that height), and a side image is obtained by capturing an image from directly to the side of the face in the direction D2 (x-axis direction).

In addition, in process 1 of step S1, a single camera may be positioned in two ways, as cameras CAM1 and CAM2 in FIG. 3, and two images may be taken to obtain a front image and a side image. When taking the two images, it is desirable that the user U to be photographed remains still so that the front image and the side image capture the same posture of the user U as much as possible. Alternatively, in process 1 of step S1, two cameras may be positioned as cameras CAM1 and CAM2 in FIG. 3, and one image may be taken with each camera to obtain a front image and a side image. In this case, it is also desirable that the front image and the side image capture the same posture of the user U as much as possible, for example by taking images simultaneously with the two cameras.

In process 2 of step S1, in the face-related data extraction unit 31, the front image analysis unit 311 analyzes the front image to extract face feature points configured as a three-dimensional coordinate point group, and the side image analysis unit 312 analyzes the side image to estimate the neck angle, and these results are stored in a linked form in the face-related data storage unit 32. Details of the processing by the respective units 311, 312 of the face-related data extraction unit 31 are as follows.

The frontal image analysis unit 311 extracts facial feature points (a facial feature point group FP as a plurality of facial feature points whose coordinates in a three-dimensional space are determined) from the frontal image by using existing methods such as those described in the following Non-Patent Documents 7 to 9, and sets a head angle θ _H as the orientation of the facial feature points group FP in the three-dimensional space.
[Non-Patent Document 7] ITAC Solutions Co., Ltd., "Estimating face direction using head-pose-estimation", [Searched on August 12, 2022], Internet <URL: https://www.itd-blog.jp/entry/peep-prevention-1>
[Non-Patent Document 8] Arnaldo Gualberto "Real-Time Face Pose Estimation with Deep Learning", [Retrieved August 12, 2022], Internet <URL: https://medium.com/analytics-vidhya/face-pose-estimation-with-deep-learning-eebd0e62dbaf>
[Non-Patent Document 9] Kazemi, V., & Sullivan, J. (2014). One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1867-1874).

In addition, in the existing methods such as Non-Patent Documents 7 to 9, facial feature points are extracted using machine learning or deep learning networks, but since the accuracy is improved by using a large number of training images, for example, training images can be prepared from the globally standardized facial image dataset "LFW (Labeled Faces in the Wild)" and the model parameters can be trained in advance. The frontal image analysis unit 311 can extract facial feature points from frontal images by using this trained model.

4 is a schematic diagram showing the facial feature points and head angles obtained by the processing of the front image analysis unit 311. As shown in the figure, the camera CAM photographs the face including the head H of the user U from the front to obtain a front image, and from this front image, as shown by white circles in the figure, a predetermined facial feature point group FP on the face, for example, a plurality of points on the eyebrows, eye outline, nose bridge and bottom end, mouth outline, and chin, can be extracted as three-dimensional coordinate points in the three-dimensional camera coordinates ( _xc , _yc , _zc ) of the camera CAM.

From the group of facial feature points FP thus extracted as three-dimensional coordinate points, a head angle θH can be determined which represents the orientation of the camera CAM of the head H of the user U in the three-dimensional camera coordinates ( _xc , _yc , _zc ) corresponding thereto. Here, for the camera CAM in Fig. 4, by positioning _it when capturing a front image directly in front of the face of the user U as in camera CAM1 in Fig. 3, the three-dimensional camera coordinates ( _xc , _yc , _zc ) in Fig. 4 can be made to match the three-dimensional world coordinates (x, y, z) in Fig. 3 (the orientations of the coordinate axes can be considered to match, thereby making it possible to set the two coordinate systems to match each other), and the head angle _θH can be determined as the orientation in the three-dimensional world coordinates (x, y, z) in Fig. 3.

The head angle _θH can be expressed, for example, as the following three angles (X, Y, Z). (Note that in FIG. 4, the rotation axes for determining these angles X, Y, and Z are depicted as X, Y, and Z.)
X: Nodding angle (pitch)
Y: The angle when you turn your head from side to side (roll)
Z: Head tilt angle (yaw)

In this embodiment, the head angle _θH in the front image analyzed by the front image analysis unit 311 in step S1 is set to a state in which the head H (direction toward the top of the head) of the user U faces vertically and the face faces forward. In other words, when this head angle _θH is expressed by the above angles (X, Y, Z), _θH =(X, Y, Z)=(0, 0, 0).

By setting in this manner, it is possible to record the facial feature points group _FP =FP(0,0,0) corresponding to the head angle θH=(X,Y,Z)=(0,0,0) with the head oriented vertically and the face oriented forward in the three-dimensional world coordinates (x,y,z) in the face-related data storage unit 32. This facial feature points group FP(0,0,0) has the meaning of an origin as a reference where the head of the user U is upright in the vertical direction and there is no inclination of the face, and therefore will be appropriately referred to as the reference facial feature points group _FPref .

The side image analysis unit 312 measures the neck angle θ _N by analyzing the side image. Specifically, the neck angle θ _N can be measured by detecting a neck joint point a and a (left or right) shoulder joint point b from the side image, and assuming that the direction of the straight line connecting these points a and b in the side image is the neck angle θ _N.

Figure 5 shows examples EX51 and EX52 of the processing by the side image analysis unit 312. Example EX51 is a schematic example of joint extraction processing for an image to detect neck joint point a and shoulder joint point b as three-dimensional spatial coordinate points from a side image.

The joint extraction process can use any existing method for extracting skeletal joints as key points from a person image, and the key points may be extracted as three-dimensional world coordinates or two-dimensional image coordinates. As in Example EX51, the neck joint point a and shoulder joint point b may be obtained from various joints extracted by key point extraction. Note that in Example EX51, various joints are shown as seen from the front camera CAM1 side in FIG. 3, but as already explained, the side image analysis unit 312 obtains the neck joint point a and shoulder joint point b from the side image taken by the side camera CAM2 in FIG. 3 as shown in Example EX52, and the direction of the straight line Lab connecting a and b is set as the neck angle θ _N. (Note that the neck joint point a and shoulder joint point b are defined in advance in the key point extraction method used, and do not necessarily have to coincide with medical joint points.)

As shown in example EX52, the straight line Lab is defined on a yz plane parallel to the side of the user U as a projection (ignoring the coordinate value in the x-axis direction) of the world coordinate (x, y, z) in the x-axis direction (towards the user's side), and the neck angle _θN can be defined as the orientation of the straight line Lab on this yz plane. That is, similarly to FIG. 4 which explains the front image, the relationship between the camera coordinates ( _xc , _yc , _zc ) and the world coordinates (x, y, z) for the side image is known, as with camera CAM2 in FIG. 3 (the relationship is (x, y, z) = ( _yc , _-xc , _zc ) in the side image taken by camera CAM2 in FIG. 3). Therefore, the detected positions of joint points a and b in camera coordinates ( _xc , _yc , _zc ) can be obtained as positions ( _xa , _ya , _za ) and ( _xb , _yb , _zb ) in world coordinates, respectively, and the orientation of the straight line Lab which connects points a ( _ya , _za ) and b ( _yb , _zb ) on the yz plane can be defined as the neck angle _θN .

The above is a case where three-dimensional keypoint extraction is applied, but the direction of the straight line Lab can be determined as the neck angle θ _N in the same manner when keypoint extraction is applied as two-dimensional image coordinates in the image coordinates (u, v) to obtain the joint points a and b. That is, the u-axis (horizontal direction) of the image coordinates is set to coincide with the direction of the y-axis of the world coordinates, the v-axis (vertical direction) is set to coincide with the direction of the z-axis of the world coordinates, and the direction of the straight line Lab connecting the points a (u _a , v _a ) and b (u _b , v _b ) on the uv plane can be determined as the neck angle θ _N as the direction in the uv plane (and the direction in the yz plane). Note that, when keypoints are extracted in two-dimensional image coordinates and the angle θ _N is measured from the straight line Lab in this way, a side image needs to be used, whereas when keypoints are extracted in three-dimensional coordinates from the image and the angle θ _N is measured from the straight line Lab, a side image does not necessarily need to be used. However, even when extracting three-dimensional coordinates, it is desirable to use a side image from the standpoint of ensuring measurement accuracy of the orientation of the straight line Lab by accurately photographing the user from the side by adjusting the camera position during shooting.

As for the neck angle θ _N , as shown in example EX52, the point where the straight line Lab coincides with the vertical direction Lz (z-axis direction) is defined as 0° (origin), and the positive value of the neck angle θ _N >0 can be determined in the direction bending forward from this vertical direction Lz.

The neck angle θ _N determined in this manner is stored in the face-related data storage unit 32 as a direction in the xz plane, which is a plane of the side surface of the user U, in order to be used for estimating the angle of backward bending of the neck, which is a cause of aspiration.

Figure 6 is a diagram showing schematic examples of a user in various states that are to be recorded and analyzed for aspiration assessment by the aspiration risk assessment device 10. In Figure 6, the user states are depicted diagrammatically in the yz plane, which is the plane of the user's side, and these correspond to examples of side images captured by the camera CAM2 arranged sideways in Figure 3.

In FIG. 6, examples EX1A and EX1B respectively show the first user U1's upright head state (the state in the front and side images acquired in step S1) and the state in which the head turns backward from this upright state (the head tilts backward at a pitch angle X), causing the neck to bend backward. Similarly, examples EX2A and EX2B respectively show the second user U2's upright head state (the state in the front and side images acquired in step S1) and the state in which the head turns backward from this upright state (the head tilts backward at a pitch angle X), causing the neck to bend backward.

In each example of FIG. 6, point c is the position of the top of the user's head, and point d is the position of the user's external ear canal. When the straight line Lcd connecting these points c and d faces the vertical direction Lz, the user's head is in a vertically erect state. However, in this embodiment, the top of the user's head c and external ear canal d are not detected from the image, but the user's head is in a vertically erect state when the user's head takes the posture of the reference face feature point group FP _ref obtained by the front image analysis unit 311 in step S1. In each example of FIG. 6, the straight line Lcd is drawn for the convenience of easily expressing the head angle θ _H , which is the inclination of the user's head, but it should be noted that the user's head angle θ _H is obtained as the posture of the face feature point group FP, not by detecting points c and d. (Note that the case where the head angle θ _H changes from 0° in the vertical state as in examples EX1B and EX2B will be described in the video analysis process during eating, which will be described later.)

As shown in Fig. 6, the first user U1 does not have a tendency to kyphosis, and the second user U2 has a tendency to kyphosis. If the value of the neck angle _θN of the first user U1 is _θN [U1] and the value of the neck angle _θN of the second user is _θN [U2], then " _θN [U1]< _θN [U2]" is satisfied, and it is assumed that the greater the tendency to kyphosis, the greater the calculated value of the neck angle _θN . The risk of aspiration is determined in the video analysis process during eating described later (note that examples EX1B and EX2B in Fig. 6 will be referred to when explaining this process), and the greater the value of the neck angle _θN, the greater the estimated neck flexion angle _θNB , so that the tendency to be determined to have a risk of aspiration is increased. In this way, the information on the neck angle _θN registered in step S1 is utilized.

Step S1 in Figure 2 has been explained above. Next, in step S2, as the second pre-registration process, input of basic information regarding the user's health condition is accepted, and the setting value calculation unit 41 calculates setting values from the basic information, and the basic information and setting values are stored in the aspiration risk management table storage unit 42 as an aspiration risk management table.

FIG. 7 is a schematic example of the aspiration risk management table TBL registered for each user. The basic information to be input is, for example, seven explanatory variables X=(x ₁ , x ₂ , ..., x ₇ ) related to the health condition of the user, where x ₁ is whether the user is 65 years old or older, x ₂ is whether or not the user has an underlying disease such as a respiratory disease, x ₃ is a history of aspiration pneumonia, x ₄ is whether or not the user has a hunched posture, x ₅ is whether or not the user has dementia, x ₆ is whether or not the user has dentures, and x ₇ is whether or not the user has a tendency to eat quickly. Here, for each of the seven variables x _i (i=1, 2, ..., 7), x _i =0 means "absent" and x _i =1 means "present", and the variables are evaluation items that are "present" when the user has a tendency to have poor health. Other similar items related to the health condition may be added to the explanatory variables X and input may be accepted as basic information, or only some of the seven items illustrated in FIG. 7 may be used instead of all of them.

The set value calculation unit 41 calculates an objective variable Y=f(X) as a function of the input explanatory variable X, and sets the value of the objective variable Y as the set value. In this embodiment, three objective variables Y=( _y1 , _y2 , _y3 ) are calculated, where _y1 is the neck retroflexion angle threshold, _y2 is the alert output grace period, and _y3 is the alert duration. Initial values _B1 , _B2 , and _B3 can be set in advance for these, respectively. For the function f that calculates the objective variable Y, for example, a weighted sum of explanatory variables such as the following can be used.

For the set value of the neck flexion angle threshold _y1 for j=1, the weight coefficient _W1i (1≦i≦7) of each explanatory variable _xi is set to a predetermined negative value, so that the value of the neck flexion angle threshold _y1 can be calculated to be smaller as the user's health condition is worse in the explanatory variable _xi input as basic information. The initial value _B1 of the neck flexion angle threshold _y1 is preset to, for example, _B1 =20°.

For the setting value of the alert output grace period _y2 for j=2, the weight coefficient _W2i (1≦i≦7) of each explanatory variable _xi is set to a predetermined negative value, so that the value of the alert output grace period _y2 becomes smaller (the alert output grace period _y2 becomes shorter) as the user's health condition in the explanatory variable _xi input as basic information becomes worse. The initial value _B2 of the alert output grace period _y2 is preset to, for example, _B2 =3 seconds.

For the alert output grace period _y2 , the weighting factor _W2i (1≦i≦7) may be set so that the set value is 0 seconds or more. Alternatively, if the set value is calculated to be less than 0 seconds, additional processing may be performed to overwrite the set value to 0 seconds.

For the set value of the alert duration _y3 for j=3, the weight coefficient _W3i (1≦i≦7) of each explanatory variable _xi is set to a predetermined positive value, so that the value of the alert duration _y3 becomes larger (the alert duration _y3 becomes longer) as the user's health condition becomes worse in the explanatory variable _xi input as basic information. The initial value _B3 of the alert duration _y3 is preset to, for example, _B3 =5 seconds.

The aspiration risk management table storage unit 42 can be realized as hardware in the form of a storage medium, and stores the basic information as pre-registered information and the setting values calculated from the basic information, and provides the pre-registered information as a reference for processing by the cervical retroflexion angle estimation unit 3 and the aspiration risk determination unit 4 when performing video analysis of the user eating, which will be described later.

In the above, the input of basic information regarding the user's health condition is accepted, but this input may be omitted. In that case, the respective setting values are stored in the aspiration risk management table storage unit 42 as corresponding to the initial values _B1 , _B2 , and _B3 , and a message indicating that no basic information was input is stored.

In addition, for the function calculated from the basic information X with the set value Y=f(X), in addition to the weighted sum described above, a machine learning model or deep learning network that has been trained in advance using a large amount of training data may be used.

Step S2 in FIG. 2 has been described above. Next, in step S3, by using the information preregistered in steps S1 and S2, respectively, the aspiration risk assessment device 10 performs processing to assess the aspiration risk of the user in real time using an image captured from the front while eating (the same user whose preregistered information was obtained in steps S1 and S2), and then the flow in FIG. 2 ends.

Figure 8 is a flowchart showing a detailed example of step S3 in Figure 2.

In step S31, the camera is installed and begins to capture video of the user eating, and the process proceeds to step S32. Specifically, the camera, which is the hardware that constitutes the imaging unit 1, is placed in front of the user U eating, similar to the camera CAM1 in FIG. 3, to capture a frontal image.

For the purpose of explanation, let F(t) be the frame image (frontal image) at time t of the frontal video captured in real time by the imaging unit 1 at each time t = 1, 2, 3, .... The imaging unit 1 outputs the frame image F(t) at each time t to the facial feature point extraction unit 2, so that the processing is repeated by the facial feature point extraction unit 2, the cervical retroflexion angle estimation unit 3, the aspiration risk determination unit 4, and the aspiration alert output unit 5 at each time t in real time. In FIG. 8, steps S32 to S37 represent the repeated processing at each time t, and step S38 represents the processing of updating the real-time time t to the next time t+1 (processing of managing the processing timing).

In step S32, the frame image F(t) at the current time t obtained by the image capture unit 1 is read into the facial feature point extraction unit 2, and then the process proceeds to step S33.

In step S33, the facial feature point extraction unit 2 extracts facial feature points from the frame image F(t) and outputs them to the cervical retroflexion angle estimation unit 3, and then the process proceeds to step S34.

In step S34, the neck bending angle estimation unit 3 estimates the neck bending angle θ NB (t) at the current time t from the difference (difference in orientation) between the facial feature points of the frame image _F (t) and the pre-registered facial feature points (reference facial feature point group), outputs the neck bending angle θ _NB (t) to the aspiration risk assessment unit 4, and then proceeds to step S35.

The processing of steps S33 and S34, similar to the processing of the front image analysis unit 311, estimates the value θ _H (t) of the head angle θ _H at the current time t using techniques such as those described in non-patent documents 7 to 9, and then adds this to the fixed value of the neck angle θ _N obtained by the side image analysis unit 312 to estimate the neck flexion angle θ _NB (t) as shown in equation (1).
θ _NB (t)=θ _N +θ _H (t) …(1)

Examples EX1B and EX2B in FIG. 6 are schematic examples of formula (1) when the head angle θ _H (t) of each user is positioned in a backward bending direction. As already explained, the neck angle θ _N as a fixed value is defined as positive on the side bending forward from the vertical direction Lz. Also, the head angle θ _H (t) is defined as positive on the side bending backward from the vertical direction Lz. Formula (1) estimates the neck backward bending angle θ _NB (t) by adding to the base of these definitions, but as shown in examples EX1B and EX2B, the neck backward bending angle θ _NB (t) means the difference in orientation between the line Lab, which is the orientation of the neck, and the line Lcd, which is the orientation of the head (the angle formed by the line Lab and the line Lcd).

In the processing of the front image analysis unit 311, the reference facial feature points group FP _ref is registered as corresponding to the origin (0°) of the head angle θ _H (t) as shown in Examples EX1A and EX2A. In step S33, the facial feature points group FP(t) at the current time t is similarly extracted from the frame image F(t), and the angle change from the reference facial feature points group FP _ref in the facial feature points group FP(t) at the current time t can be estimated as the head angle θ _H (t) at the current time t. In other words, if the direction change at the head angle θ _H (t)=(X(t),Y(t),Z(t)) due to pitch, roll, and yaw is also expressed as "θ _H (t)", the following relationship holds:
FP(t)= _θH (t)・FP _ref

In reality, due to noise and other factors that occur when detecting facial feature points from an image, FP(t) and FP _ref will not completely match after orientation conversion as described above. However, in order to minimize conversion errors, the head angle θ _H (t)=(X(t),Y(t),Z(t)) can be estimated using the methods described in Non-Patent Documents 7 to 9, etc.

For θ _H (t) in equation (1), the value of the pitch X(t) component among pitch, roll, and yaw (X(t), Y(t), Z(t)) can be used to estimate the neck tilt angle θ _NB (t) as a one-dimensional angle in a plane viewed from the side of the user (head). The schematic example in Fig. 6 is based on the premise that roll and yaw are 0° or can be ignored as small values, but even if the roll and yaw values exist at a certain level, the roll and yaw values can be ignored and only the pitch component can be focused on to perform the calculation using equation (1) similar to that in Fig. 6.

In addition, in equation (1), the neck angle θ _N is treated as a constant value. In reality, the neck angle θ _N is also expected to change over time as θ _N = θ _N (t), but in this embodiment, this change over time is assumed to be small, so the neck angle θ _N is treated as a constant value.

In step S35, the aspiration risk assessment unit 4 compares the cervical flexion angle θ _NB (t) obtained in step S34, including the history {θ _NB (k)|k=1, 2, ..., t} up to the current time t, with the threshold value of the aspiration risk management table pre-registered in the aspiration risk management table storage unit 42, to determine whether the user who is eating at the current time t is at risk of aspiration, and then proceeds to step S36.

Specifically, in step S35, the aspiration risk management table is referenced to a set value of the cervical retroflexion angle threshold _y1 (written as θ _{NB threshold} to clarify that it is a threshold) and a set value of the alert output grace period _y2 (when converted into the number of processing steps at each time t in FIG. 8, this set value _y2 is the time length of N steps), and if the following is true, it can be determined that there is a risk of aspiration.
"θ _NB (tk)≧θ _{NB threshold value} " holds for all k=0, 1, 2, . . . , N-1.

In other words, when the estimated neck tilt angle remains above the threshold for the length of the alert output grace period _y2 (N processing steps), it can be determined that there is a risk of aspiration.

As described above, the set value of the alert output grace period _y2 is calculated to be a shorter value as the user's health condition is evaluated to be on the worse side. Therefore, for users who are in poor health and at high risk of aspiration, it is possible to determine that there is a risk of aspiration even if the condition in which the neck flexion angle is determined to be large continues for a short period of time.

Furthermore, when the set value of the alert output grace period _y2 is 0 seconds, by setting N=1 above, it is possible to immediately determine that there is a risk of aspiration at time t when "θ _NB (t)≧θ _{NB threshold value} ".

In step S36, it is determined whether (Condition 1) the current time t corresponds to a state in which an alert is already being output in the aspiration alert output unit 5 described below, or (Condition 2) the current time t corresponds to a state in which an alert is not being output in the aspiration alert output unit 5 and the determination result in step S35 is that there is a risk of aspiration. If neither of these conditions 1 nor 2 is met, the process proceeds from step S36 to step S38. If either of these conditions 1 or 2 is met, the process proceeds to step S37.

In step S37, the aspiration alert output unit 5 outputs an aspiration alert, and then the process proceeds to step S38. The aspiration alert output may start at the current time t, or may have already started at a past time t-M (M>1) and continue to be output at the current time t.

The aspiration alert output unit 5 refers to the alert duration _y3 in the aspiration risk management table, and outputs an aspiration alert continuously for the alert duration _y3 from the time when the aspiration risk determination unit 4 determines that there is an aspiration risk. When the time _y3 has elapsed after the alert output, the aspiration alert output unit 5 may stop outputting the alert.

The alert output may be, for example, by generating a predetermined warning sound such as a buzzer or siren from a speaker to provide an auditory stimulus to the user, or by vibrating a vibration element such as a vibrator to provide an auditory stimulus to the user, or by flashing a display screen or light to provide a visual stimulus to the user.

When using a warning sound in the alert output, for example, a sound may be used that speaks the text "Watch out for choking! Your face is leaning back when swallowing. Please correct your posture" (the sound may be played repeatedly as appropriate so as to continue until the alert duration _y3 ). Alternatively, a display output may be used in the alert output, and the text may be displayed on the display as a warning display so that the user can see the warning text. An alert output that combines these various warning means may also be used.

As described above, the set value of the alert duration _y3 is calculated to be a longer value as the user's health condition is evaluated to be worse. Therefore, for users who are in poor health and at high risk of aspiration, the prevention of aspiration can be made more reliable by continuing the alert output for a longer period of time.

In the flow of FIG. 8, when the aspiration alert output unit 5 continues to output an alert, the processing of steps S32 to S35 may be omitted.

In step S38, time t is updated to real-time time t+1, which is the next processing timing, and the process returns to step S32, where the same process is repeated for the next time t+1. Although not shown in FIG. 8, when the user has finished eating, the process of the aspiration risk assessment device 10 can be terminated by receiving an input from the user indicating that the aspiration risk assessment process is to be terminated.

As described above, the aspiration risk assessment device 10 of this embodiment can easily assess the risk of aspiration by analyzing a frontal image of the user while eating, and can effectively prevent the user from aspiration by outputting an alert if it is determined that there is a risk of aspiration. In addition, the following effects can be achieved.

- According to this embodiment, compared to conventional methods, the user himself can take videos and place it on the dining table, and all that is required is an existing device that the user owns, such as a tablet or smartphone that runs the application of this embodiment. This is expected to have the effect of making the user aware of the risk of aspiration (preventing choking) and correcting posture in real time at low cost, even without a meal assistant.
-Instead of judging from a one-shot photograph, the system judges in real time from video, making it possible to respond to feeding behavior from start to finish and changes over time.
- By accumulating and analyzing actual data on aspiration alert outputs, the settings in the "aspiration risk management table" can be changed to values that suit the characteristics of each user, which is expected to be more effective in improving eating posture.

Below, we will explain various supplementary examples, modifications, and additional examples.

The aspiration risk assessment device 10 of this embodiment can prevent the user from aspiration while eating by outputting an alarm, making it possible to contribute to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs) "Ensure healthy lives and promote well-being for all at all ages."

The embodiment described above is used as a basic example, and various variations of this basic example will be described.

<Modification 1> It is possible to omit acquiring a lateral image and measuring the neck angle θ _N by the lateral image analysis unit 312. In this case, the following formula (1a) can be used instead of formula (1), which is equivalent to treating the neck angle θ _N =0° in the basic example and treating the head orientation as coinciding with the neck retroflexion angle.
θ _NB (t) = θ _H (t) …(1a)

<Variant 2> In the basic example, the neck angle θ _N was measured only once in pre-registration and treated as a fixed value, but by using a value θ _N (t) for the neck angle θ _N that can change at each time t in real time, the following equation (1b) can be used instead of equation (1).
θ _NB (t)=θ _N (t)+θ _H (t) …(1a)

In this case, two cameras are used, one for capturing a front image and one for capturing a side image, and in addition to analyzing the front image captured by camera CAM1 arranged in Fig. 3 in the same manner as in the basic example, the side image captured by camera CAM2 is also analyzed in the same manner as the method used by side image analysis unit 312 at the time of pre-registration, thereby obtaining the neck angle θ _N (t) at each time t. That is, by using the frame image F(t) of the side image at each time t as the user's side image and applying the same processing as that performed by side image analysis unit 312 on the side image at the time of pre-registration, the neck angle θ _N (t) at each time t can be obtained.

In addition, in all of the basic example and the first and second modified examples, the neck tilt angle θ _NB (t) is estimated as being linked to at least the head angle θ _H (t).

<Modification 3> Regarding the arrangement of the camera in the basic example In the basic example, as described in FIG. 3 and the like, both the front image and the side image are taken by cameras arranged almost directly in front and directly to the side of the user's face, but as long as the error is within an acceptable range, the camera may be arranged almost in front and almost to the side. Even if the error is within an acceptable range (assuming that the front and side of the user's face can be taken), the camera attitude can be estimated (if the camera is not fixed and moves, it may be estimated in real time) by any existing method such as attitude measurement using an attitude sensor such as a gyro installed in the camera or a method of detecting a square marker used in augmented reality display from an image to estimate the external parameters (camera attitude) of the camera, and by obtaining the conversion relationship between the camera coordinates ( _xc , _yc , _zc ) and the world coordinates (x, y, z), various angles, etc. can be estimated in the world coordinates (x, y, z) in the same way as when the camera is arranged directly in front and directly to the side.

<Hardware Configuration>
FIG. 9 is a diagram showing an example of a hardware configuration in a general computer. The aspiration risk assessment device 10 can be realized as one or more computer devices 70 having such a configuration. When the aspiration risk assessment device 10 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, a GPU (graphics processing unit) 72 as a dedicated processor that executes part or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device (memory) that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device (storage), a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 81, a speaker 82, a light 83, a vibration element 84, and a bus BS for transmitting and receiving data between them.

Each functional unit of the aspiration risk assessment device 10 can be realized by a CPU 71 and/or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, the display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, the communication interface 75 also operates in conjunction with the GPU 72.

When the aspiration alert output unit 5 outputs an alert visually, it may use a display on the display 76 or the illumination of the light 83. When the alert is output auditorily, it may use the speaker 82. When the alert is output by vibration, it may use a vibration element 84 constituted by a vibrator or the like. The face-related data storage unit 32 and the aspiration risk management table storage unit 42 may be realized in the ROM 74 or the RAM 73.

10...Aspiration risk assessment device, 1...Photographing unit, 2...Facial feature point extraction unit, 3...Neck tilt angle estimation unit, 4...Aspiration risk assessment unit, 5...Aspiration alert output unit, 31...Facial related data extraction unit, 311...Frontal image analysis unit, 312...Side image analysis unit, 32...Facial related data storage unit, 41...Setting value calculation unit, 42...Aspiration risk management table storage unit

Claims (12)

A first process of estimating a head direction of a user from a front image, which is a frame image at each time of a front image of the user who is eating taken from the front side;
A second process of estimating a neck tilt angle of the user as being at least linked to the head orientation;
A third process of determining that the user is in a state where aspiration may occur, based at least on the determination that the neck flexion angle has increased. The aspiration risk assessment device according to claim 1, characterized in that in the first process, a group of three-dimensional facial feature points of the user is extracted from the front image, and the head orientation is estimated as the change in the orientation of the three-dimensional facial feature points relative to the orientation of a reference group of three-dimensional facial feature points extracted from an image of the user captured from the front with the user's head held upright in the vertical direction, which is pre-registered for the user. In the first process, a head orientation is estimated by defining a head orientation of 0° when the head of the user is upright in the vertical direction, and estimating the head orientation by assuming that the head orientation increases to a positive value when the head of the user changes from the upright state to a direction in which the head is tilted backward;
The aspiration risk determination device according to claim 1 , wherein in the second process, the neck tilt angle is estimated as a value equal to the estimated head orientation value. The head direction is an angle in which the backward bending side is positive,
The device for assessing the risk of aspiration as described in claim 1, characterized in that in the second process, the neck tilt angle is estimated as the sum of a neck angle, which is the angle between the neck joint and shoulder joint of the user in a side view of the user and in which the forward tilt side is positive, and the head direction, which is pre-registered as a fixed value for the user. In the first process, a neck angle, which is an angle between the neck joint and the shoulder joint of the user in a side view of the user, is measured from a side image, which is a frame image at each time of a side image taken from the side of the user while eating, so that the forward bending side is positive;
The head direction is an angle in which the backward bending side is positive,
The aspiration risk determination device according to claim 1 , wherein in the second process, a neck tilt angle of the user is estimated as a sum of the neck angle and the head direction. The aspiration risk assessment device according to claim 1, characterized in that in the third process, when the state in which the neck tilt angle exceeds a threshold continues for a certain period of time, or when the neck tilt angle exceeds the threshold, it is determined that the user is in a state in which aspiration may occur. The aspiration risk assessment device according to claim 6, characterized in that the threshold value is set smaller as the user's health condition is assessed to be on the worse side based on the values of one or more evaluation items related to the user's health condition. The aspiration risk assessment device according to claim 6, characterized in that the fixed period is set shorter as the user's health condition is assessed to be on the worse side based on the values of one or more evaluation items related to the user's health condition. The aspiration risk assessment device according to claim 1, further comprising a fourth process for issuing an alarm to the user by a visual or auditory stimulus or by vibration when it is determined in the third process that the user is in a state where aspiration may occur. In the fourth process, the alarm is continuously generated for a certain period of time,
The aspiration risk assessment device according to claim 9, characterized in that the fixed period is set longer as the user's health condition is assessed to be on the worse side based on the values of one or more evaluation items related to the user's health condition. A first step of estimating a head direction of a user from front images, which are frame images at each time of a front image of a user who is eating taken from the front side;
A second step of estimating a neck tilt angle of the user as being at least linked to the head orientation;
and a third step of determining that the user is in a state where aspiration may occur, based at least on the determination that the neck flexion angle has increased. A program for determining aspiration risk that causes a computer to function as the aspiration risk determination device according to any one of claims 1 to 10.

Images