JP7754432B2 - Dysphagia assessment device and assessment method - Google Patents

Description

This disclosure relates to a device and method for determining swallowing disorders (excluding medical procedures).

Dysphagia, a condition in which the ability to swallow is weakened, is known to make it difficult to eat, increasing the risk of choking, where swallowed fluids or food get stuck in the throat, and aspiration, where swallowed fluids or food get into the trachea. If swallowing disorders are left untreated, they can lead to a prolonged state of malnutrition, leading to frailty and a need for care, or even to life-threatening conditions such as choking or aspiration pneumonia. For this reason, it is important to diagnose swallowing disorders early and take appropriate measures.

Devices have been proposed that assess swallowing disorders by detecting the movement of a subject's throat. Patent Document 1 provides a device that assesses the presence or absence of swallowing disorders by positioning a two-axis accelerometer on the subject's neck.

Patent No. 5977255

However, Patent Document 1 imposes a physical burden on the subject, as the accelerometer must be attached to the subject's neck with double-sided tape. It also places a heavy burden on the examiner who administers the test to the subject, as it requires ensuring that the sensor device does not shift out of position. For this reason, there remains room for research and development into a device that can more easily diagnose swallowing disorders, such as one that reduces the burden on both the subject and the examiner.

This disclosure has been made in consideration of this situation and aims to provide a diagnosis device that can more easily diagnose swallowing disorders.

As a result of intensive research, the inventors discovered that there is a correlation between the speech data of a subject's speech and the degree of progression of that subject's dysphagia. They also discovered that there are significant differences in certain acoustic features between groups classified based on the degree of progression of dysphagia. Based on these findings, the inventors succeeded in creating a device for assessing dysphagia based on voice analysis. Specifically, the details of the assessment device disclosed herein are as follows:

[1] A determination device for determining dysphagia by voice analysis, the determination device comprising:
an input means for inputting voice data uttered by the subject;
analysis means for analyzing the voice data input by the input means;
a determination means for determining whether a subject has dysphagia based on the analysis result by the analysis means;
A determination device comprising:

[2] The determination device according to [1], wherein the analysis means performs speech analysis on the speech data using an acoustic feature expressed by the following formula F(a):

(wherein g is a linear or nonlinear model for determining the presence or absence of dysphagia and the degree of progression thereof, _xn is a coefficient specific to the phrase input as the speech data, and f(n) is an acoustic parameter, and the acoustic parameter is one or more selected from the group consisting of formant frequency, Mel frequency cepstrum, frequency spectrum, volume envelope, waveform fluctuation information, zero crossing rate, Hurst exponent, and time from closure/opening to the start of vocal cord vibration.)

[3] The determination device described in [1] or [2], characterized in that the analysis means performs speech analysis on the speech data using acoustic features created based on formant frequencies or Mel frequency cepstrum.

[4] The determination device described in any one of [1] to [3] is subjected to machine learning processing so as to output the degree of progression of swallowing disorders when the analysis results are input.

[5] The determination device described in any one of [1] to [4], wherein the subject's voice data is input at least twice using the same phrase, and the degree of progression of the swallowing disorder is determined based on the difference or average value of the analysis results analyzed using the same phrase.

[6] A determination device described in any one of [1] to [5], wherein the input means inputs voice data in which the subject speaks phrases related to the degree of progression of the swallowing disorder.

[7] The input means is

The determination device according to any one of [1] to [6], which inputs speech data containing at least one of the sounds.

[8] A method for assessing dysphagia by speech analysis, the method comprising:
an input step of inputting voice data uttered by the subject;
an analyzing step of analyzing the voice data input in the input step;
a determination step of determining whether the subject has dysphagia based on the analysis result obtained by the analysis step;
A method comprising:

[9] A program for causing a computer to function as each means of the determination device described in any one of [1] to [7].

The present disclosure provides a determination device that determines swallowing disorders through voice analysis. It also provides a determination method that determines swallowing disorders through voice analysis. Furthermore, the determination device can not only determine whether a subject has a swallowing disorder, but also determine the degree of progression of the swallowing disorder.

FIG. 1 is a diagram illustrating the configuration of a determination device. FIG. 2 is a flowchart showing an example of a process executed by the determination device. FIG. 3 is a diagram showing an example of audio data input to the input/output unit 12. As shown in FIG. FIG. 4 is an ROC curve showing the results of one embodiment of the present invention. FIG. 5 is an ROC curve showing the results of one embodiment of the present invention. FIG. 6 is a box plot showing the results of one embodiment of the present invention. FIG. 7 is a diagram showing variations in formant analysis values when the same speech content is uttered multiple times. FIG. 8 is a diagram showing the relationship between the value of formant analysis and the degree of progression of dysphagia. FIG. 9A is a diagram showing the relationship between the value of 13th-order Mel-frequency cepstrum analysis and the degree of progression of dysphagia. FIG. 9B is a diagram showing the relationship between the value of the 13th-order Mel-frequency cepstrum analysis and the degree of progression of dysphagia. FIG. 10 is a diagram showing the relationship between phoneme classes and activated phonemes. FIG. 11 shows the results of phoneme analysis performed on the "pa" uttered by a healthy person without dysphagia and on the "pa" uttered by a person with moderate or severe dysphagia. FIG. 12 is a box plot verifying the classification performance according to the severity of dysphagia using acoustic features. FIG. 13 is a box plot verifying the classification performance according to the severity of dysphagia using acoustic features. FIG. 14 is a box plot verifying the classification performance according to the severity of dysphagia using acoustic features. FIG. 15 is a box plot verifying the classification performance according to the severity of dysphagia using acoustic features. FIG. 16 is a box-and-whisker plot verifying the classification performance according to the severity of dysphagia using acoustic features. FIG. 17 is a box plot verifying the classification performance according to the severity of dysphagia using acoustic features. FIG. 18 is a diagram showing audio data. FIG. 19 shows the experimental results showing the correlation between the voice intensity analysis results and the evaluation results of swallowing disorders. FIG. 20 shows the box-and-whisker plot results for each acoustic feature. FIG. 21 shows the box-and-whisker plot results for each acoustic feature. FIG. 22 shows the ROC curve results when determining whether swallowing is mild or severe using a machine learning model that uses an acoustic feature set created using methods including phonemic analysis and intensity analysis. FIG. 23 shows the ROC curve results when determining moderate to severe dysphagia using a machine learning model that uses an acoustic feature set created including the phoneme analysis and intensity analysis techniques.

The following describes in detail the form for implementing this disclosure with reference to the drawings. However, the description of the components described below is an example of one embodiment of this disclosure and is not limited to these contents.

First Embodiment
The swallowing disorder assessment device of the present disclosure includes an analysis unit and a assessment unit as its main components. The analysis unit performs acoustic analysis using an acoustic feature (hereinafter, sometimes referred to as "F(a)") capable of analyzing the degree of progression of the swallowing disorder. The assessment unit may also be subjected to machine learning processing that outputs the degree of progression of the swallowing disorder when the analysis results obtained by the analysis unit are input.

The configuration of the determination device (hereinafter sometimes referred to as the "information processing device") will be explained using Figure 1. The information processing device 10 comprises a control unit 11 that controls the overall operation, an input/output unit 12 that performs various inputs and outputs, a memory unit 13 that stores various data and programs, etc., a communication unit 14 that communicates with the outside world, and an internal bus 15 that connects each block so that they can communicate with each other.

The information processing device 10 is, for example, a computer, and may be a device that the subject can carry with them, such as a smartphone, PDA, tablet, or laptop, or it may be a computer that is fixed in place and not carried by the subject. PDA is an abbreviation for Personal Digital Assistant.

The control unit 11 is a device called, for example, a CPU, MCU, or MPU, and executes programs stored in, for example, the memory unit 13. CPU is an abbreviation for Central Processing Unit. MCU is an abbreviation for Micro Controller Unit. MPU is an abbreviation for Micro Processor Unit.

The input/output unit 12 is a device that inputs and outputs information to and from the subject operating the information processing device 10. The input/output unit 12 inputs and outputs information and signals using a display, keyboard, mouse, buttons, touch panel, printer, microphone, speaker, and the like. In this embodiment, the input/output unit 12 at least functions as a microphone, and inputs voice data using this microphone. Also, in this embodiment, the input/output unit 12 at least functions as a display, and displays the swallowing disorder assessment results, which will be described later, on this display.

The storage unit 13 is a device such as ROM, RAM, HDD, or flash memory, and stores programs executed by the control unit 11 and various data. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. HDD is an abbreviation for Hard Disk Drive.

The communication unit 16 communicates with the outside. Communication by the communication unit 16 may be wired communication or wireless communication. Communication by the communication unit 16 may be by any communication method. The control unit 11 can send and receive various data such as audio data via the communication unit 16. The control unit 11 may also send the swallowing disorder assessment results, described below, to an external device via the communication unit 16.

Next, an example of processing executed by the determination device (information processing device) 10 is shown using Figure 2. First, in step S201, the control unit 11 inputs the subject's voice data via the input/output unit 12. Next, in step S202, the calculation unit (or analysis unit) calculates acoustic features from the voice data. Next, in step S203, the estimation unit (or determination unit) estimates (or determines) the presence or absence of swallowing disorders and the degree of progression. Next, the estimation result (or determination result) is output to the input/output unit 12, and the flow ends.

In addition, the input/output unit 12 in step S201 may use a microphone. The subject speaks into the microphone to input voice data. Voice data may be pre-recorded.

<Phrases selected when entering voice>
The phrases selected during speech input are appropriate for speech analysis of the degree of progression of the subject's dysphagia. As the dysphagia progresses, the degree of tongue movement, the anterior-posterior position of the tongue, the degree of jaw opening, the state of tooth occlusion, the number of teeth, the amount of saliva secreted, muscle weakness, etc., affect the degree of sound resonance in the larynx and oral cavity. Phrases appropriate for analyzing the degree of progression of the dysphagia are phrases that make it easy to discover the degree of sound resonance that changes as the dysphagia progresses.

For example, the phrase selected when inputting voice is

A more detailed explanation will be given below in conjunction with the explanation of FIG.

FIG. 3 shows an example of audio data input to the input/output unit 12.

3. The voice input of the present disclosure is not limited to these phrases. The subject can select at least one of Phrase01 (Ph01) to Phrase10 (Ph10) shown in FIG. 3 as the phrase to be voice-input. Of course, a combination of several of Phrase01 (Ph01) to Phrase10 (Ph10) may also be voice-input.

In Figure 3, Ph01 is audio data uttering "pa." Ph02 is audio data uttering "ma." Ph03 is audio data uttering "ta." Ph04 is audio data uttering "ra." Ph05 is audio data uttering "ka." Ph06 is audio data uttering "go." Ph07 is audio data uttering "Panda's treasure." Ph08 is audio data uttering "tamago." Ph09 is audio data uttering "banana banana banana banana banana" as quickly as possible, repeated five or more times. Ph10 is audio data uttering "kimono kimono kimono kimono" as quickly as possible, repeated five or more times. The relationship between phrases Ph01 to Ph10 and swallowing disorders will be further explained.

<<Ph01, Ph02>>
Ph01 and Ph02 are the sounds "pa" and "ma," and require the movement of closing the lips. In terms of swallowing function, they prevent food from spilling from the mouth during chewing and are involved in the transport of food when swallowing by increasing intraoral pressure.

Other examples of pronunciation include:

etc.

<<Ph03, Ph04>>
Ph03 and Ph04 are the pronunciations of "ta" and "ra.""ta" is a movement that uses the tip of the tongue, and in terms of swallowing function, it is related to the function of mastication and the action of sending food and liquids from the mouth to the back of the throat. "ra" requires the tip of the tongue to move relatively smoothly, and is a sound that allows for the smoothness of tongue movement to be seen. Like "ta," it uses the tip of the tongue and is related to the function of mastication and the action of sending food and liquids from the mouth to the back of the throat.

Other examples of pronunciation include:

etc.

<<Ph05, Ph06>>
Ph05 and Ph06 are the pronunciations of "ka" and "go." Both involve movements that use the back of the tongue, and in terms of swallowing function, they perform a sending motion and increase intrapharyngeal pressure, and are related to the transport of food.

Other examples of sounds include:

etc.

<<Ph07, Ph08>>
Ph07 (Panda's Treasure) and Ph08 (Egg) are evaluated based on the combination of sounds made by the lips and tongue.

<<Ph09, Ph10>>
Ph09 (Banana Banana Banana Banana Banana) and Ph10 (Kimono Kimono Kimono Kimono Kimono) are evaluating nasal resonance, lip and tongue combinations, diadochokinesis, and rhythm.

The above phrases are inputted into the voice to obtain speech data, and speech analysis is performed in step S202. During the speech analysis, acoustic features are calculated in the calculation unit (analysis unit). Acoustic features are described in detail below.

<Acoustic features>
Acoustic features are numerical parameters for quantitatively expressing the features of speech data to be analyzed. In the present disclosure, the determination unit determines the presence or absence of dysphagia and the degree of progression in step S203. However, since machine learning processing is performed to output a determination process using the results of speech analysis based on acoustic features as input, the selection of acoustic features is important for improving accuracy.

In this disclosure, the acoustic feature F(a) can be expressed by the following formula:

In the formula, g is a linear or nonlinear model for determining the presence or absence of dysphagia and the degree of its progression, x _n is a coefficient specific to the phrase input as speech data, and f(n) is an acoustic parameter, which is one or more selected from the group consisting of formant frequency, Mel frequency cepstrum, frequency spectrum, speech envelope, waveform fluctuation information, zero-crossing rate, Hurst exponent, and time from closure to the onset of vocal cord vibration. Furthermore, if there are two or more speech data for the same phrase, the average value or difference between them can be included; if there are three or more speech data, the variation (variance or standard deviation) or median can be included. Furthermore, since acoustic features vary widely in value, each may be normalized. Furthermore, when determining the presence or absence or degree of progression of three or more groups of dysphagia, the features may be divided into two or more.

The types of acoustic parameters are as follows:

(1) Audio envelope (attack time, decay time, sustain level, release time)
(2) Waveform fluctuation information (Shimmer, Jitter, Harmonics to Noise Ratio (HNR), Signals to Noise Ratio (SNR))
(3) Zero-crossing rate (4) Hurst exponent (5) Voice Onset Time (VOT)
(6) Statistics of the intra-utterance distribution of a certain coefficient of the Mel-frequency cepstrum (first quartile, median, third quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the third quartile and the median, etc.)
(7) Statistics of the intra-utterance distribution of the rate of change of the frequency spectrum (first quartile, median, third quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the third quartile and the median, etc.)
(8) Statistics of the intra-utterance distribution of the time variation of a certain coefficient of the Mel-frequency cepstrum (1st quartile, median, 3rd quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the 3rd quartile and the median, etc.)
(9) Statistics of the intra-utterance distribution of the time variation of a certain coefficient of the Mel-frequency cepstrum (first quartile, median, third quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the third quartile and the median, etc.)
(10) Squared error of quadratic regression approximation of intra-utterance time change of 90 percent roll-off of frequency spectrum. (11) Arithmetic error of quadratic regression approximation of intra-utterance time change of center of gravity of frequency spectrum. Others include pitch rate, probability of being voiced, frequency power in any range, scale, speaking rate (number of moras in a certain time), pauses/intervals, volume, etc. (12) Statistical values of intra-utterance distribution of any formant frequency (first formant, second formant, third formant, fourth formant, fifth formant, sixth formant, etc.) (first quartile, median, third quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the third quartile and the median, etc.).
(13) Statistical values of the intra-utterance distribution of any formant frequency (first formant, second formant, third formant, fourth formant, fifth formant, sixth formant, etc.) over time (first quartile, median, third quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the third quartile and the median, etc.)
(14) Statistical values of the intra-utterance distribution of time-varying changes in any formant frequency (first formant, second formant, third formant, fourth formant, fifth formant, sixth formant, etc.) (first quartile, median, third quartile, 95th percentile, 98th percentile, arithmetic mean, geometric mean, difference between the third quartile and the median, etc.)

In step S203, a judgment process is performed based on the acoustic features described above that correlate with the degree of progression of dysphagia, or based on the difference in acoustic features between groups when classified by the degree of dysphagia. Examples of judgment using the above acoustic features are described using Figures 4 to 6.

The subjects were elderly people aged 65 or older, and their swallowing function was confirmed by a speech-language-hearing therapist based on the results of swallowing function tests and the degree of aspiration.The subjects were then classified into healthy individuals with no swallowing disorders, those with mild swallowing disorders, and those with moderate or severe swallowing disorders.

Example 1
Figure 4 shows an ROC curve verifying the classification performance of the presence or absence of dysphagia when a specific program was created using seven of the acoustic parameters (1) to (14) described above, with input speech data in which a subject read each of the 10 phrases shown in Figure 3 twice. The horizontal axis represents "1 - specificity," and the vertical axis represents sensitivity. The AUC was 0.941, confirming sufficient classification performance.

Example 2
FIG. 5 shows an ROC curve verifying the classification performance of a specific program created using the average value of acoustic features calculated from input speech data in which a subject read each of the 10 phrases shown in FIG. 3 twice, using one of the acoustic parameters (1) to (14) described above. The horizontal axis represents "1-specificity," and the vertical axis represents sensitivity. The AUC was 0.981, confirming sufficient classification performance. Furthermore, judging from the results of FIGS. 4 and 5, by combining and processing programs, it is possible to first determine the presence or absence of dysphagia using the program shown in FIG. 4, and then further determine the degree of progression of dysphagia (mild or moderate or more) for a group of subjects determined to have dysphagia using the program shown in FIG. 5.

Example 3
Figure 6 is a boxplot showing the classification performance of a specific program created using the average value of acoustic features calculated from input speech data in which a subject read each of the 10 phrases shown in Figure 3 twice. The program distinguishes between healthy subjects, those with mild dysphagia, and those with moderate to severe dysphagia. The horizontal axis represents healthy subjects, those with mild dysphagia, and those with moderate to severe dysphagia, and the vertical axis represents the distribution of scores for the three groups. Figure 6 also confirms that the program has sufficient classification performance.

From Figures 4-6 above and the explanations related to those figures, it has been confirmed that a determination device that makes a determination using specific acoustic features is fully practical. Next, we will explain the correspondence between speech data and acoustic parameters using several specific examples. Please note that the following specific examples are examples of implementing the present invention, and the present invention is not limited to these examples.

Example 4
Figure 7 shows the relationship between the variation in the results of formant analysis of speech data when the same speech content is uttered multiple times and the degree of progression of the subject's dysphagia. Figure 7 uses speech data from Ph07, "Panda's Treasure," as the speech content. The horizontal axis represents the time axis for the speech of subjects with normal speech, mild dysphagia, and moderate to severe dysphagia, and the vertical axis represents the value of the first formant f1. The first and second utterances are grouped into multiple utterances, and their order is plotted along the time axis. Each subject uttered twice; the first eight subjects were normal (left graph), the next 17 subjects had mild dysphagia (center graph), and the next 18 subjects had moderate to severe dysphagia (right graph).

Figure 7 shows that for healthy subjects, the difference in f1 value between the first and second utterances is small. In contrast, for subjects with mild dysphagia and subjects with moderate to severe dysphagia, the difference in f1 value between the first and second utterances is large. Therefore, if the voice data input in step S201 is Ph07, in step S203, it is possible to determine based on f1 whether the subject is healthy, or a subject with mild or moderate to severe dysphagia.

For example, in step S203, f3 or f5 of a healthy individual is stored in advance, and if the subject's f3 or f5 deviates from the healthy individual's f3 or f5 by more than a threshold, the subject can be determined to have a swallowing disorder. While Figure 7 only shows f1 of Ph07, the effectiveness can also be confirmed for other voice data and other formant analysis results (e.g., any of f2 to f5 other than f1), as well as other frequency analyses. Also, while Figure 7 focuses on the difference in f1 values between two utterances, other numbers of utterances can also be used. For example, the degree of swallowing disorder (presence or absence and severity) can be determined based on the difference between the maximum and minimum values of formant analysis results from three or more utterances.

Example 5
FIG. 8 is a table comparing the speech data shown in FIG. 3 with the acoustic feature values based on formant frequencies, and also showing the degree of progression of dysphagia. The horizontal columns indicate "healthy vs. mild,""healthy vs. moderate or above," and "mild vs. moderate or above." The vertical columns indicate the content of the utterance. In the table, "***" indicates a P value of less than 0.01, "**" indicates a P value of less than 0.03333, "*" indicates a P value of less than 0.05, and "ns" indicates no significant difference. In the present application, a P value of less than 0.1 is considered to indicate a significant difference. Furthermore, f1 is the first formant, f2 is the second formant, f3 is the third formant, f4 is the fourth formant, and f5 is the fifth formant. To evaluate significant differences, a t-test (unpaired, one-sided) was used for testing significant differences between two groups, and a Bonferroni multiple comparison test was used for multiple comparison tests of three or more groups, with a significance level of 10%. Note that any of the following may be used to evaluate significant differences between groups: parametric tests including the t-test used here, non-parametric tests, tests based on ratios, tests based on variance ratios, etc.

From the results in Figure 8, Ph01 showed significant differences in f3 and f5 in the "healthy vs. mild" category. It also showed significant differences in f1 and f5 in the "healthy vs. moderate or severe" category. It also showed significant differences in f1 and f2 in the "mild vs. moderate or severe" category.

Furthermore, Ph02 showed significant differences in f4 and f5 in the "healthy vs. mild" category. It also showed significant differences in f3 to f5 in the "healthy vs. moderate or more" category. It also showed significant differences in f3 and f4 in the "mild vs. moderate or more" category.

Furthermore, Ph03 showed a significant difference in f3 in the "healthy vs. mild" category. It also showed a significant difference in f1 and f2 in the "healthy vs. moderate or more" category. It also showed a significant difference in f1 to f4 in the "mild vs. moderate or more" category.

Furthermore, Ph04 showed significant differences in f2 to f4 in the "normal vs. moderate or severe" category. Also, in the "mild vs. moderate or severe" category, significant differences were observed in f1 to f4. Ph04 did not show significant differences in any formants in the "normal vs. mild" category, but it is possible to determine "normal vs. mild" by comparing the other two groups.

Furthermore, Ph05 showed significant differences in f2 and f5 in the "healthy vs. mild" category. It also showed a significant difference in f1 in the "healthy vs. moderate or more" category. It also showed significant differences in f1 and f2 in the "mild vs. moderate or more" category.

Additionally, Ph06 showed significant differences in f3 and f5 in the "normal vs. mild" category. It also showed a significant difference in f5 in the "normal vs. moderate or severe" category. While Ph06 did not show significant differences in any formants in the "mild vs. moderate or severe" category, it is possible to determine "mild vs. moderate or severe" by comparing the other two groups.

Furthermore, Ph07 showed significant differences in f3 and f5 in the "healthy vs. mild" category. It also showed a significant difference in f1 in the "healthy vs. moderate or more" category. It also showed significant differences in f1 and f3 in the "mild vs. moderate or more" category.

Furthermore, Ph08 showed significant differences in f1, f3, and f5 in the "healthy vs. mild" category. It also showed a significant difference in f1 in the "healthy vs. moderate or more" category. It also showed significant differences in f1, f3, and f4 in the "mild vs. moderate or more" category.

Furthermore, Ph09 showed significant differences in f2 to f4 in the "healthy vs. mild" category. It also showed significant differences in f1, f4, and f5 in the "healthy vs. moderate or severe" category. It also showed significant differences in f3 and f4 in the "mild vs. moderate or severe" category.

Furthermore, Ph10 showed significant differences in f3 and f5 in the "healthy vs. mild" category. Also, in the "healthy vs. moderate or severe" category, significant differences were observed in f1, f3, and f5. Also, in the "mild vs. moderate or severe" category, significant differences were observed in f1 and f3.

From the above results, it was confirmed that Ph01 to Ph10 can distinguish between healthy and mild dysphagia, healthy and moderate or severe dysphagia, and mild and moderate or severe dysphagia in all phrases.

It was confirmed that by inputting speech data containing at least one of the sounds, it is possible to distinguish between healthy individuals and those with mild dysphagia, healthy individuals and those with moderate or severe dysphagia, and those with mild dysphagia and those with moderate or severe dysphagia.

Conventionally, formant analysis has tended to analyze the area from the lips to the tongue using the first and second formants, and therefore has tended to emphasize the first and second formants and neglect the third and subsequent formants. However, the inventors have clarified through research on speech analysis that, particularly when determining the degree of progression of dysphagia, the shape of the pharyngeal cavity behind the lips and the linkage of its complex shape changes cannot be understood without checking the third and subsequent formants. In particular, in elderly people with dysphagia, the shape of the pharyngeal cavity also changes, so performing speech analysis from f3 onwards is effective in determining the degree of progression of dysphagia.

Example 6
Furthermore, Figures 9A and 9B are tables showing one of the effective features for determining the degree of swallowing disorder (presence or absence of swallowing disorder and its severity) as a result of 13th-order Mel-frequency cepstrum analysis of the difference between data obtained by uttering each of the voice data shown in Figure 3 twice.

The table in Figure 9A uses the average values of two utterances of each piece of speech data shown in Figure 3, spoken twice, as acoustic parameters. The table in Figure 9B uses the differences between two utterances of each piece of speech data shown in Figure 3, spoken twice, as acoustic parameters. The horizontal items indicate the average, maximum, minimum, range, average minimum, and slope of the 13th-order Mel-frequency cepstrum coefficients or dynamic features. The vertical items indicate the content of the utterance. In the table, a "○" (circle) indicates that the item is effective in determining the presence or absence of dysphagia. A double circle indicates that the item is effective in determining the degree of progression of dysphagia.

Figures 9A and 9B confirm that even when 13th-order Mel frequency cepstrum analysis is used as speech analysis, it is possible to distinguish between healthy and mild dysphagia, healthy and moderate to severe dysphagia, and mild and moderate to severe dysphagia.

Second Embodiment
Next, a second embodiment will be described. Note that the configuration of the determination device of the second embodiment is similar to that of the first embodiment, so the same reference numerals are used and the description thereof will be omitted.

The second embodiment of the determination device differs from the first embodiment in that it uses phoneme analysis when determining whether a subject has swallowing disorders.

The analysis means of the determination device of the second embodiment creates acoustic features by performing phonemic analysis on the speech data uttered by the subject.The determination means of the determination device of the second embodiment then performs speech analysis using the acoustic features that are the analysis results of the analysis means, thereby determining the degree of progression of the swallowing disorder.

Example 7
FIG. 10 is a diagram showing the relationship between phoneme classes and activated phonemes. Note that the same phoneme may belong to multiple phoneme classes. For example, as shown in FIG. 10, the phoneme /a/ belongs to "vocalic,""back,""open," and "voiced," while the phoneme /p/ belongs to "consonantal,""stop," and "labial." FIG. 11 shows the results of phoneme analysis performed on the "pa" uttered by a healthy subject without dysphagia and the "pa" uttered by a subject with moderate or severe dysphagia.

(A), (C), (E), (G), and (I) in Figure 11 are the phoneme analysis results for "pa" spoken by a healthy subject, while (B), (D), (F), (H), and (J) in Figure 11 are the phoneme analysis results for "pa" spoken by a subject with moderate to severe dysphagia. Note that the horizontal axis of each graph in Figure 11 represents time (s), and the vertical axis represents the phoneme posterior probability (phonological posteriors) of each phoneme class. The waveform in the center of Figure 11 corresponds to the speech data of the speech spoken by the subject.

(A) and (B) in Figure 11 are the phoneme analysis results when the phoneme classes are "vocalic," "back," "consonantal," and "anterior." (C) and (D) in Figure 11 are the phoneme analysis results when the phoneme classes are "open," "nasal," "close," and "stop." (E) and (F) in Figure 11 are the phoneme analysis results when the phoneme classes are "continuant," "flap," "lateral," and "trill." (G) and (H) in Figure 11 are the phoneme analysis results when the phoneme classes are "voice," "labial," "strident," and "dental." (I) and (J) in Figure 11 are the phoneme analysis results when the phoneme classes are "velar" and "pause."

As shown in Figure 11, there are significant differences in the waveforms of phoneme classes, which are the results of phoneme analysis, between healthy individuals and individuals with moderate to severe dysphagia. For this reason, the results of phoneme analysis are thought to be useful as acoustic features for determining whether or not a person has dysphagia.

Figures 12 to 17 show box-and-whisker plots verifying the classification performance of each acoustic feature for the presence or absence of dysphagia, which is the result of phoneme analysis of phoneme classes. Note that "mean" after each phoneme class in the figures (e.g., "consonantal," "close," "dental," "velar," "stop," "anterior," "back," "continuant," "open," and "labial") represents the average, "median," and "std" the standard deviation. Also, in Figures 12 to 17, "healthy" represents healthy individuals, "mild" represents individuals with mild dysphagia, and "severe" represents individuals with moderate or severe dysphagia. The vertical axes in Figures 12 to 17 represent the values of each statistical quantity, such as the mean, median, and standard deviation. Specifically, the vertical axes in FIGS. 12 to 17 represent the average, median, standard deviation, etc. of the acoustic feature values of the phoneme class at each time when a subject uttered a certain phrase.

Figure 12 shows the statistics of acoustic features when a subject utters the sound "pa." As shown in the upper part of Figure 12, when distinguishing between healthy individuals and individuals with mild or severe dysphagia, useful acoustic features include "close_std," the standard deviation of the phoneme class "close," and "consonantal_std," the standard deviation of the phoneme class "consonantal." Furthermore, as shown in the lower part of Figure 12, when distinguishing between healthy individuals and individuals with mild or severe dysphagia, useful acoustic features include "close_mean," the mean of the phoneme class "close," and "close_median," the median of the phoneme class "close."

Figure 13 also shows the statistics of acoustic features when a subject utters "pa." As shown in Figure 13, when distinguishing between healthy individuals and individuals with mild swallowing disorders and those with moderate or severe swallowing disorders, it can be seen that, as examples, "dental_mean," which is the mean value of the phoneme class "dental," "dental_median," which is the median value of the phoneme class "dental," and "velar_median," which is the median value of the phoneme class "velar," are useful acoustic features.

Figures 14 and 15 show the statistics of acoustic features when subjects uttered the "ra" sound. As shown in Figure 14, when distinguishing between healthy individuals, individuals with mild swallowing disorders, and individuals with moderate or severe swallowing disorders, useful acoustic features include, for example, "close_mean" which is the mean value of the phoneme class "close", "close_median" which is the median value, "dental_std" which is the standard deviation of the phoneme class "dental", and "stop_mean" which is the mean value of the phoneme class "stop".

Furthermore, as shown in Figure 15, when distinguishing between healthy individuals, individuals with mild swallowing disorders, and individuals with moderate or severe swallowing disorders, it can be seen that, as an example, the mean value of the phoneme class "velar," "velar_mean," and the median value, "velar_median," are useful acoustic features.

Figures 16 and 17 show the statistics of acoustic features when the subject pronounced "tamago" (egg). Although detailed explanations of Figures 16 and 17 will be omitted, similar to Figures 12 to 15, the acoustic features of each phoneme class are useful for determining the degree of swallowing disorder.

As illustrated in Figures 12 to 17, the statistics for each phoneme class are useful for determining the degree of swallowing disorder. Therefore, it can be confirmed that it is possible to distinguish between normal and mild dysphagia, normal and moderate or severe dysphagia, and mild and moderate or severe dysphagia.

For this reason, the analysis means of the determination device of the second embodiment performs phonemic analysis on the speech data of the speech uttered by the subject to identify the phonemes contained in the speech data. The analysis means of the determination device of the second embodiment then identifies the phoneme class to which the phonemes contained in the speech data belong. Note that the same phoneme may belong to multiple phoneme classes. For example, as shown in Figure 10, the phoneme /a/ belongs to "vocalic," "back," "open," and "voiced," while the phoneme /p/ belongs to "consonantal," "stop," and "labial." For this reason, the analysis means of the determination device of the second embodiment calculates the phoneme posterior probabilities (phonological posteriors) of the phoneme class for each time point in the speech data, which is time-series data. The determination means of the determination device of the second embodiment determines the degree of swallowing disorder of the subject based on statistics of the phoneme posterior probability of the phoneme class at each time point obtained by the analysis means (e.g., the mean, median, and standard deviation of the phoneme posterior probability at each time point). Therefore, according to the determination device of the second embodiment, the degree of swallowing disorder of the subject can be accurately determined by phoneme analysis of the speech data of the speech uttered by the subject.

Third Embodiment
Next, a third embodiment will be described. Note that the configuration of the determination device of the third embodiment is similar to that of the first embodiment, so the same reference numerals are used and the description thereof will be omitted.

The determination device of the third embodiment differs from the first and second embodiments in that it uses voice intensity analysis when determining a subject's swallowing disorder.

The analysis means of the determination device of the third embodiment creates acoustic features by performing a voice intensity analysis on the voice data uttered by the subject.The determination means of the determination device of the third embodiment then performs a voice analysis using the acoustic features that are the analysis results of the analysis means, thereby determining the degree of progression of the swallowing disorder.

(Example 8)
Fig. 18 is a diagram showing audio data. The horizontal axis of Fig. 18 represents time, and the vertical axis represents audio intensity. Note that black circles in Fig. 18 represent peak points of the audio data.

The top four rows of Figure 19 are experimental results showing the correlation between the voice intensity analysis results and the swallowing disorder assessment results for phrases Ph09 "Banana banana banana... banana" and Ph10 "Kimono kimono kimono... kimono" uttered by the subject. In Figure 19, "peak_ave" represents the average peak point of the voice data intensity, "peak_sd" represents the standard deviation of the peak points of the voice data intensity, "peak_span_ave" represents the average interval between peak points of the voice data intensity, and "peak_span_sd" represents the standard deviation of the interval between peak points of the voice data intensity.

Figure 19 shows the results of the Bonferroni test using "peak_ave," "peak_sd," "peak_span_ave," and "peak_span_sd" to distinguish between healthy individuals and those with mild or severe swallowing disorders, and between healthy individuals, those with mild swallowing disorders, and those with moderate or severe swallowing disorders. As shown in Figure 19, it can be seen that using "peak_sd" of Ph09 is effective for distinguishing between healthy people and people with mild swallowing disorders and people with moderate or severe swallowing disorders, using "peak_span_sd" of Ph09 is effective for distinguishing between healthy people and people with mild swallowing disorders and people with moderate or severe swallowing disorders, using "peak_ave" of Ph10 is effective for distinguishing between healthy people and people with mild swallowing disorders and people with moderate or severe swallowing disorders, and using "peak_sd" of Ph10 is effective for distinguishing between healthy people and people with mild swallowing disorders and people with moderate or severe swallowing disorders.

Figures 20 and 21 are box-and-whisker plots of each acoustic feature.

The left side of Figure 20 is a box-and-whisker plot of the acoustic feature "peak_sd" for phrase Ph09, which shows an association with the symptoms of swallowing disorders and is particularly effective in distinguishing people with moderate to severe swallowing disorders.

The right side of Figure 20 shows a box-and-whisker plot of the acoustic feature "peak_span_sd" for phrase Ph09, which shows a correlation with the symptoms of swallowing disorders and is particularly effective in distinguishing people with moderate to severe swallowing disorders.

Figure 21 shows a box-and-whisker plot of the acoustic feature "peak_sd" for phrase Ph10, which shows a correlation with the symptoms of swallowing disorders and is particularly effective in distinguishing people with moderate to severe swallowing disorders.

As shown in Figures 19 to 21, the results of intensity analysis of voice data are useful for determining the degree of swallowing disorder. Therefore, it can be confirmed that it is possible to distinguish between healthy and mild dysphagia, healthy and moderate or severe dysphagia, and mild and moderate or severe dysphagia.

For this reason, the analysis means of the assessment device of the third embodiment generates acoustic features related to the intensity analysis by performing intensity analysis on the speech data of the speech uttered by the subject. For example, the analysis means of the assessment device of the third embodiment generates the above-described data as acoustic features related to the intensity analysis. The assessment means of the assessment device of the third embodiment then assesses the degree of swallowing disorder of the subject based on the acoustic features obtained by the analysis means. Therefore, the assessment device of the third embodiment can accurately assess the degree of swallowing disorder of the subject by performing intensity analysis on the speech data of the speech uttered by the subject. Note that in the above example, phrases Ph09 "banana banana...banana" and Ph10 "kimono kimono kimono...kimono" were used as examples of phrases uttered by the subject, but these are not limited to these and any phrase that includes a repetition of a predetermined phrase may be used. For example, phrases such as "patakapatakapataka," "papapapa," and "tatatatata" may be used. Such phrases generate nasal resonance (nasal resonance) and are considered useful for assessing the degree of swallowing disorder. Furthermore, by employing phrases that include repetition, it is possible to determine the consistency of the rhythm that occurs due to repetition, which is also thought to be useful in determining the degree of swallowing disorder.

The phoneme analysis of the second embodiment and the intensity analysis of the third embodiment may be combined to determine the degree of a subject's dysphagia. In this case, the analysis means of the determination device determines the degree of the subject's dysphagia using acoustic features created by performing phoneme analysis on the speech data and acoustic features created by performing intensity analysis on the speech data. This allows the degree of the subject's dysphagia to be determined more accurately than when using only one of phoneme analysis and intensity analysis. The combination of acoustic features is not limited to a combination of acoustic features related to phoneme analysis and acoustic features related to intensity analysis, but may be any combination including formant frequency, Mel frequency cepstrum, frequency spectrum, speech envelope, waveform variation information, zero-crossing rate, Hurst exponent, and the time from closure/opening to the onset of vocal cord vibration.

Example 9
22 and 23 are ROC curves obtained when determining the degree of a subject's dysphagia using acoustic features generated by performing phonemic analysis on speech data and acoustic features generated by performing intensity analysis on speech data. FIGS. 22 and 23 are ROC curve results obtained when determining whether a subject has mild or severe dysphagia ( FIG. 22 ) or moderate or severe dysphagia ( FIG. 23 ) using a machine learning model using an acoustic feature set. More specifically, FIGS. 22 and 23 are ROC curves obtained when determining the degree of a subject's dysphagia using four acoustic features: acoustic features generated based on formant frequencies, acoustic features generated based on Mel frequency cepstrum, acoustic features generated by performing phonemic analysis, and acoustic features generated by performing intensity analysis. FIG. 22 is an ROC curve obtained when determining whether a subject has mild or severe dysphagia. FIG. 23 is an ROC curve obtained when determining whether a subject has moderate or severe dysphagia. The AUC when determining whether a subject has mild or severe dysphagia, as shown in Figure 22, was 0.9954 (P value < 0.001), confirming that the system has sufficient classification performance. Furthermore, the AUC when determining whether a subject has moderate or severe dysphagia, as shown in Figure 23, was 1.000 (P value < 0.001), confirming that the system has sufficient classification performance. Therefore, it was confirmed that the degree of a subject's dysphagia can be accurately determined by using acoustic features created by performing phonemic analysis on speech data and acoustic features created by performing intensity analysis on speech data.

The above describes preferred embodiments of the present disclosure, but the present disclosure is not limited to the above-described embodiments. The object of the present disclosure can also be achieved by providing a storage medium storing program code (computer program) that realizes the functions of the above-described embodiments to a system or device, and having the computer of the provided system or device read and execute the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present disclosure. Furthermore, in the above-described embodiment, the computer executes the program to function as each processing unit, but some or all of the processing may be configured using dedicated electronic circuits (hardware). The present disclosure is not limited to the specific embodiments described, and various modifications and variations are possible within the spirit of the present disclosure as defined in the claims, including the substitution of each component of each embodiment.

The disclosure of Japanese Patent Application No. 2021-159606, filed on September 29, 2021, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards mentioned herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims (11)

A determination device for determining a swallowing state by voice analysis, the determination device comprising:
an input means for receiving input of voice data , which is time-series data of voice uttered by the subject;
analysis means for analyzing the voice data received by the input means;
a determination means for determining the swallowing state of the subject based on the analysis result by the analysis means;
Equipped with
the analysis means performs phonemic analysis on the speech data to identify phonemes contained in the speech data, identify phoneme classes to which the phonemes contained in the speech data belong, and calculate phoneme posterior probabilities (phonological posteriors) of the phoneme classes at each time point in the speech data;
the determining means determines the degree of dysphagia of the subject based on the phoneme posterior probability of the phoneme class at each time point obtained by the analyzing means.
Judgment device. The analysis means further calculates an acoustic feature of the speech data as shown in the following formula F(a):
the determining means determines the degree of dysphagia of the subject based on the phoneme posterior probability of the phoneme class at each time point obtained by the analyzing means and the acoustic feature.
The determination device according to claim 1:

(wherein g is a linear or nonlinear model that determines the presence or absence of a swallowing state and the degree of progress, _xn is a coefficient specific to the phrase input as the speech data, and f(n) is an acoustic parameter, and the acoustic parameter is one or more selected from the group consisting of formant frequency, Mel frequency cepstrum, frequency spectrum, speech envelope, waveform fluctuation information, zero crossing rate, Hurst exponent, and time from closure/opening to the start of vocal cord vibration). The analysis means further acquires acoustic features created based on formant frequencies or Mel-frequency cepstrum of the speech data,
the determining means determines the degree of dysphagia of the subject based on the phoneme posterior probability of the phoneme class at each time point obtained by the analyzing means and the acoustic feature.
The determination device according to claim 1 . the determination means inputs the phoneme posterior probability of the phoneme class obtained by the analysis means into a machine learning model that has been subjected to machine learning processing so as to output the degree of dysphagia of the subject when the phoneme posterior probability of the phoneme class is input, thereby obtaining the degree of dysphagia of the subject output from the machine learning model and determining the degree of dysphagia of the subject;
The determination device according to claim 1 . the input means further receives each of the speech data when the subject utters the same phrase twice;
when determining the degree of dysphagia of the subject based on the acoustic features, the determination means determines the degree of dysphagia of the subject based on a difference or an average value of the acoustic features obtained from two pieces of speech data corresponding to each of the same phrases uttered two times by the subject.
The determination device according to claim 2 or 3 . the input means receives input of the voice data when the subject utters a phrase related to the degree of the swallowing disorder .
The determination device according to claim 1 or 2. The input means is

and receiving input of the voice data when the user speaks, the voice data including at least one of the sounds of
The determination device according to claim 1 or 2. The analysis means further acquires acoustic features created by performing a voice intensity analysis on the voice data,
the determining means determines the degree of dysphagia of the subject based on the phoneme posterior probability of the phoneme class at each time point obtained by the analyzing means and the acoustic feature.
The determination device according to claim 1 . The voice data is voice data when the subject repeatedly speaks a predetermined phrase.
The determination device according to claim 1 or 2. 1. A method for determining swallowing status by a computer using speech analysis, the method comprising:
an input step of receiving input of voice data which is time-series data of voice uttered by the subject;
an analyzing step of analyzing the voice data received in the input step;
a determination step of determining the swallowing state of the subject based on the analysis result obtained by the analysis step;
Equipped with
the analyzing step performs a phonemic analysis on the speech data to identify phonemes included in the speech data, identify phoneme classes to which the phonemes included in the speech data belong, and calculate phoneme posterior probabilities (phonological posteriors) of the phoneme classes at each time point of the speech data;
the determining step determines the degree of dysphagia of the subject based on the phoneme posterior probability of the phoneme class at each time point obtained by the analyzing step.
The way a computer performs a process. A program for causing a computer to function as each of the means of the determination device described in claim 1 or claim 2.