JP4586675B2 - Vocal tract cross-sectional area function estimation apparatus and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate the cross-sectional area function of a vocal tract in a real time with high precision using a parameter easy to measure. <P>SOLUTION: The estimation system 100 of the cross-sectional area function of the vocal tract includes a parameter calculation part 118 for calculating the average of the cross-sectional area function of the vocal tract and the coefficient for making a first and second main components and the weight, which makes the cross-sectional area function of the sound tract during continuous vocalization approximate by the first and second components, approximate by the linear sum of the cepstrum coefficient determined from vocal sound data 116 on the basis of a stationary MRI image 110 during vocalization of a vowel, an MRI motive image 114 during continuous vacalization and the sound data 116 during continuous vocalization, a cepstrum coefficient calculation part 182 for calculating the cepstrum coefficient at every frame, a linear conversion processing part 184 for calculating the weights of the main components with respect to respective frames on the basis of the linear sum using the coefficient, which is calculated by the parameter calculation part 118, with respect to the cepstrum coefficient and a vocal sound cross-sectional area function estimating part 186 for estimating the cross section of the vocal tract with respect to the respective frames using the weights and the main components. <P>COPYRIGHT: (C)2007,JPO&amp;INPIT

Description

  The present invention relates to a technique for analyzing a human articulation mechanism using a transmission image such as an MRI (Magnetic Resonance Imaging) image, and in particular, based on a static transmission image and a transmission moving image, based on a voice obtained from a human speech, The present invention relates to an apparatus for estimating a vocal tract cross-sectional area function that estimates a vocal tract cross-sectional area function at the time of utterance in real time.

  The conversion from acoustic to articulator structure is called an inverse estimation method, but is generally known to be nonlinear and not have a one-to-one relationship. This is due to a compensatory relationship between articulators (including the sound source), which can result in very similar sound quality from different vocal tract shapes.

  However, the difficulty of these problems is ultimately dependent on the shape and acoustic parameters of the articulation mechanism selected, and can be solved by imposing appropriate constraints. There are two of the most important types of constraints proposed in the prior art. One that proposes a human-type model that more closely simulates the human vocalization mechanism (Non-Patent Document 1) and one that imposes continuity on the temporal trajectory of the articulatory organ (Non-Patent Document 2).

Instead, if the co-variation that occurs naturally between the articulators can be captured with fewer degrees of freedom, there is a possibility that the model can be easily used in the inverse estimation method without further restrictions. is there. In fact, reducing the dimensionality of such measured articulator data has become a concern in several studies on vowel generation. For American English vowels, 90% of the changes in vocal tract shape can be explained by only two orthogonal components that describe the change centered on the average. Similar components have also been reported in Icelandic vowels, and research on German vowels supports their effectiveness across languages. In fact, according to the similarity of the results obtained by the experiment using the model described in Non-Patent Document 3, not only these two components are independent of language but also the anatomy of the human vocal tract The hypothesis that it is intrinsic to the physics and biomechanical attributes seems reasonable.
J. et al. Schleter et al., “Technique for Estimating Vocal Tract Shape from Speech Signal”, IEEE Speech and Acoustic Processing Papers, 2 (1), pp. 133-150, 1994 (Schroeter, J. & Sondhi, MM, "Techniques for collaborative vocal-tract shapes from the speech signal", IEEE Trans. Speech & Audio Proc., 2 (1), 133-150, 1994. ) J. et al. Dan et al., “Estimation of vocal tract shape from uttered speech using physiological articulation model”, Journal of Phonetics, 30 (3), pp. 511-532, 2002 (Dang, J. & Honda, K., "Estimation of vocal tract shapes from speech sounds with a physiological articulatory model", J. Phonetics., 30 (3): 511-532, 2002.) P. Perrier et al., “The degree of freedom of tongue movement in speech may be limited by biomechanics”, Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP), Beijing, China, Vol. 162-165, 2000 (Perrier, P., Perkell, J., Payan, Y., Zandipour, M., Guenther, F. & Khalighi, A., "Degrees of freedom of tongue movements in speech may be constrained by biomechanics ", in Proc. 6th Int. Conf. Spoken Lang. Process. (ICSLP), Beijing, China, Vol. II: 162-165, 2000.) B. H. Story et al., “Parameterization of vocal tract cross section function by empirical orthogonal mode”, Journal of Phonetics, 26 (3), pp. 223-260, 1998 (Story, BH, & Titze, IR, "Parameterization of vocal tract area functions by empirical orthogonal modes", J. Phonetics, 26 (3): 223-260, 1998.) P. Radfosid et al., “Generation of vocal tract shape from formant frequencies”, Journal of the Acoustical Society of America, 64 (4), pp. 1027-1035, 1978 (Ladefoged, P., Harshman, R., Goldstein, L. & Rice, L., "Generating vocal tract shapes from formant frequencies", J. Acoust. Soc. Am., 64 (4) : 1027-1035, 1978.) H. Wakita, “Normalization of vowels using vocal tract length and its application to vowel identification”, IEEE Sound, Speech, Signal Processing, 25 (2), pp. 183-192, 1977 (Wakita, H. "Normalization of vowels by vocal-tract length and its application to vowel identification", IEEE Transactions on Acoustics, Speech, and Signal Processing, 25 (2), 183-192, 1977)

  Therefore, there is a possibility that good results can be obtained by using the above-mentioned two components related to the vocal tract shape in the inverse estimation method. Such attempts themselves have been made in the past (Non-Patent Document 4). However, this conventional technology uses a formant of synthesized speech, and therefore, the inverse estimation method learns the characteristics of the specific vocal tract transmission line model used to obtain the formant, and the acoustic characteristics of the human vocal tract are It cannot be learned.

  In contrast, in Non-Patent Document 5, linear regression is applied to formants measured by means that are not easy from sound recorded simultaneously with the transmission method, and the tongue shape for each formant is determined. It has been found that there is a correlation of 0.935 and 0.902 with the element, respectively.

  However, it is well known that even for vowels whose sound generation period is long and stable, formants cannot be measured stably and accurately with simple processing. Therefore, until now, many studies have emphasized the importance of using acoustic parameters that can be measured stably and accurately for inverse estimation, but no specific measures have been given. .

  Accordingly, an object of the present invention is to provide a vocal tract cross-sectional area function estimation device capable of performing highly accurate estimation in real time using parameters that can be easily measured.

  According to the first aspect of the present invention, an apparatus for estimating a vocal tract cross-sectional area function includes, for each of a plurality of vowels, a still transmission image at the time of each utterance of a predetermined speaker, and a plurality of vowels. The average of the vocal tract cross-sectional area function of each speaker at the time of each utterance based on the transparent moving image of the predetermined speaker at the time of continuous utterance and the voice data of the predetermined speaker at the time of continuous utterance. To approximate the first to m-th principal components (m is a natural number equal to or greater than 2) representing the change of, and the vocal tract cross-sectional area function of the speaker during continuous speech by the average and the first to m-th principal components The linear sum of the first to n-th order cepstrum coefficients (n is a predetermined natural number of 10 or more) obtained by performing cepstrum analysis on the speech data of a predetermined speaker with the weights of the first to m-th principal components. Parameter preparation means for preparing the coefficients for approximation with the sound of the speaker First cepstrum analysis means for performing a cepstrum analysis for each frame on a framed audio signal obtained by capturing and calculating first to n-th order cepstrum coefficients for each frame; A first sum for each frame of the speech signal of the speaker is obtained by using a linear sum using a coefficient prepared by the parameter preparation means for the first to n-th order cepstrum coefficients calculated by the first cepstrum analysis means. A linear conversion unit for calculating a weight for the m-th principal component, a weight for the first to m-th principal component calculated by the linear conversion unit, an average prepared by the parameter preparation unit, and Vocal tract cross-sectional area estimating means for estimating the vocal tract cross-sectional area for each frame of the speaker's voice signal using the first to m-th principal components.

  From the static transmission image, an average value of the vocal tract cross-sectional area function at the time of utterance of each vowel and first to m-th principal components representing fluctuations centered on the average value are prepared. From the transmitted moving image, a first for approximating the vocal tract cross-sectional area function at the time of continuous speech with the average value of the vocal tract cross-sectional area function obtained from the static transmitted image and the first to m-th principal components. A weight for the m-th principal component is prepared for each frame. Coefficients for approximating these weights with a linear sum are prepared using first-order to n-th order cepstrum coefficients obtained by cepstrum analysis on speech data. After these are prepared, first-order to n-th cepstrum coefficients are obtained from the framed speech signal obtained by capturing the speech of the speaker. A weight for the first to m-th principal components for each frame is obtained by performing linear transformation on the cepstrum coefficients using the coefficients prepared by the parameter preparation means. An approximate value of the vocal tract cross-sectional area function is calculated for each frame using this weight, the average of the vocal tract cross-sectional area function, and the first to m-th principal components. Highly accurate estimation can be performed in real time using a cepstrum coefficient, which is an easily measured parameter, instead of a difficult parameter such as formant.

  More preferably, the parameter preparation means includes a plurality of the plurality of vowels based on a vocal tract cross-sectional area function obtained from a static transmission image of a vocal tract portion at the time of individual speech of a predetermined speaker. Principal component analysis means for obtaining the average of the vocal tract cross-sectional area function relating to the vowels and the first to mth principal components by principal component analysis, and the vocal tract during continuous utterance obtained for each frame from the transmitted video By projecting the cross-sectional area function to the space formed by the first to m-th principal components, the vocal tract cross-sectional area function at the time of continuous speech is expressed by the first to m-th principal components and the average. First to n-th order cepstrum coefficients are obtained by performing weight cepstrum analysis on voice data of a predetermined speaker at the time of continuous speech and weight calculation means for calculating the weights of the first to m-th principal components. Second cepstrum for Multiple regression analysis for approximating the weights of the first to m-th principal components at the time of continuous speech with a linear sum of first to n-th cepstrum coefficients obtained by the analyzing means and the second cepstrum analyzing means Storage means for storing coefficients for approximating the weights of the average, the first and second principal components, and the first to m-th principal components Including.

  By providing the above-described means for calculating each parameter, it is possible to perform both parameter learning and actual processing with a single vocal tract cross-sectional area function estimation device.

  The first cepstrum analysis unit and the second cepstrum analysis unit may perform cepstrum analysis on an audio signal in the same frequency band.

  By using a cepstrum coefficient obtained from an audio signal in the same frequency band as that prepared by the parameter preparation means during operation, the vocal tract cross-sectional area function can be correctly estimated.

  Preferably, both the first cepstrum analysis unit and the second cepstrum analysis unit perform cepstrum analysis on an audio signal in a frequency band of 0 to 4 kHz.

  Although it depends on the speaker, it was found that the vocal tract cross-sectional area function can be accurately estimated by using the result of the cepstrum analysis for the speech signal in this frequency band.

  More preferably, the parameter preparation means includes storage means for storing the average of the vocal tract cross-sectional area function, the first to m-th principal components, and the coefficients. m = 2 and n = 24 may be sufficient.

  According to experiments, if the first and second principal components are used as the principal components, the estimation can be performed with sufficient accuracy. Further, when the first to 24th orders are used as the cepstrum coefficients, the estimation accuracy is the highest.

  The vocal tract cross-sectional area function may include the cross-sectional area of the vocal tract at a predetermined number of section positions equally spaced from each other on the center line of the predetermined speaker's vocal tract and the distance between the section positions. Good.

  By performing the principal component analysis including the distance between the section positions, it is possible to perform the estimation including the vocal tract length which varies depending on the vowel with high accuracy.

  A computer program according to the second aspect of the present invention, when executed by a computer, causes the computer to operate as one of the above vocal tract cross-sectional area function estimation devices. Therefore, it is possible to obtain the same effect as the above vocal tract cross-sectional area function estimation device.

<Configuration>
FIG. 1 shows a block diagram of a vocal tract cross-sectional area function estimation system 100 that estimates and displays a vocal tract cross-sectional area function from speech in real time using an inverse estimation method according to an embodiment of the present invention. Referring to FIG. 1, vocal tract cross-sectional area function estimation system 100 is a vocal tract when vowels “a”, “i”, “c”, “e”, and “o” are uttered for a specific speaker. 5 sets of still MRI images 110 obtained by photographing images, and MRI moving images 114 obtained by capturing a moving image of the vocal tract when the same speaker is making a series of utterances of “Aiueo”, and the same speaker Includes a parameter calculation unit 118 for calculating parameters necessary for the inverse estimation method, using the voice data 116 recorded from the voice of when “Iueo” is making a series of utterances. Ideally, the voice data 116 may be recorded by the voice of the speaker when the MRI moving image 114 is shot. However, in reality, a very loud sound is generated in the MRI shooting. It is difficult to perform simultaneously. Therefore, in the present embodiment, the audio data 116 is recorded at a different time from when the MRI moving image 114 is captured. However, since both of them need to be synchronized with each other, the speaker is put on the headphones, and the regular synchronization sound generated by the synchronization sound generator 112 is heard from the speaker through the headphones, and the synchronization sound is adjusted. To utter “Iueo”. When recording the audio data 116, the posture of the speaker was also the same as when the MRI moving image 114 was captured so that the shape of the vocal tract matched as much as possible when the MRI moving image 114 was captured.

  The vocal tract cross-sectional area function estimation system 100 further uses the parameters calculated by the parameter calculation unit 118 to inversely estimate the vocal tract cross-sectional area function in real time from the voice of the speaker converted into the voice signal by the microphone 122. And an inverse estimation processing unit 120 for displaying on the display device 124.

  The parameter estimated by the parameter calculation unit 118 includes the first component of the principal component analysis (PCA) regarding the average of the vocal tract cross-sectional area function of the utterance during the utterance of each vowel estimated from the still MRI image 110 and the variation thereof. This is a regression coefficient obtained by multiple regression analysis between the value of the second component and the change in weight for the two principal components obtained from the MRI moving image 114 and the cepstrum coefficient obtained from the audio data 116. The details of this multiple regression analysis will be described later, but the obtained regression coefficient is the cepstrum coefficient obtained from the speech data 116 by the weights (changes) of the two principal components when the five vowels are uttered continuously. This is the weight of each cepstrum coefficient to approximate with the linear sum of.

  In the present embodiment, the MRI moving image 114 is composed of images of 35 consecutive frames obtained by photographing a vocal tract shape when a speaker continuously utters “Aiweo”. Preferably, the MRI moving image 114 is obtained from a plurality of times of imaging. By using the synchronization sound generator 112, it is possible to utter “eye ueo” at a fixed timing even in a plurality of times of shooting.

The parameter calculation unit 118 obtains a vocal tract cross-sectional area function at the time of utterance of each vowel from each of the five sets of MRI still images obtained for the five vowels of the still MRI image 110, and performs PCA processing on them. The first and second principal components (“PC I” and “PC II”, respectively) that mainly control the average value of these vocal tract cross-sectional area functions and the variation of the vocal tract cross-sectional area function centered on the average value. PCA processing unit 140 for outputting), an average value of vocal tract cross-sectional area functions output from PCA processing unit 140, PCIs and IIs, and predetermined constants associated therewith are described below. The average value of the vocal tract cross-sectional area function stored in the storage unit 142 with respect to the time-varying vocal tract cross-sectional area function obtained from the storage unit 142 and the MRI moving image 114, and the PCI I And II Thus, a moving image data projection processing unit for calculating change curves of weights k ₁ and k ₂ for each frame with respect to PC I and II corresponding to changes in the vocal tract cross-sectional area function represented by the MRI moving image 114 144 and a storage unit 146 for storing the weights k ₁ and k ₂ for each frame for the PCs I and II calculated by the moving image data projection processing unit 144. Details of the processing in the PCA processing unit 140 and the processing in the moving image data projection processing unit 144 will be described later.

The parameter calculation unit 118 further performs a discrete cosine transform (DCT) on a spectrum sample in a predetermined frequency band subjected to FFT (Fast Fourier Transform) on the dB scale for continuous utterances related to “Aiweo” obtained from the audio data 116. A cepstrum coefficient calculation unit 148 for calculating a predetermined number of cepstrum coefficients (c _{1 to} c _N ) excluding (c ₀ ), and a memory for storing the cepstrum coefficients calculated by the cepstrum coefficient calculation unit 148 The following linear format is obtained by a multiple regression analysis between the time variation curve of the weights k ₁ and k ₂ of the PC I and II stored in the storage unit 146 and the cepstrum coefficient stored in the storage unit 150. for calculating a regression coefficient alpha _m0 to? _mn approximating the _{k 1} and _{k 2} by solving And a regression analysis processing unit 152.

ただしＮは算出されたケプストラム係数の数、Ｍは主成分の数、α_ｍ0は定数、α_ｍｎ（ｎ＝１〜Ｎ）はｍ番目の主成分を近似するための、ｎ番目のケプストラム係数ｃ_ｎにかかる回帰係数である。本実施の形態では、Ｎ＝２４、Ｍ＝２であり、α_ｍ0及びα_ｍ1〜α_ｍ２４は、動画像データ射影処理部１４４により算出されたｋ_１及びｋ_２と重回帰分析処理部１５２により算出されたケプストラム係数ｃ１〜ｃ２４とを式（１）に代入してα_ｍ0及びα_ｍ1〜α_ｍ２４について式（１）を解く事により得られる。式（１）により、この重回帰分析により得られる重回帰モデルが表わされる。

Where N is the number of calculated cepstrum coefficients, M is the number of principal components, α _m0 is a constant, and α _mn (n = 1 to N) is the n th cepstrum coefficient c for approximating the m th principal component. It is a regression coefficient concerning _n . In this embodiment, N = 24 and M = 2, and α _m0 and α _{m1 to} α _m24 are calculated by k ₁ and k ₂ calculated by the moving image data projection processing unit 144 and the multiple regression analysis processing unit 152. It is obtained by substituting the calculated cepstrum coefficients c1 to c24 into equation (1) and solving equation (1) for α _m0 and α _{m1 to} α _m24 . The multiple regression model obtained by this multiple regression analysis is expressed by equation (1).

Parameter calculation unit 118 further includes a storage unit 154 for storing regression coefficients α _{m1 to} α _m24 calculated by multiple regression analysis processing unit 152.

On the other hand, the inverse estimation processing unit 120 captures the audio signal obtained from the microphone 122 and converts it into a digital signal by the audio capture processing unit 180 and an audio capture processing unit 180 for converting the audio signal into a framed digital signal. The cepstrum coefficient calculation unit 182 for calculating cepstrum coefficients c _{1 to} c ₂₄ for each frame by performing the same processing as the cepstrum coefficient calculation unit 148 on the audio signal, and the cepstrum coefficient calculated by the cepstrum coefficient calculation unit 182 c ₁ to c _24, and using the storage unit 154 regression coefficients alpha _m0 and alpha _m1 to? _m24 are stored in the weight _{k 1} and _k for the PC I and II according to a linear transformation represented by formula (1) ₂ Each of the frames is calculated by a linear conversion processing unit 184 and a linear conversion processing unit 184. Of the vocal tract cross-sectional area function in each frame according to the formula described later, using the weights k ₁ and k ₂ calculated for the image and the average vocal tract cross-sectional area function and PCI I and II stored in the storage unit 142. A vocal tract cross-sectional area function estimating unit 186 for calculating an estimated value. Details of the functions of the linear transformation processing unit 184 and the vocal tract cross-sectional area function estimation unit 186 will be described later. In the present embodiment, the vocal tract cross-sectional area function calculated for each frame in this way is given to the display device 124 and displayed. By this display, for example, it is possible to confirm a change in the vocal tract cross-sectional area function when the subject utters a predetermined utterance. How to change the vocal tract cross-sectional area function for correct pronunciation if the vocal tract cross-sectional area function is displayed along with this vocal tract cross-sectional area function in learning foreign languages, etc. You can practice pronunciation while checking.

  Hereinafter, processing performed by the PCA processing unit 140 will be described. In the present embodiment, vocal tract cross-sectional images are created from MRI images at slice intervals of 0.25 cm, and voices at 44 positions separated by equal section lengths along the center line of the vocal tract are obtained. Resample the road cross section. Further, the square root of the vocal tract cross-sectional area thus obtained is calculated. As a result, a 44-dimensional vector x representing each vocal tract cross-sectional area function is obtained. In order to compensate for the change in vocal tract length due to vowels, the section length is added as the 45th element of this vector x. However, this variation in section length is normalized so as to coincide with the maximum variation in vocal tract cross-sectional area.

Therefore, if the vocal tract cross-sectional area related to the vth (1 ≦ v ≦ 5) vowel at the i-th (1 ≦ i ≦ 44) section position composed of resampled elements is A _{i, v, the} vth Each element x _{i, v} of the vector x _v representing the vocal tract cross-sectional area function related to the vowels is expressed by the following equation (2).

ただし／δ（「／」は式中の上線を表わす。）及びσ_δはそれぞれ、ｖ個の母音に対応するｖ個のセクション長δ_ｖの平均及び標準偏差である。σ_Ａは断面積の平方根ｘ_ｉ,ｖ（ｉ＝１〜４４）の標準偏差の最大値を表わす。さらに、ＰＣＡ処理部１４０は、これらベクトルｘの平均値ベクトル／ｘも算出する。こうした値はいずれも記憶部１４２に記憶される。

However / [delta] ( "/" is. Represents the top line in the formula) the mean and standard deviation of the number of v Section length [delta] _v corresponding to and σ respectively _[delta], v number of vowels. σ _A represents the maximum value of the standard deviation of the square root x _{i, v} (i = 1 to 44) of the cross-sectional area. Further, the PCA processing unit 140 also calculates an average value vector / x of these vectors x. All of these values are stored in the storage unit 142.

The principal component analysis for this vector, the mean vector / x and the 45 eigenvectors e _m is obtained. Of these eigenvectors, at least (v−1) = 4 eigenvectors are uniquely determined and associated with non-zero eigenvalues.

  Examples of the first two principal components obtained in this way are shown in FIG. 2 (PCI) and FIG. 3 (PC II). Further, the third main component (PC III) is also shown in FIG.

  2 (A), 3 (A), and 4 (A) show eigenvectors, respectively. FIGS. 2 (B), 3 (B), and 4 (B) show the average vocal tract cross-sectional area according to each eigenvector. Indicates how the function is affected. 2B, FIG. 3B, and FIG. 4B, the solid line indicates the vocal tract cross-sectional area function obtained as described above. The broken line indicates the vocal tract cross-sectional area function obtained when the principal component is changed by an amount corresponding to the standard deviation in the direction of the corresponding eigenvector in the positive direction and the dotted line is in the negative direction. . From this graph, it can be seen how the vocal tract cross-sectional area function changes when each principal component is changed. 2 (A), 3 (A), and 4 (A), “●” indicates 44 elements and section lengths of positive eigenvectors, and “◯” indicates 44 eigenvectors with negative signs changed. Indicates the element value and section length, respectively.

  According to the preliminary experiments conducted by the inventors, the two components PCI and II represent 88.8% and 8.5% of the total variation, respectively. Therefore, these two components together represent 97% of the total variation in the vowel generation space of the speaker.

  PCI represents the variation between pharyngeal stenosis at the time of opening and oral stenosis with the pharyngeal cavity open. PCI also represents covariation to properly adjust the vocal tract length (for the speaker) so that it is short for vowels in the front of the mouth and long for vowels in the back.

  PC II shown in FIG. 3 represents the variation in the degree of constriction associated with the region near the upper pharynx and soft palate (7 to 13 cm from the glottis) and the lips. As the stenosis at these two locations increases, the length of the vocal tract increases accordingly. In contrast, the third principal component PC III shown in FIG. 4 represents only 1.5% of the rest of the total variation, and its state appears to contain errors and noise, It is difficult to explain in terms of articulation.

  As described above, the vocal tract area and vocal tract length represented by PCI and II obtained from the still MRI image are actually obtained from the moving image MRI as described later with respect to five vowels in Japanese. The inventors have confirmed that this data is in good agreement. By using these two principal components, it is considered possible to perform inverse estimation from speech to vocal tract cross-sectional area function as described below.

  Next, processing performed by the moving image data projection processing unit 144 will be described. In the present embodiment, the MRI moving image 114 is recorded at 30 frames per second (frame interval 33 milliseconds). Then, from the 3D image in each frame, the vocal tract cross section from the glottis to the radiating surface of the lips was sampled at intervals of 0.25 cm, and the vocal tract from the glottis to the lips was equally divided into 44 equal lengths. Resample the vocal tract cross section at the section location. However, before this, a tooth image is digitally added to the 3D image in order to consider the influence of the teeth. As a result, a vocal tract cross-sectional area function is obtained for each of the 35-frame images for continuous speech from “A” to “O”.

In order to inversely estimate the vocal tract cross-sectional area function, the moving image data projection processing unit 144 projects each of the 35 frames of the vocal tract cross-sectional area function of the MRI moving image onto the PC I-PC II space. Calculate the necessary PCI I and II weights k ₁ and k ₂ for each frame. In general, a weight k _m (m = 1 to the number of eigenvectors to be used) for each eigenvector for representing a vector x by an eigenvector corresponding to the principal component is obtained by the following equation (3).

ただしベクトルｋは重みｋ_ｍを要素とするベクトル、Ｅは固有ベクトルｅ_ｍを各列に持つ行列、「Ｅ^Ｔ」は行列Ｅの転置行列、／ｘはベクトルｘの平均値ベクトルを、それぞれ示す。

However the vector k is a vector of the weights k _m elements, E is a matrix with eigenvectors e _m in each row, "E ^T" is the transposed matrix of the matrix E, / x is an average vector of the vector x, respectively.

  In the present embodiment, the cepstrum coefficient calculation unit 148 calculates 24 cepstrum coefficients from the frequency band of 0 to 4 kHz in the speech of the speaker. This is because this combination showed the highest accuracy in reconstructing the vocal tract cross-sectional area function obtained from the moving image MRI in an experiment repeated for a subject. This combination may therefore vary from person to person. However, it has been experimentally confirmed that at least 10 cepstrum coefficients are necessary to obtain almost practical accuracy, and at least 11 are necessary to obtain more practical accuracy.

The multiple regression analysis processing unit 152 uses the values of k ₁ and k ₂ stored in the storage unit 146 and the cepstrum coefficients c _{1 to} c ₂₄ stored in the storage unit 150 to obtain the formula (1) described above. And the regression coefficients α _{m0 to} α _mN (m = 1, 2, N = 24) are obtained by solving the equation.

The linear conversion processing unit 184 of the inverse estimation processing unit 120 has the following functions. By processing the audio signal from the microphone 122 by the audio capture processing unit 180 and the cepstrum coefficient calculation unit 182, cepstrum coefficients c _{1 to} c ₂₄ of the voice of the speaker for each frame are obtained. On the other hand, the storage unit 154 stores regression coefficients α _{m0 to} α _m24 (m = 1, 2) used in the equation (1) by the parameter calculation unit 118. The linear transformation processing unit 184 uses an expression obtained by substituting the cepstrum coefficients c _{1 to} c ₂₄ and the regression coefficients α _{m0 to} α _m24 into the expression (1), and uses the weights k ₁ and the weights k ₁ for the _PCIs I and II in each frame. It has a function to calculate the k _2.

The vocal tract cross-sectional area function estimation unit 186 includes the weights k ₁ and k ₂ given for each frame from the linear transformation processing unit 184, the average vocal tract cross-sectional area function and the PCIs I and II stored in the storage unit 142, For each frame, first, a vector ^ x consisting of the square root of the vocal tract cross-sectional area is estimated by the following equation. In the present specification, the symbol “^” is placed on the character immediately after in the formula.

ただし本実施の形態では式（４）におけるＭ＝２、／ｘはＰＣＡ処理部１４０により算出されたベクトルｘの平均値、ｅ_ｍはＰＣＡ処理部１４０で算出されたｍ番目の固有ベクトルを、それぞれ示す。

However M = 2 in the formula (4) in this embodiment, / x has an average value of the vector x calculated by the PCA processor 140, _{e m} is the m-th eigenvector calculated by the PCA processor 140, respectively Show.

After estimating the vector ^ x for each frame in this way, the vocal tract cross-sectional area function estimation unit 186 simply performs the inverse transformation of the equation (1) according to the following equation (5), whereby the i th ( The estimated vocal tract cross-sectional area ^ A _{i at} the section position of 1 ≦ i ≦ 44) and the estimated section length ^ δ are calculated.

ただし＾ｘ_ｉはベクトル＾ｘのｉ番目の要素を、＾ｘ_４５はベクトル＾ｘの４５番目の要素を、σ_ＡはＰＣＡ処理部１４０により算出された断面積の平方根ｘ_ｉ,ｖ（ｉ＝１〜４４）の標準偏差の最大値を、／δ及びσ_δはそれぞれ、ｖ個の母音に対応するｖ個のセクション長δ_ｖの平均及び標準偏差を、それぞれ示す。ｉ個の推定声道断面積＾Ａ_ｉを要素とするベクトル＾Ａが、当該フレームにおける推定声道断面積関数を表わす。

However ^ _{x i} is the i th element of the vector ^ x, _{^ x 45} is a 45-th element of the vector ^ x, σ _A is the square root x _i of the cross-sectional area calculated by the PCA processor _140, v (i = 1 to 44), / δ and σ _δ indicate the average and standard deviation of _v section lengths δ _v corresponding to v vowels, respectively. A vector ^ A having _i estimated vocal tract cross-sectional area ^ A _i as an element represents an estimated vocal tract cross-sectional area function in the frame.

<Operation>
The vocal tract cross-sectional area function estimation system 100 operates as follows. The operation of the vocal tract cross-sectional area function estimation system 100 has two phases. The first phase is a learning phase in which the parameter calculation unit 118 performs parameter learning. More specifically, referring to FIG. 1, a still MRI image 110 when “a”, “i”, “c”, “e”, and “o” are spoken to a certain speaker, and MRI during continuous utterance “aueo”. From the voice data 116 measured in synchronization with the time of shooting of the moving image 114 and the MRI moving image 114, the voice of the speaker at the time of speaking “a” “b” “c” “e” “o” The average vocal tract cross-sectional area function, its first and second principal components PC I and II, and regression coefficients α _{m0 to} α _m24 (m = 1, 2) shown in Equation (1) are calculated. The second phase is an estimation phase. In the estimation phase, the average vocal tract cross-sectional area function and the PC I and II of the speaker obtained in the learning phase and the regression coefficients α _{m0 to} α _m24 are used, and the speech of the same speaker input from the microphone 122 is used. Based on this, the vocal tract cross-sectional area function is estimated in real time and displayed on the display device 124. Hereinafter, the operation of the vocal tract cross-sectional area function estimation system 100 in the first phase and the second phase will be described in order.

-Operation of Parameter Calculation Unit 118 in the Learning Phase-
In this learning phase, a still MRI image 110, an MRI moving image 114, and voice data 116 are first prepared for a speaker as follows.

  Still MRI image 110 asks a speaker to pronounce vowels “A”, “I”, “U”, “E”, and “O” longer, and captures an MRI image of the speaker during that time. The photographing of the still MRI image 110 is not different from a normal one.

  In the image of the MRI moving image 114, a speaker is put on headphones, and a continuous utterance called “Aiweo” is repeated over a certain time according to the synchronization sound generated by the synchronization sound generator 112, and during that time, the MRI video Take a picture. In the present embodiment, an MRI moving image consisting of 35 frames is captured in one iteration. Preferably, this photographing is repeated many times, and a large number of moving images obtained by the repetition are combined to obtain an image of 35 frames.

  When recording the audio data 116, it is necessary for the speaker to speak under the same conditions as those for capturing the MRI moving image 114. Therefore, the speaker is asked to take the same posture (back on the back) as when the MRI moving image 114 was taken, put on headphones, and follow the synchronization sound from the synchronization sound generating device 112 as in the recording of the MRI moving image 114. Repeat the utterance "Iueo".

  When the still MRI image 110 is prepared, the PCA processing unit 140 has 44 locations separated from each other by the same section length for each of the five types of vowels “A”, “I”, “U”, “E”, and “O”. Is calculated from the MRI image. Furthermore, the PCA processing unit 140 normalizes the variation in the section length that varies depending on the vowel so as to coincide with the maximum value of the variation in the vocal tract cross-sectional area. In this way, the PCA processing unit 140 calculates a vector x including 44 vocal tract cross-sectional areas and one normalized section length for each vowel. Each component of the vector x is expressed by equation (2).

  The PCA processing unit 140 further performs PCA processing on the obtained five vectors x, and the first and second values representing the average value vector / x of the vector x and the variation of the vector x centered on the average value vector / x. The second principal components PCI and II are calculated. The obtained average value vector / x and PCI and II are stored in the storage unit 142.

The moving image data projection processing unit 144 calculates a vocal tract cross-sectional area function from each frame of the MRI moving image 114. The calculated vocal tract cross-sectional area function of each frame is projected onto the PCI-PC II space according to Equation (3) using the average vocal tract cross-sectional area function and PCIs II stored in the storage unit 142. As a result, weights k ₁ and k ₂ for representing the vocal tract cross-sectional area function with eigenvectors corresponding to the first and second principal components are calculated in each frame. The weights k ₁ and k ₂ for each frame are stored in the storage unit 146.

The cepstrum coefficient calculation unit 148 calculates cepstrum coefficients c _{1 to} c ₂₄ for the frequency band of 0 to 4 kHz for the audio data of each frame of the audio data 116 and stores them in the storage unit 150. The multiple regression analysis processing unit 152 substitutes the weights k ₁ and k ₂ stored in the storage unit 146 and the cepstrum coefficients c _{1 to} c ₂₄ stored in the storage unit 150 into the formula (1). By solving, the regression coefficients α _m0 and α _{m1 to} α _m24 are calculated. The calculated regression coefficients α _m0 and α _{m1 to} α _m24 are stored in the storage unit 154.

  The operation of the parameter calculation unit 118 has been described above.

-Operation of the inverse estimation processing unit 120 in the estimation phase-
When a speaker speaks in front of the microphone 122, the voice is converted into an electric signal by the microphone 122 and given to the voice capture processing unit 180. The audio capture processing unit 180 digitizes the audio signal, frames it, and gives it to the cepstrum coefficient calculation unit 182. Cepstrum coefficient calculating unit 182 among the audio signals of each frame digitized given from the voice capturing process section 180 to calculate the cepstral coefficients _c 1 _{to c 24} from the frequency band of 0-4 kHz, linear transformation processing part 184 To give.

The linear transformation processing unit 184 calculates the cepstrum coefficients c _{1 to} c _{24 of} each frame given from the cepstrum coefficient calculation unit 182 and the regression coefficients α _m0 and α _{m1 to} α _m24 stored in the storage unit 154 using the equation (1). Assign to. Thereby, the weights k ₁ and k ₂ corresponding to the PCIs I and II, respectively, are calculated for each frame. The linear transformation processing unit 184 provides the calculated weights k ₁ and k ₂ to the vocal tract cross-sectional area function estimation unit 186 for each frame.

The vocal tract cross-sectional area function estimation unit 186, for each frame, the weights k ₁ and k ₂ given from the linear transformation processing unit 184, the average vocal tract cross-sectional area function / x and the PCI I stored in the storage unit 142. And the eigenvectors e ₁ and e ₂ corresponding to II and II are substituted into the equation (4) to calculate a vector ^ x consisting of the square root of the estimated vocal tract cross-sectional area. The vocal tract cross-sectional area function estimating unit 186 further includes each element of the obtained vector ^ x and constants stored in the storage unit 142 (section length average / δ and standard deviation σ _δ , cross-sectional area square root x _{i, v} (i is a section number, i = 1 to 44) standard deviation maximum value σ _A ), and using equation (5), the estimated vocal tract cross-sectional area ^ A _i (i = 1 to 44) in each frame Is estimated. As a result, a vector A representing the estimated vocal tract cross-sectional area function in each frame is obtained. The vocal tract cross-sectional area function prediction unit 186 visualizes the vocal tract cross-sectional area function in each frame in real time and causes the display device 124 to display it. An example of visualization will be described later with reference to FIG.

<Experiment>
In the above-described embodiment, the cepstrum coefficients c _{1 to} c ₂₄ are calculated for the voice in the frequency band of 0 to 4 kHz among the voice data 116 for a certain speaker. However, this value can vary from speaker to speaker. Therefore, the vocal tract cross-sectional area function was estimated under the same conditions using a multiple regression model obtained by changing the frequency band to be used and the number of cepstrum coefficients to be calculated. As a result, the same good results were obtained when any frequency band was used. However, when a frequency band of 4 kHz or less is used, generally good results are obtained as compared with the case of using a frequency band of more than that. The best results were obtained when the frequency band of 0 to 4 kHz described in the above embodiment was used. As the cepstrum coefficient, it is necessary to use at least c _{1 to} c ₁₀ in order to obtain a certain effect, and it is necessary to use at least c _{1 to} c ₁₁ in order to obtain a more reliable result. The best results were obtained when c _{1 to} c ₂₄ were used as described in the above description of the embodiment.

FIGS. 5A and 5B show the model with the most accurate prediction, with the actual PCI and II value variations (solid lines) obtained from measurements over 35 frames, Those values (marked with ●) estimated using 24 cepstrum constants defined in the frequency band of 4 kHz are shown. 5A and 5B further show values of PCI and II estimated by using 10 cepstrum coefficients in the frequency band of 0 to 3 kHz (Δ mark) for comparison. In the most good regression model, when estimating the weights _{k 1} and _{k 2} of the PC I and II, the correlation of each 0.991 and 0.968, and that mean squared error in the standard deviation 0.086 and 0.226 It was a result.

  FIG. 6 shows an example of visualizing the results obtained by the best performance regression model using cepstrum. FIGS. 6A to 6E respectively show the vocal tract cross-sectional area function (dotted line) at the time of pronunciation of each vowel obtained from the MRI moving image and the PCI obtained from the measured values of the vocal tract cross-sectional area function. And the vocal tract cross-sectional area function (thin line) expressed using the weight obtained by the regression model with the best performance according to the above embodiment, and Indicates.

  FIG. 6 shows only the frames corresponding to the vowels when “Iueo” is continuously spoken, but such a graph is obtained for each successive frame. Therefore, by switching and displaying the graph of these frames for each frame, it is possible to display changes in the vocal tract cross-sectional area function in real time simultaneously with the utterance of the speaker.

According to the above-described embodiment in which PCA and cepstrum are combined, an average error of 0.367 cm ² in the vocal tract cross-sectional area and 0.150 cm in the vocal tract length was obtained as an average for all 35 frames. . This error is only slightly larger compared to the vocal tract cross-sectional area and vocal tract length errors represented by the PCA calculated from the original MRI video. On the other hand, in the technique introduced in Non-Patent Document 4, the error of the vocal tract cross-sectional area obtained by the closed test is 0.334 cm ^2, and the model according to the present embodiment is not inferior to this. Furthermore, the error in the model according to the present embodiment is much smaller than the average error of 0.84 cm in the vocal tract length of the five Japanese vowels estimated by the linear prediction model in Non-Patent Document 6. In fact, the vocal tract length error of 0.150 cm in the case of using the model with the best performance in the present embodiment is shorter than the slice length of 0.25 cm when obtaining the vocal tract cross-sectional image with the MRI image, which is sufficient. It can be said that it is accurate.

  As described above, according to the present embodiment, an accurate vocal tract cross-sectional area function can be calculated in real time according to the voice of the speaker. In the above embodiment, it is assumed that the speaker at the time of learning and the speaker at the time of operation are the same person, but if the speaker has similar voice quality, Can also calculate the vocal tract area function from speech in real time. Furthermore, if the speech of the speaker can be normalized so as to eliminate the difference in sound quality among the speakers, it is expected that the range of applicable speakers will be further expanded.

<Realization by computer>
The system of this embodiment is realized by computer hardware, a program executed by the computer hardware, and data stored in the computer hardware. FIG. 7 shows the external appearance of the computer system 330, and FIG. 8 shows the internal configuration of the computer system 330.

  Referring to FIG. 7, the computer system 330 includes a computer 340 having an FD (flexible disk) drive 352 and a CD-ROM (compact disk read only memory) drive 350, a keyboard 346, a mouse 348, and a monitor 342. , And microphone 370 and speaker 372.

  Referring to FIG. 8, in addition to the FD drive 352 and the CD-ROM drive 350, the computer 340 includes a CPU (central processing unit) 356 and a bus 366 connected to the CPU 356, the FD drive 352, and the CD-ROM drive 350. A read only memory (ROM) 358 for storing a boot-up program and the like; a random access memory (RAM) 360 connected to the bus 366 for storing program instructions, system programs, work data, and the like; a bus 366, a microphone; 370 and a speaker 372, and includes a process of digitizing an electrical signal from the microphone 370 into a frame, and a sound board 368 for converting the sound generated by the CPU 356 into an analog signal and supplying the analog signal to the speaker 372. The computer system 330 may further include a printer (not shown).

  Although not shown here, the computer 340 may further include a network adapter board that provides a connection to a local area network (LAN).

  A computer program for causing the computer system 330 to operate as the vocal tract cross-sectional area function estimation system 100 according to the present embodiment or the parameter calculation unit 118 or the inverse estimation processing unit 120 forming a part thereof is a CD-ROM. The data is stored in the CD-ROM 362 or FD 364 inserted in the drive 350 or FD drive 352 and further transferred to the hard disk 354. Alternatively, the program may be transmitted to the computer 340 through a network (not shown) and stored in the hard disk 354. The program is loaded into the RAM 360 when executed. The program may be loaded directly into the RAM 360 from the CD-ROM 362, from the FD 364, or via a network.

  This program includes a plurality of instructions that cause the computer 340 to operate as the vocal tract cross-sectional area function estimation system 100 according to this embodiment, or the parameter calculation unit 118 or the inverse estimation processing unit 120 that forms a part thereof. Some of the basic functions required to perform this operation are provided by operating system (OS) or third party programs running on the computer 340 or various toolkit modules installed on the computer 340. Therefore, this program does not necessarily include all functions necessary for realizing the system of this embodiment. This program makes up the vocal tract cross-sectional area function estimation system 100 described above or a part thereof by calling an appropriate function or “tool” in a controlled manner to obtain a desired result. It is only necessary to include an instruction for executing an operation as the parameter calculation unit 118 or the inverse estimation processing unit 120. The operation of computer system 330 is well known and will not be repeated here.

<Deformation>
In the above embodiment, parameters for estimating the vocal tract cross-sectional area function are learned using the MRI image. However, the present invention is not limited to such an embodiment. Any method may be used as long as a 3D transmission image similar to the MRI image is obtained, and an image capable of measuring the vocal tract cross-sectional area can be obtained. In the above embodiment, both the still transmission image and the transmission moving image use MRI. But they don't have to match. You may make it acquire a still transmission image and a transmission moving image with a different method. Furthermore, in the above-described embodiment, the MRI moving image 114 is captured and the audio data 116 is recorded separately, but synchronized using the synchronous sound generator 112. This is due to the restriction that sound data cannot be recorded in a good state because a large noise is generated in MRI imaging. Therefore, if a transmission moving image can be taken in a quiet environment by a method other than MRI, the audio data 116 may be recorded at the same time. Even when MRI is used, if the audio data 116 can be recorded satisfactorily, imaging and audio recording may be performed simultaneously.

  In the above embodiment, the cepstrum analysis is performed on the voice data, and the weight of the principal component is approximated by the linear sum of the first to 24th coefficients among the obtained cepstrum coefficients. This is because good results were obtained using cepstrum coefficients up to the 24th order in the experiment. However, the optimum order may vary somewhat depending on the speaker, and it is needless to say that the order of the cepstrum coefficient to be used may be slightly increased or decreased even for the same person.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a block diagram which shows the functional structure of the vocal tract cross-sectional area function estimation system 100 which concerns on one embodiment of this invention. It is a figure which shows the value for every flame | frame of the 1st main component regarding a vocal tract cross-sectional area function, and a corresponding vocal tract cross-sectional area function. It is a figure which shows the value for every frame of the 2nd main component regarding a vocal tract cross-sectional area function, and a corresponding vocal tract cross-sectional area function. It is a figure which shows the value for every frame of the 3rd main component regarding a vocal tract cross-sectional area function, and a corresponding vocal tract cross-sectional area function. It is a figure which shows the value for every flame | frame of the 1st and 2nd principal component obtained from the MRI moving image, and two types of estimated 1st and 2nd principal components. The figure which shows the vocal tract cross-sectional area function estimated from the continuous speech of a speaker, the vocal tract cross-sectional area function obtained from the MRI moving image used at the time of learning, and the expression by the 1st and 2nd main component. It is. 1 is an external view of a computer system that implements a vocal tract cross-sectional area function estimation system 100 according to an embodiment of the present invention, or a parameter calculation unit 118 or an inverse estimation processing unit 120 that forms a part thereof. FIG. 8 is a block diagram of the computer system shown in FIG. 7.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Vocal tract cross-sectional area function estimation system 110 Still MRI image 112 Synchronous sound generator 114 MRI moving image 116 Audio | voice data 118 Parameter calculation part 120 Inverse estimation process part 140 PCA process part 142,146,150,154 Memory | storage part 144 Moving image data Projection processing unit 148 Cepstrum coefficient calculation unit 152 Multiple regression analysis processing unit 180 Voice capture processing unit 182 Cepstrum coefficient calculation unit 184 Linear transformation processing unit 186 Vocal tract cross-sectional area function estimation unit

Claims (4)

For each of a plurality of vowels, a static transmission image at the time of individual utterance of the predetermined speaker, a transmission video image of the predetermined speaker at the time of continuous utterance of the plurality of vowels, and at the time of the continuous utterance 1 to m-th (m is 2 or more) representing the average of the speaker's vocal tract cross-sectional area function at the time of each individual utterance and the change around the average Natural number) and the weights of the first to m-th principal components for approximating the vocal tract cross-sectional area function of the speaker during the continuous utterance with the average and the first to m-th principal components. Is prepared by a linear sum of cepstrum coefficients of the first order to the nth order (n is a predetermined natural number of 10 or more) obtained by cepstrum analysis on the voice data of the predetermined speaker. Parameter preparation means for
A first cepstrum coefficient is calculated for each frame of a framed audio signal obtained by capturing a speaker's voice, and the first to nth cepstrum coefficients are calculated for each frame. Cepstrum analysis means,
The voice signal of the speaker is obtained by using the linear sum using the coefficients prepared by the parameter preparation means for the first to n-th order cepstrum coefficients calculated by the first cepstrum analysis means. A linear transformation unit for calculating weights for the first to m-th principal components for each frame of
Using the weights for the first to m-th principal components calculated by the linear conversion unit and the average and the first to m-th principal components prepared by the parameter preparation means, An apparatus for estimating a vocal tract cross-sectional area function, comprising: a vocal tract cross-sectional area estimating means for estimating a vocal tract cross-sectional area for each frame of the speech signal of the person. The parameter preparation means includes
For each of the plurality of vowels, a vocal tract cross-sectional area related to the plurality of vowels based on a vocal tract cross-sectional area function obtained from a static transmission image of the vocal tract portion at the time of individual speech of the predetermined speaker. A principal component analysis means for obtaining an average of a function and the first to m-th principal components by principal component analysis;
By projecting the vocal tract cross-sectional area function during continuous utterance obtained for each frame from the transmitted moving image onto the space formed by the first to m-th principal components, the vocal tract cross-sectional area during continuous utterance is projected. Weight calculating means for calculating a weight of the first to m-th principal components for representing a function by the first to m-th principal components and the average;
Second cepstrum analysis means for obtaining the first to nth cepstrum coefficients by performing the cepstrum analysis on the voice data of the predetermined speaker during the continuous utterance;
Multiple regression analysis for approximating the weights of the first to m-th principal components at the time of the continuous utterance with a linear sum of the first-order to n-th order cepstrum coefficients obtained by the second cepstrum analysis means. Multiple regression analysis means for calculating the coefficient by performing
The voice according to claim 1, comprising storage means for storing the coefficients for approximating the weights of the average, the first and second principal components, and the first to m-th principal components. Road cross section function estimation device. 2. The vocal tract cross-sectional area function according to claim 1, wherein the parameter preparation means includes storage means for storing an average of the vocal tract cross-sectional area function, the first to m-th principal components, and the coefficient. Estimating device. A computer program that, when executed by a computer, causes the computer to operate as the vocal tract cross-sectional area function estimation device according to any one of claims 1 to 3.

Images