JP4536464B2 - Speech synthesis apparatus and method - Google Patents

Description

  The present invention generates stimulation speech based on a probabilistic dynamic model of articulatory motion trajectories at at least some positions of a plurality of articulators, and includes the position of articulators, the change in the positions, and the amount of change in formant frequency. The present invention relates to a speech synthesizer and a method for outputting the relationship as the stimulus speech.

There are the following methods for creating stimulus speech used in an auditory psychophysical experiment for measuring a discrimination threshold of formant frequencies of syllables including vowels or consonants.
First, a voice signal uttered by a single vowel is recorded, and three low-order formant frequencies, their bandwidth and fundamental frequency are measured from the voice signal. Next, a stimulus sound as a reference in which the value is set as a parameter of the Klatt formant synthesized sound and a stimulus sound in which the value of the formant frequency to be measured for the threshold is changed in either the upper or lower direction are created. Then, the discrimination threshold of the formant frequency is measured using a psychophysical measurement method such as a constant method or a modified up-and-down method (for example, see Non-Patent Document 1).

However, according to the technique disclosed in Non-Patent Document 1 described above, the measured formant frequency, its bandwidth, and fundamental frequency do not explicitly take into account the constraints of human vocal organs and vocalization actions.
For this reason, the stimulus sound created by changing the formant frequency value subject to threshold measurement up and down with a certain width does not necessarily exist in a range where humans can actually speak. Therefore, there is a possibility that a stimulus that cannot be heard is used in the experiment. Moreover, in the measurement of a single syllable utterance voice signal including a single vowel or consonant, it is difficult to create a stimulus voice that takes into account the effects of the front and back phonemic environment such as articulation coupling.

On the other hand, there is a method based on articulation / acoustic versus codebook search as a method of generating an audio signal from an utterance operation (see, for example, Patent Document 1 and Non-Patent Document 2). In addition, as a method for generating a dynamic model of speech operation and generating a speech signal thereof, there is a method based on an HMM (Hidden Markov Model) speech generation model and articulation / acoustic mapping (for example, Patent Document 2, (See Patent Document 3 and Non-Patent Document 3).
Japanese Patent No. 3421798 JP 2003-241776 A JP 2003-271186 A D. Kewley-Port and CS Watson. “For mant-frequency discrimination for isolated English vowels,” J. Acoust. Soc. Am, vol. 95, no. 1, 1994). Kashiwagi, Honda, Tsumura, “Speech synthesis from articulatory motion based on search of articulatory / acoustic pair codebook with phoneme label,” vol.54, no.3, Journal of the Acoustical Society of Japan, 1998) Hiroya, S. Honda. M. “Estimation of articulatorymovements from speech acoustics using an HMM-based speech production model,” IEEE Trans.Speech and Audio Processing, vol.12, no.2, 2004)

By the way, when a certain articulation position is given, the calculation of the amount of change in formant frequency with respect to the change in the articulation position does not take into account the restriction of the speech operation, and the position of each articulator organ in all possible directions. It was done by changing. Therefore, when investigating the influence of the change in formant frequency, a qualitative result can be obtained. Therefore, it is difficult to use the relationship between the change in articulation position and the amount of change in formant frequency as the stimulus sound.
In addition, since there is a correlation between the actual articulators, “moving the position of the articulator in all possible directions” performed in the previous study fully considers the dynamic behavior of the articulatory motion. Means not.

  The present invention has been made in view of the above circumstances, and since human speech is performed based on static and dynamic restrictions of the articulatory organ, the relationship between the change in articulation position and the amount of change in formant frequency. It is an object of the present invention to provide a speech synthesizer and a method thereof that make it possible to use as a stimulating speech created based on the restriction of articulatory motion.

The present invention to solve the above-mentioned problems, there is provided a speech synthesizer to force out by generating a stimulus sound, using a stochastic dynamic model of articulatory trajectory learning state series articulatory parameter vector It means for, means for interpolation calculation between the average articulation position of each state in the stochastic dynamic model of the articulatory movement trajectories, for each articulation position obtained by the interpolation operation, the speech parameter vector articulatory A means for calculating a formant frequency, a bandwidth, and a fundamental frequency with reference to an articulatory acoustic pair code book in which a pair with a parameter vector is stored , each articulation position obtained by the interpolation operation, and the calculated formant frequency against, the amount of change of the formant frequency for a change in the articulation position, the respective articulation positions obtained by the interpolation operation direction of movement In the mean square distance of the articulatory parameter vector due to the change in 10 points, by dividing the linear regression coefficient is calculated based on the values of the formant frequencies calculated on the basis of the articulation position of the 10-point, means for calculating , Means for generating and storing a correspondence table in which the formant frequency, the bandwidth, the fundamental frequency, and the amount of change of the formant frequency for each of the articulation positions are paired, and the formant inputted with reference to the correspondence table Select the data set that is the amount of change of the formant frequency, the bandwidth, the fundamental frequency, and the formant frequency corresponding to the frequency, bandwidth, and fundamental frequency by performing a full search, and select the selected Means for generating the stimulation sound based on a data set .

  Further, in the present invention, the means for performing the interpolation calculation generates a probabilistic dynamic model of the articulatory movement locus using a hidden Markov model.

  Further, in the present invention, the means for performing the interpolation calculation selects the motion direction at each articulation position based on the state transition of the hidden Markov model.

Further, the present invention provides a speech synthesis method for force out by generating by a stimulus sound arithmetic unit, said arithmetic unit, using a probabilistic dynamic model of the articulatory movement trajectory, articulatory parameter vector Learning step sequence, interpolating between average articulation positions of each state in the probabilistic dynamic model of the articulatory movement locus, and speech parameter vector for each interpolated articulation position Referring to the said articulatory parameter vector and pairs stored articulatory sound pair codebook, calculating a formant frequency and bandwidth and the fundamental frequency, which is the calculated respective articulation position obtained by the interpolation calculation relative formant frequencies, the amount of change of the formant frequency for the change in place of articulation, the articulation position obtained by the interpolation calculation There the average square distance of the articulatory parameter vector due to the change 10 in the direction of movement, by dividing the linear regression coefficient is calculated based on the value of the calculated formant frequencies on the basis of the articulation position of the 10 points, A step of calculating, a step of creating and storing a correspondence table in which formant frequency, bandwidth, fundamental frequency, and amount of change of formant frequency with respect to each articulation position are paired; and the formant input with reference to the correspondence table Select the data set that is the amount of change of the formant frequency, the bandwidth, the fundamental frequency, and the formant frequency corresponding to the frequency, bandwidth, and fundamental frequency by performing a full search, and select the selected based on the data set, it is characterized by executing the steps of: generating the stimulus voice .

According to the present invention, stimulating speech is created in consideration of a probabilistic dynamic model of articulatory movement trajectories of articulators such as jaw, tongue, lips, soft palate, and larynx during continuous speech utterance, Interpolate between the average articulation positions of motion, determine the formant frequency, its bandwidth, and fundamental frequency from each interpolated articulation position, and calculate the amount of change in formant frequency for that articulation position for each articulation position Thus, the relationship between the change in the position of the articulator and the amount of change in the formant frequency can be used as the stimulus sound. As a result, it is possible to realize precise speech synthesis that takes into account the dynamic behavior of human articulators and movements.
Furthermore, a probabilistic dynamic model of articulatory motion trajectory is generated using HMM (Hidden Markov Model), and the connection between words is selected by selecting the motion direction at each articulation position based on the state transition of the hidden Markov model. Relationships can be expressed indirectly, enabling more sophisticated speech synthesis.

FIG. 1 is a block diagram showing an expanded function of the internal configuration of the speech synthesizer according to the present invention.
As shown in FIG. 1, the speech synthesizer of the present invention includes a speech parameter storage unit 1, an articulation parameter storage unit 2, a phoneme sequence storage unit 3, an articulation / acoustic codebook generation unit 4, and an HMM model creation. Unit 5, interpolated articulation position generation unit 6, formant frequency generation unit 7, bandwidth generation unit 8, fundamental frequency generation unit 9, AFS (Articulatory Formant Sensitivity) generation unit 10, and correspondence table generation unit 11 The parameter selection unit 12 and the voice generation unit 13 are configured.

The articulation / acoustic codebook generation unit 4 generates a pair of an audio parameter vector stored in the audio parameter storage unit 1 and an articulation parameter vector stored in the articulation parameter storage unit 2, and forms them into a formant frequency generation unit 7. , To the bandwidth generator 8 and the fundamental frequency generator 9.
The HMM model creation unit 5 has a function of generating stimulation speech using a stochastic dynamic model of articulatory movement trajectory in a human articulator such as jaw, tongue, lips, soft palate, and larynx. An HMM (Hidden Markov Model) is used as a probabilistic dynamic model of the motion trajectory. In addition to the articulation parameter vector stored in the articulation parameter storage unit 2, the HMM model creation unit 5 is also supplied with the uttered phoneme sequence stored in the phoneme sequence storage unit 3. Details will be described later.

The interpolated articulation position generation unit 6 has a function of performing interpolation calculation between the average articulation positions of the respective states in the HMM described above, and selects the motion direction at each articulation position based on the state transition of the HMM and interpolates each of them. Perform the operation. Each articulation position data obtained by this interpolation calculation is supplied to each of the formant frequency generation unit 7, the bandwidth generation unit 8, and the fundamental frequency generation unit 9.
Each of the formant frequency generation unit 7, the bandwidth generation unit 8, and the fundamental frequency generation unit 9 sets a pair of a speech parameter vector and an articulation parameter vector for each articulation position obtained by the interpolation operation of the interpolation articulation position generation unit 6. The formant frequency, the bandwidth, and the fundamental frequency are calculated with reference to the articulation / acoustic codebook generation unit 4 in which is stored. The logic for this calculation is disclosed in detail in Non-Patent Document 2 described above (examination of speech synthesis method from articulation motion based on search of phoneme-labeled articulation / acoustic pair code book).

On the other hand, the AFS generator 10 has a function of calculating the amount of change (AFS) of the formant frequency with respect to the change of the articulation position for each articulation position and the formant frequency that is the output of the formant frequency generator 7. The processed AFS is supplied to the correspondence table generation unit 11. Besides, the correspondence table generation unit 11 is supplied with formant frequency, bandwidth, and fundamental frequency from the formant frequency generation unit 7, the bandwidth generation unit 8, and the fundamental frequency generation unit 9, respectively. A pair is generated and stored in a storage device (not shown).
Further, the parameter selection unit 12 performs a full search on the generated and stored correspondence table by inputting, for example, parameters of the formant frequency, the bandwidth, and the fundamental frequency selected by the user. It has a function of finding a set and supplying it to a sound generation unit 13 composed of a formant synthesizer as stimulus sound. The voice generation unit 13 outputs a synthesized voice with the data set obtained from the correspondence table as an input.

  1, the speech parameter storage unit 1, the articulation parameter storage unit 2, the phoneme sequence storage unit 3, and the codebook generated by the articulation / acoustic codebook generation unit 4 and the correspondence table generation unit 11. The correspondence table is assumed to be allocated and stored in a predetermined area of a storage device (not shown), and also includes an HMM model creation unit 5, an interpolation articulation position generation unit 6, a formant frequency generation unit 7, a bandwidth generation unit 8, Each of the fundamental frequency generation unit 9, the AFS (Articulatory Formant Sensitivity) generation unit 10, the correspondence table generation unit 11, the parameter selection unit 12, and the voice generation unit 13 sequentially reads out the program by the arithmetic device and its peripheral LSI constituting the computer. It is assumed that the functions possessed by each are realized.

FIG. 2 is a flowchart cited for explaining the operation of the speech synthesizer according to the present invention.
The operation of the embodiment of the present invention shown in FIG. 1 will be described in detail below with reference to the flowchart shown in FIG.

  Here, the formant frequency, its bandwidth, and the fundamental frequency are used as parameters, and the stationary vowel created by the sound generation unit 13 (formant synthesizer) is used as the stimulation sound. These parameters are obtained from an articulatory / acoustic codebook generated based on actual observation data of human vocalization.

Specifically, the articulation / acoustic pair codebook is created by the articulation / acoustic codebook generation unit 4 based on the articulation motion observed simultaneously using the audio signal and the magnetic sensor system.
Here, one speaker was used, and a vowel utterance of three vowel chains such as / aui / was used. The voice signal is cut out at a rate of 250 times per second at a window length of 25 ms, and the four low-order formant frequencies, the bandwidth, and the fundamental frequency are obtained as a voice parameter y (voice parameter storage unit 1). As articulation parameters, signals are measured at a rate of 250 times per second, and vectors using horizontal and vertical signals of 9 points in total, including the lower jaw, upper and lower lips, 4 points on the tongue, soft palate and larynx as positions. It is assumed that the articulation parameter vector x (articulation parameter storage unit 2) (S21, S22). The code book is also given a phoneme sequence (phoneme sequence storage unit 3) uttered.

The HMM model creation unit 5 statistically learns the dynamic behavior of the articulation parameters using the articulation parameter vector x obtained as described above. Here, HMM is used as a statistical method (S23: Creation of HMM based on articulatory motion). The structure of the HMM is a left-to-right model without skipping, which is a two-phoneme set three-state single-mixed Gaussian distribution considering the subsequent phonemes.
The HMM model λ is created so that the output probability P (x | λ) = Σq P (x | q, λ) P (q | λ) of the articulation parameter vector is maximized. Here, q is an HMM state sequence, and the output probability P (x | q, λ) of the articulation parameter vector for a given state sequence assumes a Gaussian distribution.

Next, the interpolated articulation position generation unit 6 is linear at 100 points between the average articulation positions xi, 0 and xj, 0 of each state i, j in the model of each two-phoneme set of the HMM model λ obtained as described above. An interpolated articulation position series is created (S24). Here, it is assumed that xi, j, n (i, j: state, n = 1 to 100).
This means that the movement direction at a certain articulation position is defined based on the state transition of the HMM. FIG. 4 shows an example of the movement direction of the articulation position on the median cross section determined by the HMM.

  Then, each of the formant frequency generation unit 7, the bandwidth generation unit 8, and the fundamental frequency generation unit 9 outputs the articulation / synchronization position xi, j, n output by the interpolation articulation position generation unit 6 to the articulation / Based on the articulatory acoustic pair codebook generated by the acoustic codebook generation unit 4, each of the formant frequency, the bandwidth, and the fundamental frequency yi, j, n is calculated (S25).

  FIG. 3 conceptually shows a procedure for interpolating between the average articulation positions in each state in the dynamic model of articulatory movement described above, and a procedure for determining a formant frequency, its bandwidth and fundamental frequency from the interpolated articulation positions. It is shown.

Subsequently, the AFS generation unit 10 calculates a change amount of the formant frequency with respect to the change of the articulation gesture which is the position of the articulatory organ, using the interpolated articulation position and formant frequency (S26). The calculation method is based on the linear regression coefficient determined by using the formant frequency values at the 10 points before and after the first or second formant frequency value at all the articulation positions, and by changing the articulation position by 10 points. Divide by the mean square distance for each parameter such as jaw and tongue, that is, the amount of change in articulation gesture. This is defined as the amount of change in formant frequency (AFS) with respect to the change in articulation gesture. An example of the amount of change in formant frequency with respect to a change in articulation gesture is shown in FIG.
Next, the correspondence table generating unit 11 performs calculation for all the articulation positions, and creates and stores a correspondence table in which formant frequency, bandwidth, fundamental frequency, and AFS for each articulation position are paired (S27).

Here, the parameter selection unit 12 refers to the correspondence table created by the correspondence table generation unit 11, and presents the stimulation sound in consideration of the restriction of the speech operation.
For example, when it is desired to present stimulus sounds having the same formant frequency and the same bandwidth but different fundamental frequency values, a corresponding data set may be found by a full search from the correspondence table and used as the stimulus sounds. Furthermore, it is possible to check whether or not the stimulating sound when the formant frequency is changed can be actually uttered by a person based on the restriction of the articulation position.
FIG. 6 shows an example of a data set of articulation positions on the mid-section that generates the same formant frequency and the same bandwidth, and generates a high fundamental frequency (127 Hz) and a low fundamental frequency (114 Hz).

  Finally, the voice generation unit 13 creates a stimulus voice from the formant synthesizer using the formant frequency, the bandwidth, and the fundamental frequency selected by the parameter selection unit 12 as parameters (S28, S29).

Here, a dynamic model of articulatory movement was created using three-vowel chain 540 sentences such as / aiu uttered by one Japanese male, and formant sensitivity to articulation changes was learned. A PRAAT was used for the formant synthesizer. Furthermore, the time length of the stimulation sound was 200 ms, and the sound power was naturally changed with time.
It has been found that there is a correlation between the formant frequency discrimination threshold and the amount of change (AFS) of the formant frequency with respect to the articulation gesture by using the stimulating speech that takes into account the dynamic behavior of the speech organ according to the present invention. In addition, when stimulating speech that does not exist within the range that humans can actually utter is used, the above correlation is not observed, so stimulating speech that does not take into account the limitations of conventional speech is used for stimulating psychoacoustic experiments. As insufficient. FIG. 7 shows the relationship between the amount of change in formant frequency (AFS) with respect to the change in articulation gesture and the discrimination threshold of formant frequency according to the present invention.

  As described above, according to the present invention, the stimulating sound is a probabilistic dynamic model of the articulatory movement trajectory of the articulator such as the jaw, tongue, lips, soft palate, and larynx during continuous voice utterance (here, the HMM). And the average articulation position of each motion is interpolated, and the formant frequency, its bandwidth, and the fundamental frequency are determined from each interpolated articulation position, and for each articulation position Thus, by calculating the change amount of the formant frequency with respect to the articulation position, the relationship between the change of the articulation gesture and the change amount of the formant frequency can be used as the stimulation sound. As a result, it is possible to realize precise speech synthesis that takes into account the dynamic behavior of human articulators and movements.

It is the block diagram which expanded the function and showed the internal structure of the speech synthesizer in this invention. It is the flowchart quoted in order to demonstrate operation | movement of the speech synthesizer in this invention. It is the operation | movement conceptual diagram quoted in order to demonstrate operation | movement of this invention embodiment. It is a graph which shows the experimental result by this invention embodiment. It is a graph which shows the experimental result by this invention embodiment. It is a graph which shows the experimental result by this invention embodiment. It is a graph which shows the experimental result by this invention embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Speech parameter memory | storage part, 2 ... Articulation parameter memory | storage part, 3 ... Phoneme series memory | storage part, 4 ... Articulation / acoustic codebook production | generation part, 5 ... HMM model creation part, 6 ... Interpolation articulation position production | generation part, 7 ... Formant frequency Generation unit, 8 ... Bandwidth generation unit, 9 ... Fundamental frequency generation unit, 10 ... AFS generation unit, 11 ... Correspondence table generation unit, 12 ... Parameter selection unit, 13 ... Audio generation unit

Claims (4)

A speech synthesis apparatus which forces out to generate a stimulation speech,
Means for learning a state sequence of an articulatory parameter vector using a probabilistic dynamic model of an articulatory motion locus;
Means for interpolating between the average articulation positions of each state in the probabilistic dynamic model of the articulatory movement locus;
For each articulation position obtained by the interpolation calculation, with reference to the speech parameter vector and the articulatory parameter vector and articulatory sound pair codebook pairs are stored in, means for calculating the formant frequency and bandwidth and the fundamental frequency When,
For each articulation position obtained by the interpolation calculation and the calculated formant frequency, the amount of change of the formant frequency with respect to the change of the articulation position is 10 points in the movement direction for each articulation position obtained by the interpolation calculation. Means for calculating by dividing the linear regression coefficient calculated based on the value of the formant frequency calculated based on the 10 articulation positions by the mean square distance of the articulation parameter vector due to the change ;
Means for generating and storing a correspondence table that pairs the formant frequency, the bandwidth, the fundamental frequency, and the amount of change of the formant frequency with respect to each articulation position;
Referring to the correspondence table, the formant frequency, the bandwidth, the fundamental frequency and the formant frequency corresponding to the input formant frequency, the bandwidth, the bandwidth, the fundamental frequency, and the amount of change of the formant frequency Means for selecting a set by full search and generating the stimulation sound based on the selected data set ;
A speech synthesizer characterized by comprising: The means for performing the interpolation calculation includes:
The speech synthesizer according to claim 1, wherein the probabilistic dynamic model of the articulatory motion locus is generated using a hidden Markov model. The means for performing the interpolation calculation includes:
The speech synthesizer according to claim 1 or 2, wherein the movement direction at each articulation position is selected based on the state transition of the hidden Markov model. A speech synthesis method for force out by generating by a stimulus sound arithmetic device,
The arithmetic unit is:
Learning a state sequence of an articulatory parameter vector using a probabilistic dynamic model of an articulatory motion trajectory;
Interpolating between the average articulation positions of each state in the probabilistic dynamic model of the articulatory motion trajectory;
For each articulation position as the interpolation calculation, calculating a voice parameter vector and the articulatory parameter pairs with the vector refers to the stored articulatory sound pair codebook, formant frequency and bandwidth and the fundamental frequency,
For each articulation position obtained by the interpolation calculation and the calculated formant frequency, the amount of change of the formant frequency with respect to the change of the articulation position is 10 points in the movement direction for each articulation position obtained by the interpolation calculation. Dividing the linear regression coefficient calculated based on the value of the formant frequency calculated based on the 10 articulation positions by the mean square distance of the articulation parameter vector due to the change ; and
Creating and storing a correspondence table paired with the amount of change in formant frequency, bandwidth, fundamental frequency, and formant frequency for each articulation position;
Referring to the correspondence table, the formant frequency, the bandwidth, the fundamental frequency and the formant frequency corresponding to the input formant frequency, the bandwidth, the bandwidth, the fundamental frequency, and the amount of change of the formant frequency Selecting a set by full search and generating the stimulus sound based on the selected data set ;
A speech synthesis method characterized by executing

Images