JP7709646B2 - Speech synthesis device, speech synthesis method, and program - Google Patents

Description

The present disclosure relates to a voice synthesis device, a voice synthesis method, and a program.

In speech synthesis, a module that converts acoustic features such as spectrum and pitch into a speech waveform is called a vocoder. There are two main ways to implement a vocoder. The first is a signal processing method, and well-known methods include STRAIGHT (Non-Patent Document 1) and WORLD (Non-Patent Document 2). These methods use a mathematical model to represent the conversion from acoustic features to a speech waveform, so they do not require learning and are fast in processing speed, but the quality of the analyzed and resynthesized speech is inferior to natural speech. The second is a neural network method (neural vocoder), and WaveNet is a representative method (Patent Document 1). This method can synthesize speech of a quality comparable to natural speech, but it is based on a huge convolutional neural network (CNN: Convolutional Neural Network), so it requires a lot of calculations, operates slower than a signal processing vocoder, and is difficult to operate in real time.

Therefore, in order to operate in real time on a CPU, it is necessary to reduce the amount of calculations. One of the main approaches is WaveRNN, which replaces the huge CNN used in WaveNet with a small recurrent neural network (RNN) (Patent Document 2). Also, LPCNet (Non-Patent Document 3) introduces linear predictive analysis (LPC), a knowledge of signal processing, into the process of generating speech waveforms, making it possible to synthesize speech using a deep neural network (DNN) that is even smaller than WaveRNN. In this way, WaveRNN and LPCNet use RNNs to realize small-scale speech synthesis DNNs.

Hideki Kawahara, Ikuyo Masuda-Katsuuse and Alain de Cheveigne, "Restructuring speech representations using a pitch-adaptive time frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Communication, vol. 27, no. 3-4, pp. 187-207, 1999. Masanori Morise, Fumiya Yokomori, Kenji Ozawa, "WORLD: a vocoder-based high-quality speech synthesis system for real-time applications," IEICE transactions on information and systems, vol. E99-D, no. 7, pp. 1877-1884, 2016. Jean-Marc Valin and Jan Skoglund, "LPCNET: Improving Neural Speech Synthesis through Linear Prediction," Proc. ICASSP, 2019, pp. 5891-5895

WO-A-2018/048934 WO-A-2019/155054

However, in autoregressive models such as RNNs, the predicted speech waveform value is used as the speech waveform value for the next time, so the error with the learning phase increases as the sequence to be predicted becomes longer. This can eventually cause waveform generation to fail, leading to unclear speech and, in the worst case, silence. In addition, while it is possible to avoid failure by initializing the state variables of the RNN at a certain timing, the time series information up to that time is initialized, resulting in discontinuity, and especially initialization during a speech section can lead to a decrease in the naturalness of the speech.

In addition, in order to detect any breakdowns in the waveform generation, it is possible to run waveform generation using signal processing, which is lower quality but does not cause breakdowns, together with the neural vocoder and compare the results; however, since the vocoder must be run using two different methods, this would significantly slow down the operating speed of the waveform generation.

The present invention has been made in consideration of the above points, and aims to prevent a decrease in the naturalness of voice while preventing a significant loss in the operating speed of waveform generation.

In order to solve the above problem, the invention of claim 1 is a speech synthesis device that generates a speech waveform in a learning phase, and includes: a waveform generation unit that obtains a predicted value of the speech waveform at the next time based on the combined speech waveform and acoustic features, and the state of a recurrent neural network; a failure flag conversion unit that obtains the failure flag based on the speech waveform, the predicted value of the speech waveform, and a failure flag threshold indicating whether the speech waveform at each time is corrupted; a failure detection unit that obtains a predicted value of the failure flag based on the state sequence of the recurrent neural network and a failure detection model; a failure flag error calculation unit that calculates an error between the failure flag and the predicted value of the failure flag; and a failure detection model learning unit that obtains a learned failure detection model based on the error and the failure detection model.

As described above, the present invention has the effect of preventing a decrease in the naturalness of the voice while preventing a significant loss in the operating speed of waveform generation.

1 is a schematic diagram of a communication system according to an embodiment of the present invention. 1 is a diagram illustrating a hardware configuration of a voice synthesizer and a communication terminal according to an embodiment of the present invention. FIG. 2 is a functional configuration diagram of the speech synthesis device according to the first embodiment in a learning phase. FIG. 2 is a functional configuration diagram of the speech synthesis device according to the first embodiment in an inference phase. 4 is a flowchart showing processing or operations in a learning phase of the voice synthesis device according to the first embodiment. 4 is a flowchart showing a process or operation in an inference phase of the speech synthesis device according to the first embodiment. FIG. 11 is a functional configuration diagram of a speech synthesis device according to a second embodiment in an inference phase. 13 is a flowchart showing a process or operation in an inference phase of the speech synthesis device according to the second embodiment. FIG. 13 is a functional configuration diagram of a speech synthesis device according to a third embodiment in a learning phase. FIG. 13 is a functional configuration diagram of a speech synthesis device according to a third embodiment in an inference phase. 13 is a flowchart showing a process or operation in a learning phase of the voice synthesis device according to the third embodiment. 13 is a flowchart showing a process or operation in an inference phase of the speech synthesis device according to the third embodiment. FIG. 13 is a functional configuration diagram of a speech synthesis device according to a fourth embodiment in a learning phase. FIG. 13 is a functional configuration diagram of a speech synthesis device according to a fourth embodiment in an inference phase. 13 is a flowchart showing a process or operation in a learning phase of the voice synthesis device according to the fourth embodiment. 13 is a flowchart showing a process or operation in an inference phase of a voice synthesis device according to a fourth embodiment.

Below, an embodiment of the present invention is described based on the drawings.

[System configuration of the embodiment]
First, an outline of the configuration of a communication system 1 according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of the communication system according to the present embodiment.

As shown in FIG. 1, the communication system 1 of this embodiment is constructed by a voice synthesis device 3 and a communication terminal 5. The communication terminal 5 is managed and used by user Y.

In addition, the voice synthesis device 3 and the communication terminal 5 can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

The voice synthesis device 3 is composed of one or more computers. When the voice synthesis device 3 is composed of multiple computers, it may be referred to as a "voice synthesis device" or a "voice synthesis system."

The voice synthesis device 3 is a computer that uses speech impairment detection technology to generate voice waveforms for voice synthesis.

The communication terminal 5 is a computer, and in FIG. 1, a notebook computer is shown as an example, but it is not limited to a notebook computer and may be a desktop computer. The communication terminal may also be a smartphone or a tablet terminal. In FIG. 1, user Y is operating the communication terminal 5.

[Hardware configuration of voice synthesizer and communication terminal]
Next, the hardware configuration of the voice synthesizer 3 and the communication terminal 5 will be described with reference to Fig. 2. Fig. 2 is a hardware configuration diagram of the voice synthesizer and the communication terminal according to this embodiment. Note that the hardware configuration of the voice synthesizer and the communication terminal is common to the first to fourth embodiments described later.

2, the speech synthesis device 3 has a processor 301, a memory 302, an auxiliary storage device 303, a connection device 304, a communication device 305, and a drive device 306. The hardware components constituting the speech synthesis device 3 are connected to each other via a bus 307.

The processor 301 serves as a control unit that controls the entire speech synthesis device 3, and has various computing devices such as a CPU (Central Processing Unit). The processor 301 reads various programs onto the memory 302 and executes them. The processor 301 may also include a GPGPU (General-purpose computing on graphics processing units).

The memory 302 has a primary storage device such as a read only memory (ROM) or a random access memory (RAM). The processor 301 executes various programs read onto the memory 302 to realize various functional units described below.

The auxiliary memory device 303 stores various programs and various information (such as the failure detection model 30a and the learned failure detection model 30b described below) used when the various programs are executed by the processor 301.

The connection device 304 is a connection device that connects an external device (e.g., a display device 310, an operating device 311) to the speech synthesis device 3.

The communication device 305 is a communication device for sending and receiving various information between other devices.

The drive unit 306 is a device for setting the recording medium 330. The recording medium 330 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, or a magneto-optical disk. The recording medium 330 may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory) or a flash memory.

The various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 330 in the drive device 306 and reading the various programs recorded on the recording medium 330 by the drive device 306. Alternatively, the various programs to be installed in the auxiliary storage device 303 may be installed by downloading them from a network via the communication device 305.

Figure 2 also shows the hardware configuration of communication terminal 5, but since the configuration is the same except that the reference numbers have changed from the 300s to the 500s, the explanation of these will be omitted.

First Embodiment A first embodiment will be described with reference to FIGS.

[Functional configuration of the voice synthesis device]
The functional configuration of the speech synthesis device according to the first embodiment will be described with reference to FIG. 3 and FIG.

<Functional configuration of the speech synthesis device in the learning phase>
Fig. 3 is a functional configuration diagram of the speech synthesizer in the learning phase according to the first embodiment. As shown in Fig. 3, the speech synthesizer 3 has an input unit 31, a waveform generation unit 32, a conversion to a failure flag unit 33, a failure detection unit 34, a failure flag error calculation unit 35, and a failure detection model learning unit 36.

Of these, the input unit 31 combines the input speech waveform and acoustic features.

The waveform generation unit 32 obtains a predicted value of the audio waveform at the next time based on the combined audio waveform and acoustic features, as well as the state of the recurrent neural network.

The failure flag conversion unit 33 obtains a failure flag indicating whether the audio waveform at each time is failure based on the audio waveform, the predicted value of the audio waveform, and the failure flag threshold value.

The failure detection unit 34 obtains a predicted value of the failure flag based on the state sequence of the recurrent neural network (RNN) and the failure detection model 30a.

The bankruptcy flag error calculation unit 35 calculates the error of the bankruptcy flag and the predicted value of the bankruptcy flag.

The failure detection model learning unit 36 obtains a learned failure detection model 30b based on the error and the failure detection model.

Each of the above functional configurations will be explained in detail below.

<Functional configuration of the speech synthesis device in the inference phase>
Fig. 4 is a functional configuration diagram of the speech synthesizer in the inference phase according to the first embodiment. As shown in Fig. 4, the speech synthesizer 3 has an input unit 31, a waveform generation unit 32, a conversion unit to a failure flag 33, a failure detection unit 34, and a state initialization unit 37. Note that the same reference numerals are used for the functional configurations similar to those in the learning phase, and the description thereof will be omitted.

When the state initialization unit 37 predicts a failure by indicating that the failure flag (predicted value) indicates "failure," it initializes the state of the RNN based on the initial value of the state of the RNN. This functional configuration will be described in detail later.

[Processing or operation of the voice synthesizer]
Next, the process or operation of the voice synthesis device according to the first embodiment will be described with reference to FIG. 5 and FIG.

<Processing or operation in the learning phase of the speech synthesizer>
FIG. 5 is a flowchart showing the processing or operation in the learning phase of the speech synthesis device according to the first embodiment.

First, as shown in FIG. 5, the input unit 31 receives the speech waveform of the learning data at time t.

The acoustic feature quantity corresponding to the waveform is combined with the waveform and input to the waveform generating unit 32 (S11).

Next, the waveform generation unit 32 generates the combined speech waveform and acoustic features, as well as the state of the RNN.

Based on this, the next time's audio waveform (predicted value)

Here, as the acoustic feature, spectral information such as a spectrogram or mel-cepstrum, and prosodic information such as a fundamental frequency or a pitch width are used.

Then, the failure flag change unit 33 executes the above process (S12) for times t = 1, ..., T to obtain the voice waveform (predicted value)

and audio waveform

, and the bankruptcy flag threshold value f are obtained, and the bankruptcy flag is calculated based on these.

is obtained (S13).

Here, the failure flag is a binary flag indicating whether the audio waveform at each time t is broken. In the process (S13), the failure flag conversion unit 33 converts x and

If the difference exceeds the threshold value f of the failure flag, a flag is added as a failure. If not, a flag is added as a non-failure.

Since it is often difficult to assign a waveform generator flag when comparing directly the amplitude values, it is acceptable to convert the values to power or spectrum before comparing them. This is because

If x fails, the power or spectrum will be significantly different from that of x, and the difference will be clearly visible. The mean square error or mean absolute error can be used to calculate the difference.

When acquiring the state sequence of the RNN from the waveform generating unit 32, the failure detecting unit 34 simultaneously acquires the state sequence of the RNN from the waveform generating unit 32.

, and further obtains the bankruptcy detection model 30a, and predicts the bankruptcy flag based on these, thereby obtaining the bankruptcy flag (predicted value)

(S14). When a statistical model is used as the failure detection model, a probability value of whether or not there is a failure is obtained. As the failure detection model 30a, not only DNN but also other classification models such as logistic regression and support vector machines may be used.

Next, the failure flag error calculation unit 35 acquires the failure flag and the failure flag (predicted value), and calculates the error of the failure flag and the failure flag (predicted value) (S15). Since the task of the failure detection model 30a is a classification problem of whether or not there is a failure, when a DNN is adopted as a statistical model, cross-entropy or the like can be used as an error function.

Next, the failure detection model learning unit 36 obtains a learned failure detection model 30b based on the error calculated by the failure flag error calculation unit 35 and the failure detection model 30a (S16). This process (S16) is achieved by updating the parameters of the failure detection model 30a to minimize the error, and in DNNs, error backpropagation is generally used. By repeatedly executing the above procedure for all of the training data, the prediction accuracy of the failure detection model 30a is improved. In this way, the inference phase ends.

<Processing or operation in the inference phase of the speech synthesizer>
FIG. 6 is a flowchart showing the processing or operation in the inference phase of the speech synthesis device according to the first embodiment.

First, as shown in FIG. 6, the input unit 31 inputs the speech waveform of the learning data at time t in the same manner as in the above-mentioned process (S11).

The acoustic feature quantity corresponding to the waveform is combined with the waveform and input to the waveform generating unit 32 (S111).

Next, the waveform generation unit 32 generates the combined speech waveform and acoustic features, as well as the state of the RNN, in the same manner as in the above-mentioned process (S12).

Based on this, the next time's audio waveform (predicted value)

is obtained (S112).

Next, the failure detection unit 34 receives the state of the RNN from the waveform generation unit 32.

Then, the learned failure detection model 30b is obtained, and the failure flag is predicted based on these, so that the failure flag (predicted value)

is obtained (S113).

Next, when the state initialization unit 37 predicts that the RNN has failed by indicating that the failure flag (predicted value) indicates that the RNN has failed, the state initialization unit 37 initializes the RNN state based on the initial value of the RNN state.

Initialize.

This completes the inference phase.

<Main Effects of the First Embodiment>
As described above, according to this embodiment, it is possible to prevent the deterioration of the naturalness of the voice while preventing a significant loss in the operating speed of the waveform generation. Specifically, a waveform generation failure can be detected only from information obtained by the operation of the neural vocoder. Furthermore, by initializing the state variables of the RNN at the timing when a failure is detected, it is possible to avoid problems such as unclear speech or silence.

Second Embodiment Next, a second embodiment will be described with reference to FIGS.

[Functional configuration of the voice synthesis device]
The functional configuration of the speech synthesizer 3 according to this embodiment in the learning phase is similar to the functional configuration of the speech synthesizer 3 according to the first embodiment in the learning phase, and therefore a description thereof will be omitted.

<Functional configuration of the speech synthesis device in the inference phase>
The speech synthesizer 3 according to the second embodiment further comprises a speech waveform buffering unit 41, a speech waveform averaging unit 42, and a speech waveform selection unit 48 in addition to the components of the speech synthesizer 3 according to the first embodiment.

Of these, the audio waveform buffering unit 41 is constructed in memory 302 and stores audio waveforms (predicted values).

The audio waveform averaging unit 42 obtains an averaged audio waveform based on the previous audio waveform (predicted value) stored in the buffering unit 41.

The audio waveform selection unit 48 outputs an audio waveform (predicted value) for predicting the audio waveform at the next time based on the predicted value of the bankruptcy flag, the audio waveform (predicted value), and the averaged audio waveform.

Each of the above functional configurations will be explained in detail below.

[Processing or operation of the voice synthesizer]
Next, the processing or operation of the speech synthesizer according to the second embodiment will be described with reference to Fig. 8. The processing or operation of the speech synthesizer 3 according to the present embodiment is similar to the processing or operation of the speech synthesizer 3 according to the first embodiment in the learning phase, and only a part of the inference phase is different, so the description of the processing in the learning phase will be omitted.

<Processing or operation in the inference phase of the speech synthesizer>
8 is a flowchart showing the process or operation in the inference phase of the speech synthesis device according to the second embodiment. Note that the process (S121 to S124) is similar to the process (S111 to S114) of the first embodiment, and therefore the description will be omitted.

In addition, in the process (S122), the waveform generation unit 32 generates the voice waveform (predicted value) for the next time.

Each time a speech waveform (predicted value) is obtained, the speech waveform buffering unit 41 inputs the speech waveform (predicted value) and the speech waveform buffering unit 41 stores the speech waveform (predicted value) (S125).

Next, the audio waveform averaging unit 42 obtains the immediately preceding audio waveform (predicted value) stored in the buffering unit 41, and calculates an averaged audio waveform based on this audio waveform (predicted value).

Here, the voice waveform averaging unit 42 can use a simple moving average that takes the average of the most recent N samples, or a weighted moving average or an exponential moving average that places importance on the voice waveform at the most recent time.

Next, the audio waveform selection unit 48 selects the failure flag (predicted value)

Averaged audio waveform

and speech waveform (predicted value)

Based on this, the speech waveform (prediction value) for predicting the speech waveform at the next time is output (S127). In this case, at time t+1, the failure detection unit 34 detects the failure flag

is determined to be broken, the voice waveform selection unit 48 selects and outputs the averaged voice waveform, without selecting the voice waveform (predicted value) obtained by the waveform generation unit 32. In this manner, the inference phase ends.

<Main Effects of the Second Embodiment>
As described above, according to this embodiment, in addition to the effects of the first embodiment, it is possible to eliminate the discontinuity of the voice, which is a side effect when the state h of the RNN is initialized. Furthermore, in the state h of the RNN, information on the voice waveform generated up to that time is accumulated, and the information up to that time is lost by the initialization of the state h. In other words, the continuity of the voice before and after the initialization of the state h is also lost. In this embodiment, in order to eliminate this discontinuity, several samples of the voice waveform immediately before the initialization are buffered in advance. As the voice waveform at the time of initialization, the average value of the buffered voice is used instead of the one predicted by the waveform generation unit 32. As a result, while initializing the state h, the continuity with the immediately previous voice can be secured, and failure can be prevented without causing significant deterioration in quality.

Third Embodiment Next, a third embodiment will be described with reference to FIGS.

[Functional configuration of the voice synthesis device]
The voice synthesizer 3 according to this embodiment further includes a statistics calculation unit 49 in addition to the functional configuration in the learning phase and the inference phase of the voice synthesizer 3 according to the first embodiment.

The statistics calculation unit 49 obtains statistics of the state of the RNN. This functional configuration will be described in detail later.

[Processing or operation of the voice synthesizer]
Next, the process or operation of the voice synthesis device according to the third embodiment will be described with reference to FIG.

<Processing or operation in the learning phase of the speech synthesizer>
11 is a flowchart showing the processing or operation in the learning phase of the speech synthesis device according to the third embodiment. Note that the processing (S31, S32, S33, S34, S35, S36) corresponds to the processing (S11, S12, S13, S14, S15, S16) in the first embodiment, respectively, and is mostly the same as the first embodiment, so only the differences will be described.

In this embodiment, as in the first embodiment, after processing (S31), the waveform generation unit 32 generates the speech waveform (predicted value) for the next time.

(S32). In addition, each time a prediction is made, the waveform generating unit 32 obtains the state of the RNN.

get.

In this embodiment, the statistics calculation unit 49 obtains the state of the RNN from the waveform generation unit 32 and calculates the statistics of the state of the RNN.

The statistics calculation unit 49 then executes this process for times t=1, ..., T to obtain the statistics of the RNN state.

(S32-1). Usually, the state h of the RNN is composed of a vector with a number of dimensions exceeding at least 100 in order to ensure the continuity of the speech waveform. If this is used as the input to the breakdown detection unit 34 as is, the amount of calculation increases in proportion to the number of dimensions, so the statistics calculation unit 49 converts the state h into a low-dimensional feature. Specifically, the statistics calculation unit 49

The statistical calculation unit 49 may use a vector obtained by combining the average value, standard deviation, maximum value, minimum value, etc. of the above. Alternatively, the statistical calculation unit 49 may perform dimensional compression into a low-dimensional vector by principal component analysis or linear discriminant analysis.

Then, the statistics calculation unit 49 outputs the statistics of the RNN state to the failure detection unit 34.

The flow from inputting the above to predicting the bankruptcy flag is the same as that in the first embodiment, and therefore the description thereof will be omitted. In this manner, the learning phase is completed.

<Processing or operation in the inference phase of the speech synthesizer>
12 is a flowchart showing the processing or operation in the inference phase of the speech synthesis device according to the third embodiment. Note that the processing (S131, S132, S133, S134) corresponds to the processing (S111, S112, S113, S114) in the first embodiment, respectively, and is mostly the same as the first embodiment, so only the differences will be described.

In this embodiment, as in the first embodiment, after the process (S131), the waveform generation unit 32 generates the speech waveform (predicted value) for the next time.

(S132). In addition, each time a prediction is made, the waveform generating unit 32 obtains the state of the RNN.

get.

Then, the statistics calculation unit 49 outputs the statistics of the RNN state to the failure detection unit 34.

The flow of the process thereafter is the same as in the first embodiment, and therefore will not be described. In this manner, the inference phase is completed.

<Main Effects of the Third Embodiment>
As described above, this embodiment provides the following effects in addition to the effects of the first embodiment. That is, if the state h of the RNN is used as is in the breakdown detection process described in the second embodiment, the number of dimensions of the state h is large, so the amount of calculation is large and the operation speed of the waveform generation is impaired. In contrast, this embodiment can reduce the amount of calculation required for breakdown detection, and can improve the operation speed of waveform generation including breakdown detection. In addition, it can be combined with the second embodiment, and it is possible to generate a waveform that ensures the continuity of the voice while reducing the amount of calculation required for breakdown detection.

Fourth Embodiment Next, a fourth embodiment will be described with reference to FIGS.

[Functional configuration of the voice synthesis device]
The functional configuration of a speech synthesis device according to the fourth embodiment will be described with reference to FIG. 13 and FIG.

<Functional configuration of the speech synthesis device in the learning phase>
In the speech synthesis device 3 of this embodiment, the failure flag conversion unit 33, failure detection unit 34, failure flag error calculation unit 35, and failure detection model learning unit 36 in the speech synthesis device 3 of the first embodiment are replaced with a failure detection index conversion unit 43, a failure detection index prediction unit 44, a failure detection index error calculation unit 45, and a failure detection index prediction model learning unit 46, respectively.

Of these, the failure detection index conversion unit 43 obtains a failure detection index based on the audio waveform and the audio waveform (predicted value).

The failure detection index prediction unit 44 obtains a failure detection index (predicted value) based on the RNN state sequence and the failure detection index prediction model 40a.

The failure detection index error calculation unit 45 calculates the failure detection index and the error of the failure detection index (predicted value).

The failure detection indicator prediction model learning unit 46 obtains a learned failure detection indicator prediction model 40b based on the error and the failure detection indicator prediction model 40a.

Each of the above functional configurations will be explained in detail below.

<Functional configuration of the speech synthesis device in the inference phase>
In the speech synthesizer 3 according to this embodiment, the failure detection unit 34 in the speech synthesizer 3 of the first embodiment is replaced with a failure detection index prediction unit 44. Also, instead of a failure flag (predicted value) being input to a state initialization unit 37, a failure detection index (predicted value) is input, and further a failure flag threshold value f is input.

[Processing or operation of the voice synthesizer]
Next, the process or operation of the voice synthesis device according to the fourth embodiment will be described with reference to FIG. 15 and FIG.

<Processing or operation in the learning phase of the speech synthesizer>
15 is a flowchart showing the processing or operation in the learning phase of the speech synthesis device according to the fourth embodiment. Note that since the processing (S41, S42) corresponds to the processing (S11, S12) in the first embodiment, respectively, only the differences will be described.

In this embodiment, the conversion unit 43 converts the speech waveform (predicted value) into a failure detection index by executing the above process (S42) for times t = 1, ..., T.

and audio waveform

Based on these, the indicators for detecting bankruptcy are obtained.

is obtained (S43).

Here, the indicators for detecting a failure are the audio waveform x and its predicted value used to generate the failure flag in the first embodiment.

and is the power calculated from the amplitude value of the audio waveform or the vibration value, or the error calculated from the spectrum.

In addition, the conversion unit 43 converts the audio waveform (predicted value)

When acquiring the state sequence of the RNN from the waveform generating unit 32, the failure detection index predicting unit 44 simultaneously acquires the state sequence of the RNN from the waveform generating unit 32.

, and further obtains a failure detection index prediction model 40a, and based on these, a failure detection index (prediction value)

is obtained (S44).

In addition, the failure detection index prediction unit 44 calculates the statistics of the RNN state via the statistics calculation unit 49 in the same manner as in the third embodiment.

may be acquired.

Next, the failure detection index error calculation unit 45 acquires the failure detection index and the failure detection index (predicted value), and calculates the error of the failure detection index and the failure detection index (predicted value) (S45). Since the failure detection index and the failure detection index (predicted value) are continuous values, the mean square error or the mean absolute error can be used as a method of calculating the error, as in the process of the first embodiment (S14).

Next, the failure detection index prediction model learning unit 46 obtains a learned failure detection index prediction model 40b based on the error calculated by the failure detection index error calculation unit 45 and the failure detection index prediction model 40a (S46). This process (S46) is achieved by updating the parameters of the failure detection index prediction model 40a to minimize the error, and in DNNs, backpropagation is generally used. By repeatedly executing the above procedure for all of the training data, the prediction accuracy of the failure detection index prediction model 40a is improved. In this way, the inference phase is completed.

<Processing or operation in the inference phase of the speech synthesizer>
10 is a flowchart showing the processing or operation in the inference phase of the speech synthesis device according to the fourth embodiment. Note that since the processing (S141, S142) corresponds to the processing (S111, S112) in the first embodiment, respectively, only the differences will be described.

In this embodiment, the conversion unit 43 converts the state of the RNN from the waveform generation unit 32 to an index for detecting a failure.

and further obtain a learned failure detection index prediction model 40b. By predicting the failure detection index based on these, the failure detection index (predicted value)

is obtained (S143).

Next, if the failure detection index (prediction value) is greater than the threshold value f, the state initialization unit 37 determines that the waveform generation has "failed" and resets the state of the RNN based on the initial value of the RNN state.

Initialize.

This completes the inference phase.

<Main Effects of the Fourth Embodiment>
As described above, in the failure detection processing of the first to third embodiments, when tuning the accuracy of the failure detection model trained as a discrimination model, re-learning is required, for example, by changing the threshold f of the failure flag in the learning phase or by changing learning conditions such as hyperparameters of the failure detection model. Therefore, a lot of effort is required for tuning to obtain a model that is considered optimal. In addition, since the failure detection model 30a is trained specifically for the waveform generation unit 32, when the model used in the waveform generation unit 32 is changed, the failure detection model 30a must be linked to the new model and re-tuned.

In contrast, in this embodiment, a discriminative model that predicts a discrete value failure flag is not trained, but rather a generative model that predicts an indicator for continuous failure detection is trained. Specifically, the speech synthesis device 3 of the fourth embodiment does not directly predict the failure flag from a statistical model, but predicts the value that will be the indicator, and indirectly detects failure based on whether it exceeds the threshold f. This makes it possible to reduce the issues of the first to third embodiments by simply tuning the threshold f without re-learning the model used for failure detection.

This embodiment can also be combined with the second or third embodiment, making it possible to generate a waveform that ensures audio continuity while reducing the amount of calculations required for breakdown detection.

〔supplement〕
The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

The voice synthesis device 3 can be realized by a computer and a program, but this program can also be recorded on a (non-temporary) recording medium or provided via the communication network 100.

REFERENCE SIGNS LIST 1 Communication system 3 Voice synthesis device 5 Communication terminal 30a Impairment detection model 30b Learned impairment detection model 31 Input unit 32 Waveform generation unit 33 Conversion to impair flag unit 34 Impairment detection unit 35 Impairment flag error calculation unit 36 Impairment detection model learning unit 37 State initialization unit 40a Impairment detection index prediction model 40b Learned impairment detection index prediction model 41 Voice waveform buffering unit 42 Voice waveform averaging unit 43 Conversion to impair detection index unit 44 Impairment detection index prediction unit 45 Impairment detection index error calculation unit 46 Impairment detection index prediction model learning unit

Claims (10)

A speech synthesis device for generating speech waveforms in a training phase, comprising:
a waveform generation unit that obtains a predicted value of a speech waveform at a next time based on the combined speech waveform and acoustic feature amount and a state of a recurrent neural network;
a conversion unit for converting the speech waveform into a failure flag, the conversion unit obtaining the failure flag based on the speech waveform, the predicted value of the speech waveform, and a threshold value of a failure flag indicating whether the speech waveform at each time point is failed;
a failure detection unit that obtains a predicted value of the failure flag based on a state sequence of the recurrent neural network and a failure detection model;
a bankruptcy flag error calculation unit that calculates an error between the bankruptcy flag and a predicted value of the bankruptcy flag;
a failure detection model learning unit that obtains a learned failure detection model based on the error and the failure detection model;
A speech synthesis device having the above configuration. 2. The speech synthesis device according to claim 1,
a total metric calculation unit for obtaining statistics of a state of the recurrent neural network based on the state of the recurrent neural network;
The failure detection unit obtains a predicted value of the failure flag based on statistics of the state of the recurrent neural network replaced from the state series of the recurrent neural network and the failure detection model. A speech synthesis device for generating speech waveforms in a training phase, comprising:
a waveform generation unit that obtains a predicted value of a speech waveform at a next time based on the combined speech waveform and acoustic feature amount and a state of a recurrent neural network;
a conversion unit for converting the speech waveform into a speech imperfection detection index, the conversion unit obtaining a speech imperfection detection index based on a difference between the speech waveform and a predicted value of the speech waveform;
a failure detection index prediction unit that obtains a predicted value of a failure detection index based on a state sequence of the recurrent neural network and a failure detection index prediction model;
a failure detection index error calculation unit that calculates an error between the failure detection index and a predicted value of the failure detection index;
a failure detection index prediction model learning unit for obtaining a learned failure detection model based on the error and the failure detection index prediction model;
A speech synthesis device having the above configuration. A speech synthesis device for generating a speech waveform in an inference phase, comprising:
a waveform generation unit that obtains a predicted value of a speech waveform at a next time based on the combined speech waveform and acoustic feature amount and a state of a recurrent neural network;
a failure detection unit that obtains a predicted value of a failure flag indicating whether the speech waveform at each time point is failed, based on a state of the recurrent neural network and a trained failure detection model;
a state initialization unit that initializes a state of the recurrent neural network based on an initial value of the state of the recurrent neural network when the predicted value of the failure flag indicates a failure;
A speech synthesis device having the above configuration. 5. A speech synthesis device according to claim 4,
a speech waveform buffering unit that stores the speech waveform prediction value;
a speech waveform averaging unit for obtaining an averaged speech waveform based on a predicted value of the immediately preceding speech waveform stored in the buffering unit;
a speech waveform selection unit that outputs a speech waveform prediction value for predicting a speech waveform at a next time point based on the failure flag prediction value, the speech waveform prediction value, and the averaged speech waveform;
A speech synthesis device having the above configuration. 5. A speech synthesis device according to claim 4,
a total metric calculation unit for obtaining statistics of a state of the recurrent neural network based on the state of the recurrent neural network;
The speech synthesis device, wherein the failure detection unit obtains a predicted value of the failure flag based on statistics of the state of the recurrent neural network replaced with a state series of the recurrent neural network and the learned failure detection model. A speech synthesis device for generating a speech waveform in an inference phase, comprising:
a waveform generation unit that obtains a predicted value of a speech waveform at a next time based on the combined speech waveform and acoustic feature amount and a state of a recurrent neural network;
a failure detection index prediction unit that obtains a predicted value of a failure detection index based on a state of the recurrent neural network and a learned failure detection index prediction model;
a state initialization unit that initializes a state of the recurrent neural network based on an initial value of the state of the recurrent neural network when the predicted value of the indicator of failure detection is greater than a threshold value;
A speech synthesis device having the above configuration. A speech synthesis method executed by a speech synthesis device that generates a speech waveform in a learning phase, comprising:
The speech synthesizer comprises:
A waveform generation process for obtaining a predicted value of a speech waveform at a next time based on the combined speech waveform and acoustic feature quantity and a state of a recurrent neural network;
A conversion process to obtain the failure flag based on the speech waveform, the predicted value of the speech waveform, and a failure flag threshold value indicating whether the speech waveform at each time is failure;
A failure detection process for obtaining a predicted value of the failure flag based on a state sequence of the recurrent neural network and a failure detection model;
A bankruptcy flag error calculation process for calculating an error between the bankruptcy flag and a predicted value of the bankruptcy flag;
a failure detection model learning process for obtaining a learned failure detection model based on the error and the failure detection model;
A speech synthesis method that performs A speech synthesis method executed by a speech synthesizer that generates a speech waveform in an inference phase, comprising:
The speech synthesizer comprises:
A waveform generation process for obtaining a predicted value of a speech waveform at a next time based on the combined speech waveform and acoustic feature quantity and a state of a recurrent neural network;
a failure detection process for obtaining a predicted value of a failure flag indicating whether the speech waveform at each time point is failed, based on a state of the recurrent neural network and a trained failure detection model;
a state initialization process for initializing a state of the recurrent neural network based on an initial value of the state of the recurrent neural network when the predicted value of the failure flag indicates a failure;
A speech synthesis method that performs A program for causing a computer to execute the method according to claim 8 or 9.