JP7680893B2 - Speech processing training program, speech processing training device, speech processing training method, speech processing program, speech processing device, and speech processing method - Google Patents

Description

The present invention relates to a speech processing learning program, a speech processing learning device, a speech processing learning method, a speech processing program, a speech processing device, and a speech processing method.

A voice processing device has been developed that converts the voice of any speaker into a voice with the voice quality of another speaker. For example, a technology has been disclosed that applies CycleGAN, an image conversion technology, to voice conversion (Non-Patent Document 1).

Takuhiro Kaneko and Hirokazu Kameoka, Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks arXiv:1711.11293,Nov. 2017 (EUSIPCO 2018) http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/cyclegan-vc/

In speech processing devices that synthesize and output the speech of another speaker from an original speaker, there is a demand for the voice quality and phrasing of the synthesized speech to be as natural as possible. However, with conventional training methods for speech processing devices, there were cases where the synthesized speech could not be made sufficiently natural.

One aspect of the present invention is a speech processing training program that causes a computer to function as a speech processing training device that includes an acoustic feature extractor that converts speech into input acoustic features, a speaker encoder that converts a speaker label of speech into speaker features, a speech encoder that includes a variational autoencoder having two or more sampling hierarchies that converts the input acoustic features and the speaker features into latent representations, and a speech decoder that includes a variational autoencoder having two or more sampling hierarchies that generates acoustic features using at least the latent representations and the speaker features, and that trains the speech encoder, the speech decoder, and the speaker encoder to reduce the distance between the input acoustic features input to the speech encoder and the output acoustic features generated in the speech decoder.

Here, it is preferable that the speech decoder limits the hierarchical levels at which speaker features are input in the two or more sampling hierarchical levels.

It is also preferable that the speech decoder does not input speaker features to layers prior to a predetermined layer in the two or more sampling layers, and inputs speaker features to layers subsequent to the predetermined layer.

It is also preferable that the audio decoder samples from a posterior distribution in layers prior to the predetermined layer in the two or more sampling layers, and samples from a prior distribution in layers subsequent to the predetermined layer.

Furthermore, it is preferable that the speech decoder inputs the speaker features into a conditional instance normalization layer.

Another aspect of the present invention is a speech processing program that causes a computer to function as a speech processing device that includes an acoustic feature extractor that converts speech into acoustic features, a speaker encoder that converts a speaker label of speech into speaker features, a speech encoder that includes a variational autoencoder having two or more sampling hierarchies that converts source acoustic features obtained by converting the speech of a source speaker in the acoustic feature extractor and source speaker features obtained by converting the speaker label of the source speaker in the speaker encoder into latent representations, a speech decoder that includes a variational autoencoder having two or more sampling hierarchies that generates target acoustic features using at least the latent representations and target speaker features obtained by converting the speaker label of a target speaker in the speaker encoder, and a vocoder that converts the target acoustic features generated by the speech decoder into speech.

The speech processing program is characterized in that the speech encoder, speech decoder, and speaker encoder are trained to reduce the distance between the acoustic features input to the speech encoder and the acoustic features generated by the speech decoder.

It is also preferable that the speech decoder limits the hierarchical level to which the target speaker features are input in the two or more sampling hierarchical levels.

It is also preferable that the speech decoder does not input the target speaker features to a layer prior to a predetermined layer in the two or more sampling layers, and inputs the target speaker features to a layer subsequent to the predetermined layer.

It is also preferable that the audio decoder samples from a posterior distribution in layers prior to the predetermined layer in the two or more sampling layers, and samples from a prior distribution in layers subsequent to the predetermined layer.

Furthermore, it is preferable that the speech decoder inputs the target speaker features into a conditional instance normalization layer.

Another aspect of the present invention is a speech processing training device comprising: an acoustic feature extractor that converts speech into input acoustic features; a speaker encoder that converts a speaker label of speech into speaker features; a speech encoder including a variational autoencoder having two or more sampling hierarchies that converts the acoustic features and the speaker features into latent representations; and a speech decoder including a variational autoencoder having two or more sampling hierarchies that generates acoustic features using at least the latent representations and the speaker features, wherein the speech encoder, the speech decoder, and the speaker encoder are trained to reduce the distance between the input acoustic features input to the speech encoder and the output acoustic features generated in the speech decoder.

Another aspect of the present invention is a speech processing device comprising: an acoustic feature extractor that converts speech into acoustic features; a speaker encoder that converts a speaker label of speech into speaker features; a speech encoder including a variational autoencoder having two or more sampling hierarchies that converts source acoustic features obtained by converting the speech of a source speaker in the acoustic feature extractor and source speaker features obtained by converting the speaker label of the source speaker in the speaker encoder into latent representations; a speech decoder including a variational autoencoder having two or more sampling hierarchies that generates target acoustic features using at least the latent representations and target speaker features obtained by converting the speaker label of a target speaker in the speaker encoder; and a vocoder that converts the target acoustic features generated by the speech decoder into speech.

Another aspect of the present invention is a speech processing training method comprising: a speech processing training device including an acoustic feature extractor that converts speech into input acoustic features; a speaker encoder that converts a speaker label of speech into speaker features; a speech encoder including a variational autoencoder having two or more sampling hierarchies that converts the acoustic features and the speaker features into latent representations; and a speech decoder including a variational autoencoder having two or more sampling hierarchies that generates acoustic features using at least the latent representations and the speaker features, the speech encoder, the speech decoder, and the speaker encoder being trained to reduce the distance between the input acoustic features input to the speech encoder and the output acoustic features generated in the speech decoder.

Another aspect of the present invention is a speech processing method for converting the speech of the source speaker to the speech of the target speaker using a speech processing device including: an acoustic feature extractor that converts speech into acoustic features; a speaker encoder that converts the speaker label of the speech into speaker features; a speech encoder including a variational autoencoder having two or more sampling hierarchies that converts source acoustic features obtained by converting the speech of the source speaker in the acoustic feature extractor and source speaker features obtained by converting the speaker label of the source speaker in the speaker encoder into latent representations; a speech decoder including a variational autoencoder having two or more sampling hierarchies that generates target acoustic features using at least the latent representations and target speaker features obtained by converting the speaker label of the target speaker in the speaker encoder; and a vocoder that converts the target acoustic features generated by the speech decoder into speech.

According to the present invention, it is possible to provide a speech processing training program, a speech processing training device, a speech processing training method, a speech processing training program, a speech processing training device, and a speech processing training method that appropriately convert speech uttered by any speaker into speech uttered by a target speaker. Other objects of the embodiments of the present invention will become apparent by referring to this specification as a whole.

1 is a diagram illustrating a configuration of a voice processing device according to an embodiment of the present invention. 1 is a functional block diagram showing a configuration of a speech processing training device according to an embodiment of the present invention. FIG. 1 is a diagram showing the configuration of a variational auto-encoder. FIG. 1 shows the configuration of a nouveau variational auto-encoder. FIG. 2 is a diagram showing the configuration of a neural network in each layer of a variational auto-encoder in an embodiment of the present invention. FIG. 4 is a diagram for explaining a voice learning process according to the embodiment of the present invention. 1 is a functional block diagram showing a configuration of a pronunciation learning device according to an embodiment of the present invention; FIG. 2 is a diagram for explaining audio processing according to the embodiment of the present invention.

As shown in FIG. 1, the voice processing device 100 according to the embodiment of the present invention includes a processing unit 10, a storage unit 12, an input unit 14, an output unit 16, and a communication unit 18. The processing unit 10 includes a means for performing arithmetic processing such as a CPU. The processing unit 10 executes a voice processing learning program stored in the storage unit 12 to learn voice processing in the present embodiment. The processing unit 10 also executes a voice processing program stored in the storage unit 12 to realize functions related to voice processing in the present embodiment. The storage unit 12 includes a storage means such as a semiconductor memory or a memory card. The storage unit 12 is connected to the processing unit 10 in an accessible manner, and stores the voice processing learning program, the voice processing program, and information required for the processing. The input unit 14 includes a means for inputting information. The input unit 14 includes, for example, a keyboard, a touch panel, a button, etc. that receives information input from a user. The input unit 14 also includes a voice input means for receiving input of the voice of an arbitrary speaker and a specific target speaker. The voice input means may include, for example, a microphone, an amplifier circuit, etc. The output unit 16 includes a user interface screen (UI) for receiving input information from an administrator and a means for outputting processing results. The output unit 16 includes, for example, a display for presenting an image. The output unit 16 also includes an audio output means for outputting a synthetic voice generated by the voice processing device 100. The audio output means may include, for example, a speaker, an amplifier, and the like. The communication unit 18 includes an interface for communicating information with an external terminal (not shown) via the network 102. Communication by the communication unit 18 may be wired or wireless. Note that the audio information provided for audio processing may be acquired from an external terminal via the communication unit 18.

The voice processing device 100 performs voice processing to convert a voice uttered by an arbitrary speaker into the quality of the voice of a specified speaker (target speaker). The voice processing device 100 also functions as a voice processing learning device that learns for the voice processing.

[Voice learning process]
2 is a functional block diagram showing the configuration of the speech processing device 100 during speech processing training. The speech processing device 100 functions as a speech analyzer 20, a speaker encoder 22, a speech encoder 24, a speech decoder 26, and a learner 28. Specifically, the speech processing device 100 functions as a speech processing training device that realizes the following speech training method by executing a speech processing training program.

The voice analysis unit 20 functions as an acoustic feature extractor that acquires voice data and extracts acoustic features from the voice data. That is, the processing unit 10 of the voice processing device 100 functions as the voice analysis unit 20. Voice data may be acquired by converting the speaker's voice into voice data using a microphone that constitutes the input unit 14. Voice data that has been pre-recorded in an external computer or the like may also be received via the communication unit 18. The acquired voice data is stored in the memory unit 12.

The voice data acquisition process is performed on voices uttered by any speaker. In the voice training process, the voice encoder 24 and the voice decoder 26 are trained using voices from multiple speakers. The voices obtained from each speaker do not need to be identical in content.

Further, the voice analysis unit 20 performs voice analysis necessary for voice processing. For example, the voice analysis unit 20 performs cepstrum analysis of the voice based on the frequency characteristics of the input voice, and obtains acoustic features such as Mel Frequency Cepstrum Coefficients (MFCC) including spectrum envelope (information indicating the thickness of the voice, etc.) and fine structure information, and fundamental frequency and resonance frequency of the voice (information indicating the pitch of the voice, hoarseness, etc.). The acoustic features can be, for example, an (80×T)-dimensional Euclidean space with respect to the length T of the voice segment. Specifically, the voice analysis unit 20 generates and outputs an acoustic feature x _i from the voice uttered by a speaker with a speaker ID (speaker label) of i. The acoustic features extracted by the voice analysis unit 20 are input to the voice encoder 24 and the learning device 28.

The speaker encoder 22 converts the ID of the speaker of the voice input to the voice analysis unit 20 into a speaker feature that can be used for voice processing and outputs it. The speaker encoder 22 can be configured to include an embedded module that converts the speaker ID into a speaker feature and outputs it. For example, when the speaker ID is i, the speaker encoder 22 generates and outputs a speaker feature y _i . The speaker feature generated by the speaker encoder 22 is input to the voice encoder 24 and the voice decoder 26.

In training the speech processing apparatus 100, a set of combinations (x _i , y _i ) of acoustic features x _i and speaker features y _i obtained from speech uttered by a plurality of speakers is used.

The speech encoder 24 receives the acoustic features and speaker features as input, and performs a process of converting the acoustic features and speaker features into latent representations. The speech decoder 26 receives the latent representations and speaker features obtained by the speech encoder 24 as input, and performs a process of converting the latent representations and speaker features into acoustic features. The latent representations represent the linguistic features of the input speech data.

As shown in FIG. 2, the speech encoder 24 and the speech decoder 26 receive an input of an acoustic feature x _i from the speech analysis unit 20, convert it to a latent representation z in the speech encoder 24, and further reconstruct it from the latent representation z into an acoustic feature x _i ^ in the speech decoder 26, and are trained so that the output acoustic feature x _i ^ restores the input acoustic feature x _i .

In this embodiment, the speech encoder 24 and speech decoder 26 are configured by a variational auto-encoder (VAE). The variational auto-encoder is a type of variational auto-encoder, and generates a latent representation by sampling based on a probability distribution, as shown in FIG. 3. The probability distribution is assumed to be a normal distribution defined by the mean μ and variance σ. The variational auto-encoder is a combination of an encoder that generates a latent representation z by sampling based on the mean μ and variance σ for the input X, and a decoder that generates an output X^ from the latent representation z. In the variational auto-encoder, the speaker encoder 22, speech encoder 24, and speech decoder 26 are trained so that the restoration error (restoration distance) E between the input X and the output X^ is small.

As shown in FIG. 4, a typical variational auto-encoder is composed of a single-layer neural network, but in this embodiment, it is preferable to use a nouveau variational auto-encoder (NVAE) composed of a neural network with two or more layers. In other words, the nouveau variational auto-encoder is composed of a variational auto-encoder having two or more sampling layers. For example, in the audio processing device 100, it is preferable to configure the audio encoder 24 and the audio decoder 26 with n=35 layers of neural networks, respectively.

As shown in FIG. 5, each layer of the nouveau variational auto-encoder of the speech encoder 24 and the speech decoder 26 is configured by combining a conditional-instance-normalization layer (CIN layer), a convolution layer (CONV layer), and a squeeze-and-excitation layer (SE layer). The CIN layer is a layer provided in place of a batch normalization hypothesis (BN layer) in a general nouveau variational auto-encoder. The CIN layer is one of the normalization layers, and is a conditional instance normalization layer that performs normalization by setting different parameters for each style. In this embodiment, the CIN layer uses a speaker feature as one of the inputs and performs normalization conditioned by the input speaker feature. The Swish activation function is an activation function expressed as f(x)=x/(1+e ^−βx ). The convolution layer is a layer that applies a convolution operation to an input and outputs the operation result to the next layer. The SE layer is a layer that adaptively applies an attention to the input based on the relationship between channels and outputs a weighted feature.

The voice learning process in the voice processing device 100 will be described with reference to FIG. 6. The voice encoder 24 and the voice decoder 26 are each configured with a neural network having a layer number n. The layer number n can be, for example, 35 layers. Each layer is configured by combining the Conditional-Instance-Normalization layer (CIN layer), the Convolution layer (CONV layer), and the Squeeze-and-Excitation layer (SE layer) shown in FIG. 5. Note that the latent expression output from layer k (where k indicates the layer number from 1 to n) of the voice encoder 24 is represented by h _k , and the latent expression output from the layer represented by the layer number k of the voice decoder 26 is represented by z _k .

In the speech encoder 24, the acoustic feature x _i and the speaker feature y _i are input to the layer n, and a latent expression h _n is output. In the next layer n-1, the latent expression h _n and the speaker feature y _i output from the previous layer n are input, and a latent expression h _n-1 is output. Similarly, in the layer k, the latent expression h _k+1 and the speaker feature y _i output from the previous layer k+1 are input, and a latent expression h _k is output. In the layer 1, which is the final layer, the latent expression h ₂ and the speaker feature y _i output from the previous layer 2 are input, and a latent expression h ₁ is output. From the latent expression h ₁ , the latent expression z ₁ of the layer 1, which is the first layer of the speech decoder 26, is sampled. In this way, in the speech encoder 24, it is preferable to include the speaker feature y _i in the input in all layers 1 to n.

In the speech decoder 26, latent expression _z1 is input to layer 1, which is the first stage, and latent expression _z2 is output. Furthermore, latent expression _zk in layer k of the speech decoder 26 can be obtained by sampling from a prior distribution p( _zk |zk _-1 , _hk , yi ₎ based on latent expression zk _{-1 in} the previous layer k-1 of the speech decoder 26, latent expression hk in the kth layer of the speech encoder 24, and speaker feature _yi . Furthermore, latent expression _zk can be obtained by sampling from a posterior distribution p(zk|zk-1, zk-2... _z1 , hk) based on latent expression zk- ₁ in the previous layer k _-1 , layer _{k -2} ...layer 1 of the speech decoder 26, latent expression zk _{-2 ...} latest expression _z1 , and latent expression hk in the _kth layer of the speech encoder 24. The distribution p(a|b) is a likelihood function indicating the likelihood that a will be the output with b as a prerequisite.

In the speech learning process, sampling is performed from the speech encoder 24 across layers from close to the output of the speech decoder 26 to layers far from it. That is, as shown in Fig. 6, it is preferable to perform sampling from the latent representation _hk of the kth layer of the speech encoder 24 in all layers 1 to n. It is also preferable not to include the speaker feature _yi in the input when sampling from the posterior distribution.

That is, it is preferable that the speech decoder 26 includes the speaker feature _yi in the input only in layers close to the output, and does not include the speaker feature _yi in the input in layers far from the output. In this case, it is preferable that the speaker feature _yi is included in the input in layers where sampling is not performed from the speech encoder 24 and sampling is performed from a prior distribution, and the speaker feature _yi is not included in the input in layers where sampling is performed from the speech encoder 24 and sampling is performed from a posterior distribution.

In addition, in a layer in which the speaker feature y _i is not included in the sampling, the speaker feature y _i is not input to the Conditional-Instance-Normalization layer (CIN layer).

In such a configuration, the learning device 28 adjusts various parameters (such as the weighting coefficient or bias of each neuron) of the neural networks of each layer included in the speaker encoder ₂₂ , the speech encoder 24, and the speech decoder 26 so as to reduce the error (distance) between the acoustic feature x _i input to the speech decoder 26 and the reconstructed acoustic feature x i ^ output from the speech decoder 26.

Here, in order to reduce the error (distance) between the acoustic feature x _i input to the speech decoder 26 and the reconstructed acoustic feature x _i ^ output from the speech decoder 26, a hierarchical layer that serves as the boundary between a hierarchical layer in which sampling is performed from a prior distribution that takes into account the speaker feature y _i in the speech decoder 26 and a hierarchical layer in which sampling is performed from a posterior distribution that does not take into account the speaker feature y _i can be appropriately set.

As described above, the speech encoder 24 and the speech decoder 26 are trained so that the speech represented by the acoustic feature x _i input to the speech encoder 24 approaches the speech represented by the acoustic feature x _i ^ reconstructed in the speech decoder 26.

[Audio processing]
7 is a functional block diagram showing the configuration of the voice processing device 100 during voice processing for converting the voice uttered by the source speaker into a voice uttered by the target speaker. The voice processing device 100 functions as a voice analysis unit 20, a speaker encoder 22, a voice encoder 24, a voice decoder 26, and a vocoder 30. Specifically, the voice processing device 100 functions as a voice processing device that realizes the following voice processing by executing a voice processing program.

The speech analysis unit 20 acquires speech data of the speech uttered by the source speaker and performs the speech analysis required for speech processing. The acoustic features extracted by the speech analysis unit 20 are input to the speech encoder 24.

The speaker encoder 22 converts the IDs of the source speaker and the target speaker into speaker features that can be used for speech processing and outputs them. When the source speaker ID is a speaker of s, the speaker encoder 22 generates a source speaker feature y _s and outputs it to the speech encoder 24. When the target speaker ID is a speaker of t, the speaker encoder 22 generates a target speaker feature y _t and outputs it to the speech decoder 26.

The speech encoder 24 receives the acoustic features and source speaker features obtained from the speech of the source speaker, and performs a process of converting the acoustic features and the source speaker features into latent representations. The speech decoder 26 receives the latent representation and target speaker features obtained by the speech encoder 24, and performs a process of reconstructing the acoustic features from the latent representation and the target speaker features.

The voice processing in the voice processing device 100 will be described with reference to FIG. 8. The voice processing is performed using the voice encoder 24 and the voice decoder 26 that have been trained in the above-mentioned voice training process.

In the speech encoder 24, acoustic features _xs and source speaker features _ys obtained from the speech of the source speaker are input to layer n, and a latent representation _hn is output. Thereafter, as in the learning process, latent representation hk ₊₁ and source speaker features _ys output from the previous layer k+1 are input to layer k, and latent representation _hk is output. In layer 1, which is the final layer, latent representation _h2 and source speaker features _ys output from the previous layer 2 are input, and latent representation _h1 is output. From this latent representation _h1 , latent representation _z1 of layer 1, which is the first layer of the speech decoder 26, is sampled.

In the speech decoder 26, latent expression _z1 is input to the first layer, layer 1, and latent expression _z2 is output. In layers far from the output of the speech decoder 26, the target speaker feature _yt is not included in the input, and sampling is performed from the posterior distribution p(zk|zk-1, zk-2 _... z1, hk) based on the latent expression zk _{-1 ,} latent expression zk _-2 ...latent expression z1 of the previous layers _{k -1} , _k-2 ...layer 1 in the speech decoder 26 and the latent expression _hk of the _kth layer of the speech encoder 24. In layers close to the output of the speech decoder 26, sampling is not performed from the speech encoder 24, but is performed from the prior distribution p( _zk |zk _-1 , _yt ) based on the latent expression zk _- 1 of the immediately preceding layer k-1 and the target speaker feature _yt . FIG. 8 shows an example of sampling from the prior distribution in layers n-1 and n of the speech decoder 26. In this case, it is preferable to include target speaker features _yt as input for sampling from the prior distribution, rather than source speaker features _ys .

Through speech processing in the speech encoder 24 and speech decoder 26, acoustic features xt are constructed and output from the final stage, hierarchical layer _n , of the speech decoder 26, by matching the acoustic features _xs obtained from the source speaker's speech to the target speaker's speech.

The vocoder 30 converts the acoustic feature _xt output from the voice decoder 26 into voice data and outputs it. The vocoder 30 can convert the acoustic feature _xt into voice data by performing a process reverse to the process performed by the voice analysis unit 20 to extract the acoustic feature from the voice data.

As described above, the voice processing device 100 of this embodiment can provide a voice processing device, a voice processing program, and a voice processing method, as well as a voice learning processing device, a voice learning processing program, and a voice learning processing method, that appropriately convert the voice uttered by an arbitrary speaker into the sound quality of the voice uttered by a target speaker. In other words, the voice processing device 100 including the trained voice encoder 24 and voice decoder 26 can realize voice processing that converts the voice uttered by a source speaker into a voice that sounds like the voice uttered by a target speaker.

In particular, by applying the Nouveau Variational Auto-Encoder (NVAE) to the speech encoder 24 and speech decoder 26, the speech of the source speaker can be converted into a more natural-sounding speech of the target speaker than in the past.

10 Processing unit, 12 Memory unit, 14 Input unit, 16 Output unit, 18 Communication unit, 20 Voice analysis unit, 22 Speaker encoder, 24 Voice encoder, 26 Voice decoder, 28 Learning device, 30 Vocoder, 100 Voice processing device, 102 Network.

Claims (10)

Computer,
an acoustic feature extractor that converts speech into input acoustic features;
A speaker encoder that converts the speaker labels of the speech into speaker features;
A speech encoder including a variational autoencoder having a sampling hierarchy of hierarchy n (where n is an integer equal to or greater than 2) that converts input acoustic features and speaker features into latent representations;
A speech decoder including a variational autoencoder having a sampling hierarchy of n layers that generates acoustic features using at least a latent representation and speaker features;
and functioning as a speech processing learning device having the
In a sampling hierarchical level n of the first stage in the speech encoder, the input acoustic feature and the speaker feature are input, and a latent representation h is output;
In each sampling hierarchical layer (m-1) (where m is an integer from n to 2) from the second stage onwards in the speech encoder, a latent representation h(m-1) is output in response to an input of the latent representation h(m-1) output from the sampling hierarchical layer (m) in the previous stage in the speech encoder and the speaker feature;
In the sampling layer of the first layer 1 of the speech decoder, a latent representation h1 output from the final layer 1 of the speech encoder is received as an input and a latent representation Z1 is output;
In each sampling hierarchical layer k (where k is an integer from 2 to n) from the second stage onwards in the speech decoder, a latent representation Z(k-1) output from a sampling hierarchical layer (k-1) in the preceding stage in the speech decoder and a latent representation hk output from hierarchical layer k in the speech encoder are input, and a latent representation Zk is output;
The speech encoder, the speech decoder, and the speaker encoder are trained to reduce the distance between an input acoustic feature input to the speech encoder and an output acoustic feature generated from a latent representation Zn output from the speech decoder. 2. The speech processing training program according to claim 1,
2. A speech processing training program, comprising: a speech decoder, the speech decoder being configured such that a hierarchy for inputting speaker features is limited in the sampling hierarchy. 3. The speech processing training program according to claim 2,
the speech decoder does not input speaker features to layers preceding a predetermined layer in the sampling layers, and inputs speaker features to layers following the predetermined layer. 4. The speech processing training program according to claim 3,
The speech decoder is characterized in that it samples from a posterior distribution in layers prior to the specified layer in the sampling layer, and samples from a prior distribution in layers subsequent to the specified layer. The speech processing training program according to any one of claims 1 to 4,
The speech processing training program is characterized in that the speech decoder inputs speaker features to a conditional instance normalization layer. Computer,
an acoustic feature extractor that converts speech into acoustic features;
The speaker encoder, the speech encoder, and the speech decoder trained by the speech processing training program according to claim 1;
A vocoder for converting acoustic features into speech;
and make it work.
inputting source acoustic features obtained by converting the speech of the source speaker in the acoustic feature extractor and source speaker features obtained by converting the speaker label of the source speaker in the speaker encoder into the speech encoder and converting them into latent representations;
The latent representation and a target speaker feature obtained by converting the speaker label of the target speaker in the speaker encoder are input to the speech decoder to generate a target acoustic feature;
A speech processing program, comprising : inputting the target acoustic feature generated by the speech decoder into the vocoder to convert the target acoustic feature into speech. an acoustic feature extractor that converts speech into input acoustic features;
A speaker encoder that converts the speaker labels of the speech into speaker features;
a speech encoder including a variational autoencoder having a sampling hierarchy of n (where n is an integer equal to or greater than 2) that converts acoustic features and speaker features into a latent representation;
A speech decoder including a variational autoencoder having a sampling hierarchy of n layers that generates acoustic features using at least a latent representation and speaker features;
Equipped with
In a sampling hierarchical level n of the first stage in the speech encoder, the input acoustic feature and the speaker feature are input, and a latent representation h is output;
In each sampling hierarchical layer (m-1) (where m is an integer from n to 2) from the second stage onwards in the speech encoder, a latent representation h(m-1) is output in response to an input of the latent representation h(m-1) output from the sampling hierarchical layer (m) in the previous stage in the speech encoder and the speaker feature;
In the sampling layer of the first layer 1 of the speech decoder, a latent representation h1 output from the final layer 1 of the speech encoder is received as an input and a latent representation Z1 is output;
In each sampling hierarchical layer k (where k is an integer from 2 to n) from the second stage onwards in the speech decoder, a latent representation Z(k-1) output from a sampling hierarchical layer (k-1) in the preceding stage in the speech decoder and a latent representation hk output from hierarchical layer k in the speech encoder are input, and a latent representation Zk is output;
The speech processing training device is characterized in that the speech encoder, the speech decoder, and the speaker encoder are trained to reduce the distance between an input acoustic feature input to the speech encoder and an output acoustic feature generated from a latent representation Zn output from the speech decoder. an acoustic feature extractor that converts speech into acoustic features;
The speaker encoder, the speech encoder, and the speech decoder trained by the speech processing training device according to claim 7;
A vocoder for converting acoustic features into speech;
Equipped with
inputting source acoustic features obtained by converting the speech of the source speaker in the acoustic feature extractor and source speaker features obtained by converting the speaker label of the source speaker in the speaker encoder into the speech encoder and converting them into latent representations;
The latent representation and a target speaker feature obtained by converting the speaker label of the target speaker in the speaker encoder are input to the speech decoder to generate a target acoustic feature ;
The speech processing device according to claim 1, wherein the target acoustic feature generated by the speech decoder is input to the vocoder to convert the target acoustic feature into speech. an acoustic feature extractor that converts speech into input acoustic features;
A speaker encoder that converts the speaker labels of the speech into speaker features;
a speech encoder including a variational autoencoder having a sampling hierarchy of n (where n is an integer equal to or greater than 2) that converts acoustic features and speaker features into a latent representation;
A speech decoder including a variational autoencoder having a sampling hierarchy of n layers that generates acoustic features using at least a latent representation and speaker features;
A speech processing training device comprising:
In a sampling hierarchical level n of the first stage in the speech encoder, the input acoustic feature and the speaker feature are input, and a latent representation h is output;
In each sampling hierarchical layer (m-1) (where m is an integer from n to 2) from the second stage onwards in the speech encoder, a latent representation h(m-1) is output in response to an input of the latent representation h(m-1) output from the sampling hierarchical layer (m) in the previous stage in the speech encoder and the speaker feature;
In the sampling layer of the first layer 1 of the speech decoder, a latent representation h1 output from the final layer 1 of the speech encoder is received as an input and a latent representation Z1 is output;
In each sampling hierarchical layer k (where k is an integer from 2 to n) from the second stage onwards in the speech decoder, a latent representation Z(k-1) output from a sampling hierarchical layer (k-1) in the preceding stage in the speech decoder and a latent representation hk output from hierarchical layer k in the speech encoder are input, and a latent representation Zk is output;
The speech processing training method is characterized in that the speech encoder, the speech decoder, and the speaker encoder are trained to reduce the distance between an input acoustic feature input to the speech encoder and an output acoustic feature generated from a latent representation Zn output from the speech decoder. an acoustic feature extractor that converts speech into acoustic features;
The speaker encoder, the speech encoder and the speech decoder trained by the speech processing training method of claim 9;
A vocoder for converting acoustic features into speech;
In a voice processing device comprising:
inputting source acoustic features obtained by converting the speech of the source speaker in the acoustic feature extractor and source speaker features obtained by converting the speaker label of the source speaker in the speaker encoder into the speech encoder and converting them into latent representations;
The latent representation and a target speaker feature obtained by converting the speaker label of the target speaker in the speaker encoder are input to the speech decoder to generate a target acoustic feature;
a voice processing method comprising: inputting the target acoustic feature generated by the voice decoder to the vocoder to convert the target acoustic feature into voice;

Images