JP7243760B2 - Audio feature compensator, method and program - Google Patents

Description

The present invention relates to a feature compensation device, a feature compensation method, and a program for robustly compensating feature vectors of utterances and speech.

Speaker recognition is the recognition of a person from their voice. No two voices sound alike because of differences in the shape of the vocal tract, the size of the larynx, and other parts of the sound-producing organs. Given the uniqueness of the human voice, speaker recognition is increasingly applied to telephone-based services, such as telephone banking, where evidence of unauthorized access is to be discovered.

Speaker recognition systems can be divided into text-dependent and text-independent systems. In text-dependent systems, recognition phrases are either fixed or pre-recognized. In a text-independent system, there are no restrictions on the words a speaker can use. Text-independent recognition is broadly applicable, far more challenging for the two tasks, and has been consistently improved over the past decades.

Since reference (what is spoken in training) and test (test (what is said in actual use)) utterances in text-independent speaker recognition applications can be quite different, recognition systems should take into account this voice discrepancy. Performance is highly dependent on audio length. If the user speaks for a long period of time, typically a minute or more, it is assumed that most phonemes have been covered. As a result, the recognition accuracy is high even if the speech contents are different. However, in the case of short-time speech, the speaker feature vector of the utterance extracted by the statistical method is unreliable for accurate recognition.

In real speaker verification applications, only short speech intervals are often observed during testing. In general, short speech intervals of less than 10 seconds are common. Therefore, it is important to recover speaker feature vectors to improve text-independent speaker recognition with short utterances.

Patent Literature 1 discloses a technique of using a denoising autoencoder (DAE) to restore a speaker feature vector of short speech containing limited pronunciation information.

As shown in FIG. 23, the DAE-based feature correction device described in Patent Document 1 first estimates the degree of acoustic diversity of an input utterance as a posteriori probability based on a speech model. Both the degree of acoustic diversity and the recognition feature vector are then provided to the input layer 401 . As used herein, "feature vector" means a set of numerical values (specific data) representing an object. A DAE-based transform comprising an input layer 401, one or more hidden layers 402, and an output layer 403, with the help of supervised training using pairs of long and short speech intervals, in the output layer A reconstructed recognition feature vector can be generated.

Non-Patent Document 1 discloses MFCCs (Mel-Frequency Cepstrum Coefficients) as acoustic features.

U.S. Patent Application Publication No. 2016/0098993

Najim Dehak, Patrick J. Kenny, Reda Dehak, Pierre Dumouchel, and Pierre Ouellet, "Front-End Factor Analysis for Speaker Verification", IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 19, NO. 4, MAY 2011

However, in US Pat. No. 5,700,000, only mean squared error minimization is used in DAE optimization. Such an objective function is too simple for accurate execution. Also, when using a simple objective function, better results cannot be obtained unless the short speech is restricted to a portion of the long speech. In reality, only long-duration speech can be used to train such networks (short-duration speech is clipped from it). The information of the brief speech of the speaker present is wasted. The system requires a sufficient number of speakers with multiple long speeches for training. That may not be practical for all applications.

SUMMARY OF THE INVENTION It is an object of the present invention to provide robust feature compensation for short-duration speech in view of the above situation.

An exemplary aspect of the speech feature compensator is a first feature extracted from a short speech segment by a predetermined speaker and a second feature extracted from a longer speech segment than the short speech segment by the predetermined speaker. extracted from the input short - term speech using and generating means for generating a reconstructed feature amount based on the feature amount.

An exemplary aspect of the speech processing method is a first feature extracted from a short speech segment by a predetermined speaker and a second feature extracted from a longer speech segment than the short speech segment by the predetermined speaker. train a GAN generator and classifier using features, output trained parameters, and use the trained parameters to extract features from the input short-term speech to generate restored features.

An exemplary aspect of the speech processing program is to provide a computer with a first feature extracted from a short speech interval by a predetermined speaker and a speech interval longer than the short speech interval by the predetermined speaker. A process of training a GAN generator and a discriminator using the second feature extracted from and outputting the trained parameters, and using the trained parameters, the input short time and a process of generating a restored feature amount based on the feature amount extracted from the voice.

According to the present invention, a speech compensator, speech feature compensation method, and program can provide robust feature compensation for short-time speech.

1 is a block diagram of a robust feature compensator of a first embodiment of the present invention; FIG. It is a figure which shows an example of the content of a short time audio|voice data storage part. FIG. 4 is a diagram showing an example of the contents of a long-time voice data storage unit; FIG. 4 is a diagram showing an example of the contents of a generator parameter storage unit; 1 is a diagram showing the concept of NN architecture in the first embodiment; FIG. 4 is a flow chart showing the operation of the robust feature compensator of the first embodiment; 4 is a flow chart showing the operation of the training phase of the robust feature compensator of the first embodiment; 4 is a flow chart showing the operation of the robust feature compensation phase of the robust feature compensation device of the first embodiment; Fig. 2 is a block diagram of a robust feature compensator of the second embodiment of the present invention; FIG. 4 is a diagram showing the concept of NN architecture in the second embodiment; 9 is a flow chart showing the operation of the robust feature compensator of the second embodiment; FIG. 10 is a flow chart showing the operation of the training phase of the robust feature compensator of the second embodiment; FIG. 9 is a flow chart showing the operation of the robust feature compensation phase of the robust feature compensator of the second embodiment; Fig. 3 is a block diagram of a robust feature compensator of a third embodiment of the present invention; FIG. 10 is a diagram showing the concept of NN architecture in the third embodiment; 10 is a flow chart showing the operation of the robust feature compensator of the third embodiment; FIG. 10 is a flow chart showing the operation of the training phase of the robust feature compensator of the third embodiment; FIG. FIG. 12 is a flow chart showing the operation of the robust feature compensation phase of the robust feature compensator of the third embodiment; FIG. FIG. 2 illustrates a computer configuration that can be used with embodiments in accordance with the present invention; FIG. 2 illustrates a computer configuration that can be used with embodiments in accordance with the present invention; 1 is a block diagram showing main parts of an audio feature compensation device; FIG. FIG. 11 is a block diagram showing another aspect of the audio feature compensation device; 1 is a block diagram showing a feature compensator disclosed in Patent Document 1; FIG.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The following detailed description is merely exemplary and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any concept presented in the preceding background of the invention or the following detailed description.

Skilled artisans appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in figures illustrating integrated circuit architecture may be exaggerated relative to other elements to help facilitate understanding of this and other embodiments.

In practical speaker recognition applications, text-independent speaker recognition is often used and short speech intervals (less than 10 seconds) are observed. In such cases, voice inconsistencies need to be taken into account. This is because an unbalanced speech distribution results in less reliable speaker feature vectors extracted from short-term speech. Shorter interval lengths result in lower performance. Therefore, there is a need to improve text-independent speaker recognition with short utterances by speaker feature restoration methods.

In view of the above, the following embodiments utilize a Generative Adversarial Network (GAN) that includes generators and classifiers that improve each other during the iterative training process. The generator produces robust feature vectors for short duration speech by compensation.

First Embodiment The robust feature compensator of the first embodiment can use a generator to provide robust feature vectors for short speech intervals from raw feature vectors of short speech. . That is, in this embodiment, a GAN generator trained on short and long speech can generate robust feature vectors even from short speech. The period of long-term speech is longer than the period of short-term speech.

<Configuration of Robust Feature Compensator>
In a first embodiment of the present invention, a robust feature compensator for feature recovery using GAN generators is described.

FIG. 1 is a block diagram showing a robust feature compensator 100 of the first embodiment. A robust feature compensation device 100 comprises a training section 100A and a feature restoration section 100B.

The training unit 100A includes a short-time speech data storage unit 101, a long-time speech data storage unit 102, feature extraction units 103a and 103b, a noise storage unit 104, a generator/discriminator training unit 105, and a generator parameter storage unit 106. include. The feature restoration unit 100B includes a feature extraction unit 103c, a generator 107, and a generated feature storage unit . Feature extractors 103a, 103b, and 103c have the same function.

The short speech data storage unit 101 stores short speech recordings with speaker labels as shown in FIG.

The long-term voice data storage 102 stores long-term voice recordings with speaker labels as shown in FIG. Long-term speech data storage 102 contains at least one long-term speech recording for each speaker whose short-term speech recording is included in short-term speech data storage 101 .

The noise storage unit 104 stores random vectors representing noise.

The generator parameter storage unit 106 stores generator parameters as shown in FIG. The generator includes an encoder and a decoder, as can be seen from FIG. Both encoder and decoder parameters are stored in generator parameter storage 106 .

The feature extraction unit 103 a extracts feature vectors from the short-time speech data in the short-time speech data storage unit 101 . The feature extraction unit 103b extracts a feature vector from the long-time speech stored in the long-time speech data storage unit 102. FIG. A feature vector is an individually measurable property of an observation. The feature vector is, for example, an i-vector, that is, a fixed-dimensional feature vector extracted from acoustic features such as MFCC described in Non-Patent Document 1.

The generator/discriminator training unit 105 receives feature vectors of short speech segments from the feature extraction unit 103a, receives feature vectors of long speech segments from the feature extraction unit 103b, and receives noise from the noise storage unit 104. FIG. The generator/discriminator training unit 105 iteratively trains the generator and the classifier (not shown in FIG. 1) to obtain “true” (feature vectors are extracted from long speech) or “false” ( A feature vector is generated based on the feature vector from the short speech), and the speaker label to which the feature vector belongs. Each generator and classifier includes an input layer, one or more hidden layers, and an output layer.

In training, if true, the received long speech feature vector is fed to the input layer of the classifier. If "false", the received short speech feature vector is fed to the input layer of the generator. The output layer of the generator is the input layer of the discriminator. Additionally, the "true/false" and speaker labels are provided to the output layer of the classifier. Details of these layers are described later. After training, the generator/discriminator training unit 105 stores the generator parameters in the generator parameter storage unit 106 .

In the feature restoration section 100B, the feature extraction section 103c extracts a feature vector from the short-time speech. The generator 107 receives the noise stored in the noise storage unit 104 and the generator parameters stored in the generator parameter storage unit 106 along with the feature vector. Generator 107 produces robust reconstructed features.

FIG. 5 shows the concept of the architecture of the generator and discriminator. The generator has two neural networks (NN), an encoder NN and a decoder NN. The discriminator has one NN. Each NN includes three types of layers: input layer, hidden layer, and output layer. The hidden layer may include multiple layers. At least between the input and hidden layers, and between the hidden and output layers, there are linear transformations and/or activation functions (transfer functions). The input layer of the encoder NN is the feature vector of short-term speech recordings. The output layer of the encoder NN is speaker coefficients (feature vectors). The input layer of the decoder is the addition or concatenation of noise and speaker coefficients in the output layer of the encoder NN. The output layer of the decoder is the reconstructed feature vector. For the discriminator, the input layer is the feature vector of long-term speech or the recovered feature vector that is the output of the decoder NN. The outputs of the classifier are "true/false" and speaker labels.

In the training unit 100A, the input layer of the encoder NN (feature vector of the short-term speech recording), part of the input layer of the decoder NN (noise), one of the two types of input layers for the discriminator (long-term speech recording ), an output layer of classifiers (outputting “true/false” and speaker labels), resulting in a hidden layer of three NN (encoder, decoder, classifier) parameters, an output layer of the encoder NN (speaker coefficients), the decoder's output layer NN (reconstructed feature vector) are determined. For example, the number of layers in the encoder, decoder and discriminator are 15,15,16.

The evaluation unit of the training unit 100A is provided with encoder parameters, decoder parameters, an input layer of the encoder NN (feature vector of short-time speech), and a part (noise) of the input layer of the decoder NN. The output layer of the reconstructed feature vector) is determined.

In the discriminator, the output layer consists of (2+n) neurons. n is the number of speakers in the training data and 2 is "true/false". In the training section 100A, the neuron can take the value "1" or "0" corresponding to "true/false" and "true speaker label/false speaker label".

In the training section 100A, the generator (encoder and decoder) and discriminator are repeatedly trained with each other. At each iteration, the generator parameters are updated once while the discriminator parameters are fixed, and then the discriminator parameters are updated once while the generator parameters are fixed. For this purpose, various optimization techniques can be applied, such as steepest descent, known as backpropagation, mean squared error, which minimizes a pre-defined cost function as cross-entropy.

For example, the objective function can be expressed as
For generator:

For discriminator:

Value (a) is the objective variable for the generator and value (b) is the objective variable for the discriminator. A is the feature vector of the given short speech. B is the feature vector of the given long speech. Element (c) is noise that models non-speaker variations. G(A, z) is the feature vector generated from the generator. Element (d) is the result of speaker classification, ie, the element for the posterior probability of the speaker. N ^d is the total number of speakers in the training set. Element (e) is for “true/false” feature vector classification. Element (f) is the i-th element of ^Dd . The operators (g) and (h) are the expectation and mean squared error operators, respectively. Constant (i) is a predefined constant. ^yd is the true speaker ID (correct answer).

It can also be expressed as follows.
For generator:

for discriminator

<Operation of Robust Feature Compensator>
Next, the operation of robust feature compensator 100 will be described with reference to the drawings.

The overall operation of robust feature compensator 100 will now be described with reference to FIG. FIG. 6 includes operations of the training section 100A and the feature restoration section 100B. However, this is an example, and the training and feature recovery operations can be performed sequentially, or time intervals can be inserted.

In step A01 (training unit), the generator/discriminator training unit 105 generates the short-time speech and the long-term speech from the same speaker stored in the short-time speech data storage unit 101 and the long-time speech data storage unit 102, respectively. Both the generator and the discriminator are iteratively trained based on temporal speech. Specifically, at each iteration, the discriminator parameters are initially fixed and the generator parameters are updated using the objective function. Then the generator parameters are fixed and the classifier parameters are updated using the objective function. In an iteration, the order of updating the generator and discriminator parameters can be changed. For training, various optimization techniques can be applied, such as steepest descent, known as backpropagation, mean squared error, which minimizes a pre-defined cost function as cross-entropy. The objective function used to update the generator can update the generator so that it can generate recovered feature vectors that the discriminator cannot discriminate. On the other hand, the objective function in updating the discriminator can update the discriminator so that the generated feature vector can be discriminated.

In step A02 (feature restoration unit), the generator 107 uses the generator parameters stored in the generator parameter storage unit 106 to generate a restoration feature vector from the given short speech utterance in the output layer. .

FIG. 7 is a flow chart showing that the generator and discriminator are trained together using short and long speech feature vectors along with noise. FIG. 7 shows the training part in FIG.

First, in step B01, the feature extraction unit 103a reads short-time speech data with a speaker label from the short-time speech data storage unit 101 as the beginning of the training section.

At step B02, the feature extraction unit 103a further extracts a feature vector from the short-time speech.

In step B03, the feature extraction unit 103b reads the long-term speech data with the speaker label from the long-term speech data storage unit 102. FIG.

At step B04, the feature extraction unit 103b further extracts feature vectors from the long-time speech.

In step B<b>05 , the generator/discriminator training unit 105 reads noise data stored in the noise storage unit 104 .

In step B06, the generator/discriminator training unit 105 extracts the feature vector of the short speech with the speaker label transmitted from the feature extraction unit 103a and the long speech with the speaker label transmitted from the feature extraction unit 103b. We train the generator and classifier together using the feature vectors of , as well as the noise.

In step B07, the generator/discriminator training unit 105 generates a generator parameter and a discriminator parameter as a result of training, and stores the generator parameter in the generator parameter storage unit 106. FIG.

The order of B01-B02 and B03-B04 is not limited to the format shown in FIG. 7 and can be changed.

FIG. 8 is a flow chart showing the feature restoration unit 100B.

First, in step C01, the feature extraction unit 103c reads short-time speech data provided via an external device (not shown in FIG. 1).
At step C02, the feature extraction unit 103c extracts feature vectors from the supplied short-time speech data.

In step C03, the generator 107 reads noise data stored in the noise storage unit 104. FIG.

In step C04, the generator 107 reads generator parameters from the generator parameter storage unit 106. FIG.

At step C05, the generator 107 restores the feature vector of the short-time speech to reproduce a robust feature vector.

Note that the order of C03 and C04 can be switched.

Effects of First Embodiment As described above, in the first embodiment, it is possible to improve the robustness of feature vectors of short-time speech. The reason is that the joint training of the generator and the classifier improves each other's performance, and the relationship between the feature vectors of long and short speech in training is learned. As a result, such a NN can generate feature vectors for short-duration speech as robustly as for long-duration speech.

Second Embodiment The robust feature compensator of the second embodiment can use an encoder to provide robust features for short speech segments from raw features of short speech. That is, in this embodiment, the encoder (part of the generator of the GAN trained on short and long speech) is able to generate robust speaker feature vectors for short speech.

<Configuration of Robust Feature Compensator>
In a second embodiment of the present invention, a robust feature compensation apparatus for speaker feature extraction using GAN generators and classifiers is described.

FIG. 9 shows a block diagram of the robust feature compensator 200 of the second embodiment. The robust feature compensator 200 includes a training section 200A and a speaker feature extraction section 200B.

The training unit 200A includes a short-time speech data storage unit 201, a long-time speech data storage unit 202, feature extraction units 203a and 203b, a noise storage unit 204, a generator/discriminator training unit 205, and an encoder parameter storage unit 206. . The speaker feature extraction unit 200B includes a feature extraction unit 203c, an encoding unit 207 as generation means, and a generated feature storage unit 208. FIG. Feature extractors 203a, 203b, and 203c have similar functions.

The short speech data storage unit 201 stores short speech recordings with speaker labels, as shown in FIG.

The long-term voice data storage 202 stores long-term voice recordings with speaker labels, as shown in FIG. Long-term voice data store 202 contains at least one long-term voice recording for each speaker whose short-term voice recording is included in short-term voice data store 201 .

The noise storage unit 204 stores random vectors representing noise.

The encoder parameter storage unit 206 stores encoder parameters. Each encoder parameter is part of the result of the generator/discriminator training unit 205 . The generator (not shown in FIG. 9) consists of an encoder and a decoder, similar to the first embodiment that can be understood from FIG.

The feature extraction unit 203 a extracts features from the short-time voice in the short-time voice data storage unit 201 . The feature extraction unit 203 b extracts features from the long-time voice in the long-time voice data storage unit 202 . A feature is an independently measurable property of an observation. Features are, for example, fixed-dimensional feature vectors extracted from acoustic features such as i-vectors or MFCCs.

The generator/discriminator training unit 205 receives the feature vector of short-time speech from the feature extraction unit 203a, the feature vector of long-time speech from the feature extraction unit 203b, and noise from the noise storage unit 204. FIG. The generator/discriminator training unit 205 selects true (the feature vector is extracted from the long speech) or false (the feature vector is generated based on the feature vector from the short speech), and the feature vector is Iteratively train the generator and classifier (not shown in FIG. 9) to determine the belonging speaker labels. Training details are given in the first embodiment. After training, the generator/discriminator training unit 205 outputs the generator parameters and the discriminator parameters and stores them in the encoder parameter storage unit 206 .

In the speaker feature extraction section 200B, the feature extraction section 203c extracts a feature vector from the short-time speech. The encoding unit 207 receives the noise stored in the noise storage unit 204 and the encoder parameters stored in the encoder parameter storage unit 206 together with the feature vector. Encoder 207 encodes the robust speaker features.

FIG. 10 shows the architectural concept of the generator and discriminator of the second embodiment. The generator has two NNs (encoder NN and decoder NN) and the discriminator has one NN. Each NN includes three types of layers: input layer, hidden layer, and output layer. The hidden layer may include multiple layers. At least between the input and hidden layers, and between the hidden and output layers, there are linear transformations and/or activation functions (transfer functions). The input layer of the encoder NN is the feature vector of short-time speech. The output layer of the encoder NN is the speaker coefficients. The input layer of the decoder is the addition or concatenation of noise and speaker coefficients in the output layer of the encoder NN. The output layer of the decoder is the reconstructed feature vector. For the discriminator, the input layer is the feature vector of long-term speech or the recovered feature vector that is the output of the decoder NN. The outputs of the classifier are "true/false" and speaker labels.

200 A of training parts of 2nd Embodiment are the same as that of the training part of 1st Embodiment mentioned above.

In the evaluation unit, the encoder parameters and the input layer of the encoder NN (short-term speech feature vectors) are provided, resulting in the output layer of the encoder NN (speaker coefficients).

<Operation of Robust Feature Compensator>
Next, the operation of robust feature compensator 200 will be described with reference to the drawings.

The overall operation of robust feature compensator 200 will now be described with reference to FIG. FIG. 11 includes operations of the training section 200A and the speaker feature extraction section 200B. However, this is an example, and the operations of training and speaker feature extraction can be performed sequentially, or time intervals can be inserted.

In step D01 (training section), the generator/discriminator training section 205 generates short-term speech data from the same speaker stored in the short-time speech data storage section 201 and the long-term speech data storage section 202, respectively. Iteratively train both the generator and the discriminator based on the speech and the long speech. Specifically, at each iteration, the discriminator parameters are initially fixed and the generator parameters are updated using the objective function. Then the generator parameters are fixed and the classifier parameters are updated using the objective function. In an iteration, the order of updating the generator and discriminator parameters can be changed. For training, various optimization techniques can be applied, such as steepest descent, known as backpropagation, mean squared error, which minimizes a pre-defined cost function as cross-entropy. The objective function used to update the generator can update the generator so that it can generate recovered feature vectors that the discriminator cannot discriminate. On the other hand, the objective function in updating the discriminator can update the discriminator so that the generated feature vector can be discriminated.

In step D02 (speaker feature extraction unit), the encoding unit 207 uses the encoder parameters stored in the encoder parameter storage unit 206 to extract robust speaker features from a given short-time utterance in the output layer of the encoder. Encode the vector.

FIG. 12 is a flow chart showing that the generator and discriminator are trained together using short and long speech feature vectors along with noise. FIG. 12 shows the training portion of FIG.

First, in step E01, the feature extraction unit 203a reads short-time speech data with a speaker label from the short-time speech data storage unit 201 as the beginning of the training section.

At step E02, the feature extraction unit 203a further extracts a feature vector from the short-time speech.

At step E03, the feature extraction unit 203b reads the long-term voice data with the speaker label from the long-term voice data storage unit 202. FIG.

At step E04, the feature extraction unit 203b further extracts feature vectors from the long-time speech.

At step E05, the generator/discriminator training unit 205 reads the noise data stored in the noise storage unit 204. FIG.

In step E06, the generator/discriminator training unit 205 extracts the feature vector of the short-term speech with the speaker label transmitted from the feature extraction unit 203a and the long-term speech with the speaker label transmitted from the feature extraction unit 203b. We train the generator and classifier together using the feature vectors of , and the noise.

In step E<b>07 , the generator/discriminator training unit 205 trains the generator and the discriminator as a result of the training, and stores the parameters of the encoder (a part of the generator) in the encoder parameter storage unit 206 .

The order of E01 to E02 and E03 to E04 is not limited to the format shown in FIG. 12, and can be changed.

FIG. 13 is a flow chart showing the speaker feature extraction unit 200B.

First, in step F01, the feature extraction unit 203c reads short-time speech data provided via an external device (not shown in FIG. 9).

At step F02, the feature extraction unit 203c extracts a feature vector from the given short-time speech data.

At step F<b>03 , the encoding unit 207 reads noise data stored in the noise storage unit 204 .

At step F<b>04 , the encoding unit 207 reads encoder parameters from the encoder parameter storage unit 206 .

At step F05, the encoding unit 207 encodes the feature vector of short-time speech and extracts a robust speaker feature vector.

Note that the order of F03 and F04 can be switched.

Effects of the second embodiment

As described above, the second embodiment can improve the robustness of feature vectors for short speech. In a first embodiment, robust feature recovery is performed. The same training structure can simultaneously generate robust speaker feature vectors at the output layer of the encoder. Using speaker feature vectors is more straightforward for speaker verification applications.

Third Embodiment The robust feature compensator of the third embodiment uses a generator and a discriminator to extract from the raw features of short-duration speech the bottleneck generated in the last layer of the discriminator. Feature vectors can be used to provide robust features for short speech intervals. That is, in this embodiment, the generator and discriminator of the GAN trained on short and long speech can produce robust bottleneck features for short speech.

<Configuration of Robust Feature Compensator>
In a third embodiment of the present invention, a robust feature compensator for bottleneck feature extraction using GAN generator encoders is described.

FIG. 14 shows a block diagram of the robust feature compensator 300 of the third embodiment. The robust feature compensator 300 includes a training section 300A and a bottleneck feature extraction section 300B.

The training unit 300A includes a short-time speech data storage unit 301, a long-time speech data storage unit 302, feature extraction units 303a, 303b, and 303c, a noise storage unit 304, a generator/discriminator training unit 305, and a generator parameter storage unit 306. , and a discriminator. The bottleneck feature extraction unit 300B includes a feature extraction unit 303c, a generator 308, and a bottleneck feature storage unit 309. FIG. Feature extractors 303a, 303b, and 303c have similar functions.

The short speech data storage 301 stores short speech recordings with speaker labels, as shown in FIG.

Long-term audio data storage 302 stores long-term audio recordings with speaker labels, as shown in FIG. Long-term voice data store 302 contains at least one long-term voice record for each speaker that has a short-term voice record in short-term voice data store 301 .

The noise storage unit 304 stores random vectors representing noise.

The generator parameter storage unit 306 stores generator parameters. The generator (not shown in FIG. 14) consists of encoders and decoders similar to the first embodiment that can be understood from FIG. Therefore, both encoder and decoder parameters are stored in generator parameter storage 306 .

The discriminator parameter storage unit 307 stores parameters of the discriminator (not shown in FIG. 14).

The feature extraction unit 303a extracts features from the short-time voice stored in the short-time voice data storage unit 301. FIG. The feature extraction unit 303 b extracts features from the long-time voice stored in the long-time voice data storage unit 302 . Features are, for example, fixed-dimensional feature vectors extracted from acoustic features such as i-vectors or MFCCs.

The generator/discriminator training unit 305 receives the feature vector of short-time speech from the feature extraction unit 303a, the feature vector of long-time speech from the feature extraction unit 303b, and the noise from the noise storage unit 304. FIG. The generator/discriminator training unit 305 selects true (the feature vector is extracted from the long speech) or false (the feature vector is generated based on the feature vector from the short speech), and the feature vector is We iteratively train the generator and classifier to determine which speaker labels belong. Training details are given in the first embodiment. After training, the generator/discriminator training unit 305 outputs the generator parameters and the discriminator parameters, and stores them in the generator parameter storage unit 306 and the discriminator parameter storage unit 307 .

In the bottleneck feature extraction section 300B, the feature extraction section 303c extracts a feature vector from the short-time speech. Generator 308 receives the noise stored in noise store 304 and the generator parameters stored in generator parameter store 306 along with the feature vector. Generator 308 generates one or more robust bottleneck features representing speaker coefficients.

FIG. 15 shows the architectural concept of the generator and discriminator of the second embodiment. The generator has two NNs (encoder NN and decoder NN) and the discriminator has one NN. Each NN includes three types of layers: input layer, hidden layer, and output layer. The hidden layer may include multiple layers. At least between the input and hidden layers, and between the hidden and output layers, there are linear transformations and/or activation functions (transfer functions). The input layer of the encoder NN is the feature vector of short-time speech. The output layer of the encoder NN is the speaker coefficients. The input layer of the decoder is the addition or concatenation of noise and speaker coefficients in the output layer of the encoder NN. The output layer of the decoder is the reconstructed feature vector. For the discriminator, the input layer is the feature vector of long-term speech or the recovered feature vector that is the output of the decoder NN. The outputs of the classifier are the "true/false" and speaker labels in training, and in the evaluator the original output layer is discarded and the previous last layer is used as the output layer.

The training section of the third embodiment is similar to the training section of the first embodiment.

The evaluation unit includes encoder parameters, decoder parameters, discriminator parameters, an input layer of the encoder NN (feature vector of short-time speech), and a part of the input layer of the decoder NN (noise). An output layer of bottleneck feature vectors) is obtained.

<Operation of Robust Feature Compensator>
Next, the operation of robust feature compensator 300 will be described with reference to the drawings.

The overall operation of robust feature compensator 300 will now be described with reference to FIG. FIG. 16 includes operations of the training section 300A and the bottleneck feature extraction section 300B. However, this is an example, and the training and feature recovery operations can be performed sequentially or interspersed with time intervals.

In step G01 (training section), generator/discriminator training section 305 generates short-term speech and long-term speech from the same speaker stored in short-time speech data storage section 301 and long-time speech data storage section 302, respectively. Both the generator and the discriminator are iteratively trained based on temporal speech. Specifically, at each iteration, the classifier parameters are initially fixed and the generator parameters are updated using the objective function. Then the generator parameters are fixed and the classifier parameters are updated using the objective function. In an iteration, the order of updating the generator and discriminator parameters can be changed. For training, various optimization techniques can be applied, such as steepest descent, known as backpropagation, mean squared error, which minimizes a pre-defined cost function as cross-entropy. The objective function used to update the generator can update the generator so that it can generate recovered feature vectors that the discriminator cannot discriminate. On the other hand, the objective function in updating the discriminator can update the discriminator so that the generated feature vector can be discriminated.

In step G02 (bottleneck feature extraction unit), the generator 308 uses the generator parameters stored in the generator parameter storage unit 306 to restore feature vectors from the given short speech utterance in the output layer. Generate and input to the discriminator. Generator 308 extracts the final hidden layers as robust bottleneck features.

FIG. 17 is a flow chart showing that the generator and discriminator are trained together using short and long speech feature vectors along with noise. FIG. 17 shows the training portion of FIG.

First, in step H01, the feature extraction unit 303a reads short-time speech data with a speaker label from the short-time speech data storage unit 301 as the beginning of the training section.

In step H02, the feature extraction unit 303a further extracts feature vectors from the short-time speech data.

In step H03, the feature extraction unit 303b reads the long-term voice data with the speaker label from the long-term voice data storage unit 302. FIG.

At step H04, the feature extraction unit 303b further extracts feature vectors from the long-time speech data.

In step H 05 , the generator/discriminator training unit 305 reads noise data stored in the noise storage unit 304 .

In step H06, the generator/discriminator training unit 305 extracts the feature vector of the short speech with the speaker label transmitted from the feature extraction unit 303a and the long speech with the speaker label transmitted from the feature extraction unit 303b. We train the generator and classifier together using the feature vectors of , and the noise.

In step H07, generator/discriminator training section 305 generates a generator parameter and a discriminator parameter as a result of training, and stores them in generator parameter storage section 306 and discriminator parameter storage section 307, respectively.

The order of H01-H02 and H03-H04 is not limited to the format shown in FIG. 17, and can be changed.

FIG. 18 is a flow chart showing the bottleneck feature extraction unit 300B.

First, in step I01, the feature extraction unit 303c reads short-time speech data provided from an external device (not shown in FIG. 14).

At step I02, the feature extractor 303c extracts a feature vector from the given short-time speech data.

At step I03, the generator 308 reads noise data stored in the noise storage unit 304. FIG.

At step I04, the generator 308 reads generator parameters from the generator parameter storage unit 306. FIG.

At step I05 , the generator 308 reads discriminator parameters from the discriminator parameter storage unit 307 .

Note that the order of I03 to I05 can be changed.

At step I06, the generator 308 extracts bottleneck features generated in the final layer of the discriminator NN.

Effects of the third embodiment

As described above, the third embodiment can improve the robustness of feature vectors of short speech. As a result, such a NN can generate feature vectors for short-term speech as robustly as features for long-term speech. In a first embodiment, robust feature recovery is performed. Using the same training structure, we can simultaneously generate robust bottleneck features in the output layer of the classifier (the original output layer "true/false" and speaker labels are discarded after the training part).

Note that in all embodiments, the speaker labels in the output layer of the classifier in training can be replaced with emotion labels, language labels, etc. for use in feature compensation for emotion recognition, language recognition, etc. . Similarly, the output layer of the encoder can be modified to represent emotional feature vectors or verbal feature vectors.

Fourth Embodiment A robust feature compensator of the fourth embodiment is shown in FIG. The GAN-based speech feature compensator 500 generates generator and discriminator parameters based on at least one short speech feature vector and at least one long speech feature vector from the same speaker. It includes a generator-discriminator training unit 501 for training a GAN model, and a robust feature compensation unit 502 for compensating short-time speech feature vectors based on short-time speech vectors, generator parameters, and classifier parameters. .

The speech feature compensation device 500 can provide robust feature compensation for short duration speech. The reason is that in order to learn the relationship between the short speech feature vector and the long speech feature vector, the feature vector of the short speech and the feature vector of the long speech are used in the generator and the discriminator. are trained jointly to iteratively improve each other's performance.

<Configuration of information processing device>
FIG. 20 illustrates the configuration of an information processing device 900 (computer) capable of implementing the robust feature compensation device of the embodiment of the present invention. That is, FIG. 20 shows the configuration of a computer (information processing device) that can implement the devices shown in FIGS. indicates

The information processing apparatus 900 shown in FIG. 20 includes the following elements.
- CPU (Central Processing Unit) 901;
- ROM (Read Only Memory) 902;
- RAM (Random Access Memory) 903;
- hard disk 904 (storage device);
- a communication interface 905 with external devices;
- a reader/writer 908 capable of reading and writing data stored in a storage medium 907 such as a CD-ROM (Compact Disc Read Only Memory);
- input/output interface 909;

The information processing device 900 is a general computer connected via a bus 906 (communication line).

The present invention described in the exemplary embodiments above is incorporated into the information processing apparatus 900 shown in FIG. , FIGS. 11 to 13 and FIGS. 16 to 18) are supplied, and the CPU 901 in such hardware reads, interprets, and executes the computer program. be. Computer programs provided to the device may be stored in volatile read/write storage memory (RAM 903 ) or non-volatile storage such as hard disk 904 .

In the above cases, standard procedures can be used to provide computer programs to such hardware. These procedures include, for example, installing a computer program on the device via any of a variety of storage media 907 such as a CD-ROM, or downloading the program from an external source via a communication line such as the Internet. is included. In those cases, the invention can be considered to consist of code forming such a computer program or of a storage medium storing code.

It should be understood that the processes, techniques, and methodologies described and illustrated herein are not limited to or related to any particular apparatus. Implementation is possible using a combination of components. Also, various types of general purpose devices may be used in accordance with the teachings herein. The invention has been described using a few specific examples. However, they are merely examples and are not limiting. For example, the described software can be implemented in various languages such as C/C++, Java, MATLAB and Python. Moreover, other implementations of the inventive technique will be apparent to those skilled in the art.

FIG. 21 is a block diagram showing the essential parts of the audio feature compensator according to the present invention. As shown in FIG. 21, the speech feature compensator 10 extracts a first feature vector extracted from a short speech segment and a long speech segment from the same speaker whose speech is longer and shorter than the short speech segment. Training means 11 (in the embodiment, the generator and a feature extracting means 12 for extracting feature vectors from input short-time speech (implemented by feature extractors 103c, 203c, and 303c in the embodiment). ), and generating means 13 (in embodiments implemented in generators 107, 308 or encoding unit 207) that uses the trained parameters to generate robust feature vectors based on the extracted feature vectors. ) and

As shown in FIG. 22, the generator 21 includes an encoder 211 that inputs a first feature vector and outputs a feature vector, and a decoder 212 that outputs a reconstructed feature vector, and is trained on at least the encoder. Outputting the parameters, the generating means 13 may include an encoding unit that generates a robust feature vector by encoding the feature vector of the input short-duration speech using the trained parameters.

100, 200, 300 robust feature compensator 101, 201, 301 short-time speech data storage unit 102, 202, 302 long-time speech data storage unit 103a, 203a, 303a feature extraction unit 103b, 203b, 303b feature extraction unit 103c, 203c, 303c feature extraction unit 104, 204, 304 noise storage unit 105, 205, 305 generator/discriminator training unit 106 generator parameter storage unit 206 encoder parameter storage unit 306 generator parameter storage unit 107 generator 207 encoding unit 307 discriminator parameter storage unit 108, 208 generated feature storage unit 308 generator 309 bottleneck feature storage unit

Claims (5)

GAN ( training means for training a generator and classifier of a Generative Adversarial Network and outputting trained parameters;
A speech feature compensator, comprising: generating means for generating a restored feature amount based on the feature amount extracted from the input short-time speech using the trained parameters. 2. The speech feature compensator according to claim 1 , wherein said generating means generates a restored feature amount corresponding to said feature amount extracted from an input short-time speech. GAN ( train the generator and discriminator of a Generative Adversarial Network, output the trained parameters,
A speech feature compensation method for generating restored features based on features extracted from input short-term speech using the trained parameters. 4. The speech feature compensation method according to claim 3 , further comprising generating a restored feature quantity corresponding to the feature quantity extracted from the input short-time speech. to the computer,
GAN ( A process of training a generator and a discriminator of a Generative Adversarial Network) and outputting the trained parameters;
A speech feature compensation program for executing a process of generating a restored feature amount based on a feature amount extracted from an input short-time speech using the trained parameters.

Images