JP7655798B2 - Speech activity detection device, speech activity detection method, and speech activity detection device program - Google Patents

Description

The present invention relates to a speech segment determination device, a speech segment detection method, and a speech segment detection device program that determine speech segments for each speaker from an audio signal.

In general, in speech recognition, in order to achieve recognition performance that is robust to differences between speakers, a method is adopted in which an acoustic model is adapted to a specific speaker using speaker features calculated for each speaker. However, when speech is composed of utterances from multiple speakers, the problem arises of which section of the input speech should be adapted to which speaker.

Speaker diarization technology has been developed as a technique for determining the speech intervals of each speaker. Speaker diarization is a technique for determining "who spoke and when" for a given audio recording. Speaker features calculated from the speech intervals of each speaker obtained by speaker diarization can be used to perform speaker adaptation of the acoustic model.

Target speaker speech recognition technology has been developed as a technology for recognizing only the speech of a specific speaker, and a representative patent document is Patent Document 1. In Patent Document 1, the speech section of the speaker is extracted from the input speech, and target speaker speech recognition is performed using the speaker features calculated from this speech section.

Meanwhile, technology has been developed to generate speaker features simultaneously with speaker diarization, and a representative non-patent document is Non-Patent Document 1. In Non-Patent Document 1, speaker features are calculated for each speaker's speech interval obtained by speaker diarization.

JP 2021-1988 A (Patent Application No. 2019-116065)

Keisuke Kinoshita, et al. , “Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds,” 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021 (to appear).

When extracting speaker features for each speaker from voice data that contains a mixture of the voices of multiple speakers, one method is to determine the speech periods of all speakers present in the input voice data and calculate speaker features from the speech periods determined for each speaker. However, this method cannot ensure a sufficient amount of speech for the target speaker to be recognized when there are few periods in which only the target speaker is speaking, or when there are only speech periods that overlap with other speakers.

On the other hand, if speaker features are learned independently for a speech recognition system, the learned speaker features cannot be used in existing speech recognition systems. Therefore, it is necessary to build a new speech recognition system.

The objective of the present invention is to provide a speech activity detection device that can recognize the voice of each individual speaker from voice data that contains a mixture of the voices of multiple speakers, in addition to existing voice recognition systems.

The speech activity detection device of one aspect of the present invention is a speech activity detection device having a processor and a memory, and has a speech activity determination unit that receives voice data in which the voices of multiple speakers are mixed, determines the speech activity of each speaker from the voice data, and extracts intermediate speaker features from the voice data that can identify the speaker, and a speaker feature extraction unit that extracts speaker features that can identify the speaker based on the voice data and the intermediate speaker features, and the processor adapts the speaker features input to the voice recognition device to a specific speaker.

According to one aspect of the present invention, existing speech recognition systems can perform individual speech recognition from speech data that contains a mixture of the speech of multiple speakers.

FIG. 1 is a block diagram illustrating an example of a voice activity determination device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of speech segment determination and speaker feature extraction performed by the speech segment determination device according to the embodiment of the present invention. FIG. 2 illustrates an example of a speech segment determination unit according to the embodiment of the present invention. 11 is a flowchart illustrating an example of a process performed by the speech segment determination unit according to the embodiment of the present invention. FIG. 2 illustrates an example of a speaker feature extraction unit according to the embodiment of the present invention. 11 is a flowchart illustrating an example of a process performed by the speaker feature extraction unit according to the embodiment of the present invention. FIG. 2 illustrates an example of learning of the voice activity detection device according to the embodiment of the present invention. FIG. 2 is a diagram illustrating an example of processing performed by the voice utterance period determination device 100.

The following describes an embodiment of the present invention with reference to the attached drawings.

FIG. 1 is a block diagram showing an example of a voice activity detection device 100 according to an embodiment of the present invention.
The voice activity detection device 100 is configured as a computer including a processor 1 , a memory 2 , a storage device 3 , an input device 4 , an output device 5 , and a network interface 6 .

The speech section determination unit 21 and the speaker feature extraction unit 22 are loaded as programs into the memory 2 and executed by the processor 1. The storage device 3 stores data used by each program, and in this embodiment, stores voice data for determination 31.

The input device 4 is composed of a mouse, keyboard, touch panel, etc. The output device 5 is composed of a display, etc. The network interface 6 is connected to a network (not shown).

The processor 1 operates as a functional unit that provides a specific function by processing according to the program of each functional unit. For example, the processor 1 functions as the speech section determination unit 21 by processing according to a speech section determination program. The same applies to other programs. Furthermore, the processor 1 also operates as a functional unit that provides each function of the multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units.

The programs, tables, and other information that realize each function of the speech segment determination device 100 can be stored in a storage device 3, a non-volatile semiconductor memory, a hard disk drive, a solid state drive (SSD), or other storage device, or in a computer-readable non-transitory data storage medium, such as an IC card, an SD card, or a DVD.

FIG. 2 is a diagram showing an example of the process of the voice utterance period determination device 100. As shown in FIG.
In this embodiment, an example is shown in which the determination use voice data 31 is stored in advance in the storage device 3. Here, the determination use voice data 31 is voice data in which voices of a plurality of speakers are mixed.

First, the speech section determination unit 201 receives the judgment voice data 31, and generates the speech section determination result 202 of the speaker included in the judgment voice data 31, and the intermediate speaker feature 203 that can identify the speaker in the judgment voice data 31. The speech section determination unit 201 generates the speech section determination result 202 and the intermediate speaker feature 203 according to the number of speakers included in the voice.

The number of speakers included in the determination speech data 31 may be input by a user of the voice utterance period determination device 100 via the input device 4, or may be automatically estimated.
Next, the speaker feature extraction unit 204 generates speaker features 205 from the judgment speech data 31 and the intermediate speaker features 203 .

Here, the speech segment determination result 202 output by the speech segment determination unit 201 includes a determination as to whether each speaker is present at each time.
Finally, a speech recognizer 206 receives the speaker features 205 and performs speaker specific adaptation.

The speech recognition device 206 can use, for example, a publicly known or well-known method that can adapt to a specific speaker using speaker features that can identify the speaker, such as GMM-HMM (Gaussian Mixture Model, Hidden Markov Model) or DNN-HMM (Deep Neural Network).

FIG. 3 shows an example of components of the speech segment determination unit 21.
The speech segment determination unit 21 includes a speech feature amount calculation unit 10 , an intermediate feature calculation unit 303 , a speech segment determination calculation unit 304 , and an intermediate speaker feature extraction unit 305 .

First, the speech section determination unit 21 receives the determination speech data 31, and the speech feature calculation unit 10 calculates the speech feature 302 of the determination speech data 31.

The intermediate representation calculation unit 303, the speech segment determination calculation unit 304, and the intermediate speaker feature extraction unit 305 are configured, for example, as a neural network, and the intermediate representation calculation unit 303 has an input layer that receives the speech feature 302.

The network consisting of the intermediate representation calculation unit 303, the speech segment determination calculation unit 304, and the intermediate speaker feature extraction unit 305 can be, for example, a transformer or a long short-term memory (LSTM). Alternatively, it may be a DNN.

Next, the intermediate representation calculation unit 303 accepts the speech features 302 and calculates the intermediate representation. Then, the intermediate speaker feature extraction unit 305 accepts the intermediate representation and extracts the intermediate speaker features 203.

Finally, the speech section determination calculation unit 304 outputs the speech section determination result 202 for each time and each speaker included in the determination speech data 31 from the intermediate representation and intermediate speaker features 203, and calculates the existence probability for a one-dimensional variable controlled by, for example, a time index and a speaker index.

4 is a flowchart showing an example of the process performed in the speech segment determination unit 21. This process is the process performed in the speech segment determination unit 201 in FIG.
First, the speech segment determination unit 21 receives the determination-use speech data 31 (step 501), and the speech feature calculation unit 10 calculates the speech feature 302 of the determination-use speech data 31 (step 502).

The speech feature 302 output by the speech feature calculation unit 10 is obtained, for example, by processing the input speech data 31 for determination using FFT (Fast Fourier Transform) and calculating the Mel Frequency Cepstral coefficient (MFCC) based on the results of the FFT processing.

Then, the audio feature calculation unit 10 can calculate the difference in MFCC in units (frames) of a predetermined time interval (e.g., 10 msec) to obtain the audio feature 302. Note that the audio feature 302 of the determination audio data 31 is not limited to the above, and any known or publicly known method may be applied.

Next, the intermediate representation calculation unit 303 receives the speech features 302 and calculates an intermediate representation, and the intermediate speaker feature extraction unit 305 receives the intermediate representation and extracts intermediate speaker features 203 according to the number of speakers included in the judgment speech data 31 (step 503).
Finally, the speech segment decision calculation unit 304 calculates the speaker features 205 from the intermediate representation and the intermediate speaker features 203 (step 504).

FIG. 5 shows an example of components of the speaker feature extraction unit 22.
The speaker feature extraction unit 22 includes a speech feature amount calculation unit 10 , an intermediate representation calculation unit 403 , and a speaker feature generation unit 404 .

First, the speaker feature extraction unit 22 receives the judgment speech data 31, and the speech feature calculation unit 10 calculates the speech feature 402 of the judgment speech data 31.

The intermediate representation calculation unit 403 and the speaker feature generation unit 404 are configured, for example, by a neural network, and the intermediate representation calculation unit 403 has an input layer that receives the speech feature 402. The speaker feature generation unit 404 has an input layer that receives the intermediate representation and intermediate speaker features 203. Note that the intermediate representation calculation unit 303 of the speech segment determination unit 21 may share part or all of the network.

Next, the intermediate representation calculation unit 403 receives the speech features 402 and calculates an intermediate representation.
Finally, the speaker feature generation unit 404 receives the intermediate representation and the intermediate speaker features 203 extracted from the speech segment determination unit 21 , and extracts speaker features 205 .

6 is a flowchart showing an example of processing performed in the speaker feature extraction unit 22. This processing is processing performed in the speaker feature extraction unit 204 in FIG.
First, the speaker feature extraction unit 22 receives the judgment-use speech data 31 (step 601), and the speech feature calculation unit 10 calculates the speech feature 402 of the judgment-use speech data 31 (step 602).

Next, the intermediate representation calculation unit 403 receives the speech features 402 and calculates the intermediate representation. The speaker feature generation unit 404 receives the intermediate representation and the intermediate speaker features 203 extracted from the speech segment determination unit 21, and extracts speaker features 205 according to the number of speakers included in the determination speech data 31 (step 603).

FIG. 7 shows an example of learning of the voice activity detector.
The parameters to be learned here are the intermediate representation calculation unit 303, the speech segment determination calculation unit 304, the intermediate speaker feature extraction unit 305, the intermediate representation calculation unit 403, and the speaker feature generation unit 404. Note that a common intermediate representation calculation unit 801 (see FIG. 8) may be used for the calculation of the intermediate representation.

First, speech segment determination 201 is performed on training speech data input 701, and speech segment determination result 202 and intermediate speaker features 203 are calculated. Then, speaker feature extraction 204 is performed using training speech data input 701 and intermediate speaker features 203 as inputs, and speaker features 205 are calculated.

For the speech segment determination results 202, a speech segment loss calculation 703 is performed for the training speech labels 702. For the intermediate speaker features 203, an intermediate speaker feature loss calculation 704 is performed for the training speech labels 702.

The loss calculated in the speech section loss calculation 703 and the intermediate speaker feature loss calculation 704 may be calculated using a publicly known or well-known method, such as binary cross entropy, which is widely known. Note that the speech section loss is not limited to the above, and other well-known or well-known methods used in classification problems may also be used.

For the speaker features 205, speaker feature loss calculation 705 is performed on the training speaker feature labels 706. For example, the training speaker feature labels 706 can be i-vectors. A publicly known or well-known method can be applied as a method for calculating i-vectors, and is widely known, for example, as described in "Front-end factor analysis for speaker verification." (Dehak, Najim, et al. IEEE Transactions on Audio, Speech, and Language Processing 19.4 (2010): 788-798.). Note that the speaker features 205 are not limited to the above, and other publicly known or well-known methods used in speech recognition systems may also be applied.

The loss calculated by the speaker feature loss calculation 705 may be a publicly known or well-known method, for example, least square error, which is widely known. Note that the speaker feature loss is not limited to the above, and other publicly known or well-known methods used in regression problems may also be used.
Finally, parameter update 707 is performed to obtain updated parameters.

The corresponding parameters of the intermediate representation calculation unit 303, speech segment determination calculation unit 304, intermediate speaker feature extraction unit 305, intermediate representation calculation unit 403, and speaker feature generation unit 404 are replaced with the updated parameters obtained. This completes the learning process.

FIG. 8 is a diagram showing an example of processing performed by the voice utterance period determination device 100.
In this embodiment, an example is shown in which the determination voice data 31 is stored in advance in the storage device 3. Here, the determination voice data 31 is voice data in which voices of a plurality of speakers are mixed, and may include overlapping speech sections.

First, the intermediate representation calculation unit 801 receives the speech data for determination 31 and calculates an intermediate representation 802 that represents features that are effective for speech segment determination. The intermediate representation 802 is used as input to the intermediate speaker feature extraction unit 305, the speech segment determination calculation unit 304, and the speaker feature generation unit 404.

Here, the intermediate representation calculation unit 801 is common to the intermediate speaker feature extraction unit 305, the speech segment determination calculation unit 304, and the speaker feature generation unit 404, but it is also possible to use a different one, such as the intermediate representation calculation unit 303 (see Figure 3) for the intermediate speaker feature extraction unit 305 and the speech segment determination calculation unit 304, or the intermediate representation calculation unit 403 (see Figure 5) for the speaker feature generation unit 404.

Then, the intermediate speaker feature extraction unit 305 receives the intermediate representation 802 and generates intermediate speaker features 203 that can identify the speaker only for the input judgment speech data 31. Here, the intermediate speaker features 203 do not guarantee that the intermediate speaker features 203 of the same speaker generated from each judgment data will be the same for different judgment data containing the same speaker.

Next, the speech segment determination calculation unit 304 receives the intermediate representation 802 and the intermediate speaker features 203, and generates a speech segment determination result 202 according to the number of speakers included in the determination speech data 31.

The speaker feature generator 404 receives the intermediate representation 802 and the intermediate speaker features 203, and generates speaker features 205 that can uniquely identify a speaker for any judgment speech data 31. Here, the speaker features 205 can be used to adapt the speech recognition device 206 to the particular speaker represented by the speaker features 205.

By carrying out the above series of processes, the speech segment determination device 100 performs speech segment determination using the speech segment determination unit 201 while extracting intermediate speaker features 203 from the speech included in the determination speech data 31. Furthermore, the speaker feature extraction unit 204 extracts speaker features based on the intermediate speaker features 203, and generates speaker features 205.

The generated speaker features 205 are then input to a speech recognition device 206 and used for adaptation to a specific speaker. This makes it possible to extract speaker features with high accuracy from sections where the speech of multiple speakers overlap.

In addition, this embodiment may be operated online by dividing the determination speech data 31 into short time intervals and then inputting the data to the speech utterance interval determination device 100. In this case, the speaker features 205 can be used to ensure speaker consistency across the divided time intervals.

According to the above embodiment, since speaker features can be generated by utilizing the voice data of the overlapping sections, it is expected that the accuracy of the speaker features can be improved. In addition, since the voice utterance section determination model is trained to output speaker features incorporated in an existing voice recognition model, the speaker features output by the voice utterance section determination model can be used to adapt the existing voice recognition model to a specific speaker.

REFERENCE SIGNS LIST 1 Processor 2 Memory 3 Storage device 4 Input device 5 Output device 6 Network interface 10 Speech feature amount calculation unit 21 Speech period determination unit 22 Speaker feature extraction unit 303 Intermediate feature calculation unit 304 Speech period determination calculation unit 305 Intermediate speaker feature extraction unit 403 Intermediate representation calculation unit 404 Speaker feature generation unit

Claims (6)

A voice activity detection device having a processor and a memory,
a speech section determination unit that receives speech data in which speeches of a plurality of speakers are mixed, determines speech sections of each speaker from the speech data, and extracts intermediate speaker features from the speech data that enable identification of the speakers;
a speaker feature extraction unit that extracts speaker features capable of identifying the speaker based on the speech data and the intermediate speaker features,
The processor,
Using the speaker features to adapt a speech recognition system to a particular speaker;
The speech segment determination unit
The speech feature calculation unit includes an intermediate representation calculation unit, a speech segment determination calculation unit, and an intermediate speaker feature extraction unit,
The speech feature calculation unit
Calculating a speech feature of the speech data;
The intermediate representation calculation unit:
receiving the speech features and computing an intermediate representation;
The intermediate speaker feature extraction unit
receiving the intermediate representation and extracting the intermediate speaker features according to the number of speakers included in the speech data;
The speech segment determination calculation unit is
outputting a speech segment determination result for each time and each speaker included in the speech data from the intermediate representation and the intermediate speaker features;
The speaker feature extraction unit is
The speech feature calculation unit, the intermediate representation calculation unit, and the speaker feature generation unit are included,
The speaker feature generation unit includes:
receiving the intermediate representation and the intermediate speaker features, and extracting speaker features according to the number of speakers included in the speech data;
Regarding learning of parameters used in the intermediate representation calculation unit, the speech segment determination calculation unit, the intermediate speaker feature extraction unit, and the speaker feature generation unit,
Executing the speech interval determination for the training speech data input, and calculating the speech interval determination result and the intermediate speaker features;
Executing the extraction of the speaker features using the training speech data input and the intermediate speaker features as inputs, and calculating the speaker features;
Execute a speech segment loss calculation for a learning utterance label with respect to the speech segment determination result;
performing an intermediate speaker feature loss calculation for the training utterance labels on the intermediate speaker features;
performing a speaker feature loss calculation on the speaker features against training speaker feature labels;
a speech segment detection device which performs parameter updating to obtain updated parameters, and completes the learning by using the updated parameters to replace the corresponding parameters of the intermediate representation calculation unit, the speech segment judgment calculation unit, the intermediate speaker feature extraction unit, and the speaker feature generation unit. 2. The speech activity detection device according to claim 1,
The input device further includes:
2. The speech activity detection device according to claim 1, wherein the number of speakers included in the voice data is input from the input device. 2. The speech activity detection device according to claim 1,
The speech segment determination unit
A speech activity detection device that determines whether each speaker is present at each time. 2. The speech activity detection device according to claim 1,
The processor,
The speech segment determination unit and the speaker feature extraction unit are regarded as one neural network having the parameters,
a speech section loss for a correct answer of the speech section determination result obtained by the speech section determination unit by executing the speech section loss calculation, an intermediate speaker feature loss for the intermediate speaker features obtained by the speech section determination unit by executing the intermediate speaker feature loss calculation, and a speaker feature loss for the speaker features obtained by the speaker feature extraction unit by executing the speaker feature loss calculation are each reduced. 1. A method for voice activity detection having a processor and a memory, comprising:
a speech interval determination step of receiving speech data in which speeches of a plurality of speakers are mixed, determining speech intervals of each speaker from the speech data, and extracting intermediate speaker features from the speech data that enable identification of the speakers;
a speaker feature extraction step of extracting speaker features capable of identifying the speaker based on the speech data and the intermediate speaker features;
an adapting step for adapting a speech recognition device to a particular speaker using said speaker features;
having
The speech segment determination step includes:
The speech feature calculation step, the intermediate representation calculation step, the speech segment determination calculation step, and the intermediate speaker feature extraction step are included.
The speech feature calculation step includes:
Calculating a speech feature of the speech data;
The intermediate representation calculation step includes:
receiving the speech features and computing an intermediate representation;
The intermediate speaker feature extraction step includes:
receiving the intermediate representation and extracting the intermediate speaker features according to the number of speakers included in the speech data;
The speech segment determination calculation step includes:
outputting a speech segment determination result for each time and each speaker included in the speech data from the intermediate representation and the intermediate speaker features;
The speaker feature extraction step includes:
The speech feature calculation step, the intermediate representation calculation step, and the speaker feature generation step are included,
The speaker feature generating step includes:
receiving the intermediate representation and the intermediate speaker features, and extracting speaker features according to the number of speakers included in the speech data;
Regarding learning of parameters used in the intermediate representation calculation step, the speech segment determination calculation step, the intermediate speaker feature extraction step, and the speaker feature generation step,
Executing the speech interval determination for the training speech data input, and calculating the speech interval determination result and the intermediate speaker features;
Executing the extraction of the speaker features using the training speech data input and the intermediate speaker features as inputs, and calculating the speaker features;
Execute a speech segment loss calculation for a learning utterance label with respect to the speech segment determination result;
performing an intermediate speaker feature loss calculation for the training utterance labels on the intermediate speaker features;
performing a speaker feature loss calculation on the speaker features against training speaker feature labels;
2. A speech segment detection method comprising: performing parameter updating to obtain updated parameters; and using the updated parameters to replace the corresponding parameters in the intermediate representation calculation step, the speech segment determination calculation step, the intermediate speaker feature extraction step, and the speaker feature generation step, thereby completing the learning. A speech activity detection program for causing a computer having a processor and a memory to execute speech activity detection, comprising:
a speech segment determination program that receives speech data in which the speeches of a plurality of speakers are mixed, determines speech segments of each speaker from the speech data, and extracts intermediate speaker features from the speech data that enable identification of the speakers;
a speaker feature extraction program for extracting speaker features capable of identifying the speaker based on the speech data and the intermediate speaker features,
Using the speaker features to adapt a speech recognition system to a particular speaker;
The speech segment determination program includes:
causing the computer to function as a speech feature calculation unit, an intermediate representation calculation unit, a speech segment determination calculation unit, and an intermediate speaker feature extraction unit;
The speech feature calculation unit
Calculating a speech feature of the speech data;
The intermediate representation calculation unit:
receiving the speech features and computing an intermediate representation;
The intermediate speaker feature extraction unit
receiving the intermediate representation and extracting the intermediate speaker features according to the number of speakers included in the speech data;
The speech segment determination calculation unit is
outputting a speech segment determination result for each time and each speaker included in the speech data from the intermediate representation and the intermediate speaker features;
The speaker feature extraction program
causing the computer to function as the speech feature calculation unit, the intermediate representation calculation unit, and the speaker feature generation unit;
The speaker feature generation unit includes:
receiving the intermediate representation and the intermediate speaker features, and extracting speaker features according to the number of speakers included in the speech data;
Regarding learning of parameters used in the intermediate representation calculation unit, the speech segment determination calculation unit, the intermediate speaker feature extraction unit, and the speaker feature generation unit,
Executing the speech interval determination for the training speech data input, and calculating the speech interval determination result and the intermediate speaker features;
Executing the extraction of the speaker features using the training speech data input and the intermediate speaker features as inputs, and calculating the speaker features;
Execute a speech segment loss calculation for a learning utterance label with respect to the speech segment determination result;
performing an intermediate speaker feature loss calculation for the training utterance labels on the intermediate speaker features;
performing a speaker feature loss calculation on the speaker features against training speaker feature labels;
a speech segment detection program for performing parameter updating to obtain updated parameters, and completing the learning by using the updated parameters to replace the corresponding parameters of the intermediate representation calculation unit, the speech segment judgment calculation unit, the intermediate speaker feature extraction unit, and the speaker feature generation unit.

Images