JP7754428B2 - Speech recognition device, speech recognition method, and program - Google Patents

Description

The present invention relates to a voice recognition device, a voice recognition method, and a program.

Speech recognition has a wide range of applications and is used in a variety of environments. It is known that when speech containing noise is used for speech recognition, the recognition rate is lower than when clean speech without noise is used. To improve the recognition rate in noisy environments, speech enhancement is sometimes applied to speech recognition systems. With speech enhancement, the speech components of the recorded input speech components are emphasized, thereby relatively reducing the noise components. Noise suppression can be considered a form of speech enhancement.

Missing data speech recognition processing has been proposed as a method of applying speech enhancement to speech recognition. For example, the methods described in Non-Patent Documents 1 and 2 apply an evidence model. The evidence model is a decoding process model that provides statistical information from speech enhancement to speech recognition. The evidence model can be considered a mathematical model for evaluating the expected value of the classification score that gives the recognition result, and is expressed using a probability density function trained to reduce misclassification.

A. C. Morris, J. Baker, and H. Bourlard, “FROM MISSING DATA TO MAYBE USEFUL DATA: SOFT DATA MODELLING FOR NOISE ROBUST ASR”, Proceedings of Workshop Innovation Speech Process, 2001 M. Kuhne, R. Togneri, and S. Nordholm, “Recognition with Applications in Reverberant Multi-Source Environments”, IEEE Transactions on Audio, Speech and Language Processing, vol. 19, No. 2, pp. 372-384, 2011

The proposed evidence model is applied to each frame, resulting in independent processing between frames. As a result, acoustic features representing the acoustic characteristics after processing may become discontinuous between frames, resulting in nonlinear distortion. Unexpected nonlinear distortion can cause a decrease in recognition rate. On the other hand, for speech recognition, the use of end-to-end (E2E) models has been proposed, which produce recognition results consisting of multiple character strings from a single utterance. It is possible to extend the evidence model to multiple frames to accommodate E2E models. Evidence models are high-dimensional statistical models, and simply extending the model would result in an extremely large amount of computation. It is also possible to reduce the amount of computation by sampling acoustic features from the evidence model. To prevent a decrease in recognition rate, it is expected that a sufficient number of samples commensurate with the size of the model will be required.

(1) One aspect of the present embodiment is a speech recognition device including: a speech section processing unit that determines a speech section based on acoustic characteristics of an input speech signal; a speech enhancement unit that determines, for each frame, emphasis features in which speech components are emphasized among acoustic features of the input speech signal using a first model; a hidden state processing unit that determines a hidden state feature series that is a series of hidden state features based on a target feature series that is a series of target features using a second model; a sampling processing unit that samples sample values of the target feature series multiple times using a third model that indicates a probability distribution of the emphasis feature series that is the series of emphasis features within the speech section and a target feature series corresponding to the acoustic feature series that is a series of the acoustic features, causes the hidden state processing unit to calculate sample values of the hidden state feature series for the sample values of the target feature series, and determines an expected value of the hidden state feature series from the sample values of the hidden state feature series; and a speech processing unit that determines a speech content of the speech section based on the expected value of the hidden state feature series using a fourth model.

(2) One aspect of this embodiment is the speech recognition device of (1), wherein the target feature sequence is a weighted sum of the emphasis feature sequence and the acoustic feature sequence, the probability distribution is a probability distribution of a ratio between the emphasis feature sequence and the acoustic feature sequence, and the sampling processing unit may sample a sample value of the ratio using the third model, and synthesize the emphasis feature sequence and the acoustic feature sequence based on the sample value of the ratio to calculate a sample value of the target feature.

(3) One aspect of this embodiment is the speech recognition device of (1), wherein the probability distribution may include a first probability distribution indicating the likelihood that the target feature sequence will be equal to the emphasis feature sequence, and a second probability distribution indicating the likelihood that the target feature sequence will diverge from the emphasis feature sequence.

(4) One aspect of this embodiment is the speech recognition device of (3), wherein the sampling processing unit may determine the first probability distribution based on a posterior probability distribution of the emphasis feature for each frame in the speech section.

(5) One aspect of this embodiment is the speech recognition device of (3) or (4), wherein the sampling processing unit may sample some sample values of the target feature series as sample values of a first type target feature series using the first probability distribution, sample other sample values of the target feature series as sample values of a second type target feature series using the second probability distribution, and determine, as an expected value of the hidden state feature series, an average value of the sample values of the hidden state feature series for the sample values of the first type target feature series and the sample values of the hidden state feature series for the sample values of the second type target feature series.

(6) One aspect of this embodiment is the speech recognition device of (1), wherein the fourth model includes an attention decoder and a connectionist time series classification (CTC) decoder, the attention decoder calculates a first posterior probability for each candidate utterance content for an expected value of the hidden state feature sequence, the CTC decoder calculates a sample value of a second posterior probability for each candidate utterance content for a sample value of the hidden state feature sequence, calculates an expected value of the sample value of the second posterior probability for each candidate utterance content as the second posterior probability, and determines the utterance content based on a score obtained by combining the first posterior probability and the second posterior probability.

(7) One aspect of the present embodiment may be a program for causing a computer to execute: a speech section processing step of determining a speech section based on acoustic characteristics of an input speech signal; a speech enhancement step of determining, for each frame, an emphasis feature in which speech components of acoustic features of the input speech signal are emphasized using a first model; a hidden state processing step of determining, based on a target feature series that is a series of target features using a second model; a sampling processing step of sampling sample values of the target feature series multiple times using a third model that indicates a probability distribution of an emphasis feature series that is a series of the emphasis features within the speech section and a target feature series corresponding to the acoustic feature series that is a series of the acoustic features, instructing the hidden state processing step to calculate sample values of the hidden state feature series for sample values of the target feature series and determine expected values of the hidden state feature series from the sample values of the hidden state feature series; and a speech processing step of determining speech content of the speech section based on the expected values of the hidden state feature series using a fourth model .

(8) One aspect of the present embodiment is a speech recognition method for a speech recognition device , in which the speech recognition device executes: a speech section processing step of determining a speech section based on acoustic characteristics of an input speech signal; a speech enhancement step of determining, for each frame, emphasis features in which speech components of acoustic features of the input speech signal are emphasized using a first model; a hidden state processing step of determining a hidden state feature series that is a series of hidden state features based on a target feature series that is a series of target features using a second model; a sampling processing step of sampling sample values of the target feature series multiple times using a third model that indicates a probability distribution of an emphasis feature series that is a series of the emphasis features within the speech section and a target feature series corresponding to the acoustic feature series that is a series of the acoustic features, instructing the hidden state processing step to calculate sample values of the hidden state feature series for sample values of the target feature series and determine expected values of the hidden state feature series from the sample values of the hidden state feature series; and a speech processing step of determining a speech content of the speech section based on the expected value of the hidden state feature series using a fourth model.

According to one aspect of the present embodiment, it is possible to suppress a decrease in speech recognition rate due to speech enhancement processing.
For example, according to (1), (7), or (8), sample values of multiple target feature sequences corresponding to an emphasis feature sequence and an acoustic feature sequence within an utterance section are obtained, and an expected value of the target feature sequence is obtained from the sample values of the multiple target feature sequences. The speech content is determined based on the expected value of the hidden state feature sequence obtained from the expected value of the target feature sequence. Since the target feature sequence can express the continuity of acoustic features as a change trend within an utterance section, discontinuity in acoustic features between frames due to random sampling can be avoided. Therefore, a decrease in speech recognition rate due to discontinuity in acoustic features can be avoided. Furthermore, sampling the target feature sequence within the utterance section can suppress an increase in processing volume due to high dimensionality.
According to (2), the probability distribution of the target feature sequence can be expressed as the ratio of the emphasis feature sequence to the acoustic feature sequence, thereby reducing the amount of processing required for sampling while maintaining speech recognition accuracy.
According to (3), the first probability distribution can be used to quantify the degree to which an emphasized feature sequence resulting from emphasis of speech components is adopted as a target feature sequence, and the second probability distribution can be used to quantify the degree to which the emphasized feature sequence deviates from the target feature sequence. Speech recognition accuracy can be maintained by sampling that takes into account differences in the reliability of the emphasized feature sequence depending on the usage environment.
According to (4), the accuracy of speech recognition can be maintained by sampling the target feature sequence taking into consideration the error of the emphasized feature for each frame as well as the continuity of the target feature in the speech section.
According to (5), the first probability distribution and the second probability distribution are used for each sample when sampling sample values of a target feature sequence. Addition of the first probability distribution and the second probability distribution for each sample can be avoided, thereby reducing the amount of processing. Furthermore, parallel processing between samples allows for effective use of computational resources.
According to (6), the CTC decoder receives samples of the hidden state feature sequence and outputs sample values of the second posterior probability. By including the CTC decoder processing, which is performed independently of the attention decoder, in the processing for each sample, further utilization of computational resources can be achieved.

1 is a schematic block diagram illustrating an example of the functional configuration of a voice recognition device according to an embodiment of the present invention. 1 is a schematic block diagram illustrating an example of the hardware configuration of a voice recognition device according to an embodiment of the present invention. FIG. 2 is a schematic block diagram showing a first example of the functional configuration of a sampling processing unit, a hidden state processing unit, and a speech processing unit according to the present embodiment. 10 is a graph showing a first example of a third model according to the present embodiment. FIG. 10 is a schematic block diagram showing a second example of the functional configuration of the sampling processing unit, the hidden state processing unit, and the speech processing unit according to the present embodiment. 4 is a flowchart showing a first example of a speech recognition process according to the present embodiment. 10 is a flowchart illustrating a second example of the speech recognition process according to the present embodiment. 1 is a table illustrating an example of CER (Character Error Rate); FIG. 10 is a schematic block diagram showing a functional configuration of a comparative example.

Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. First, an example of the functional configuration of a speech recognition device 10 according to this embodiment will be described. Fig. 1 is a schematic block diagram showing an example of the functional configuration of the speech recognition device 10 according to this embodiment.
A voice signal is input as an input voice signal to the voice recognition device 10 from the microphone 20. The microphone 20 has an electro-acoustic transducer that picks up sound arriving at the microphone 20 and converts the amplitude of the sound pressure into a voltage. The microphone 20 outputs an electrical signal having a signal value indicating the converted voltage to the voice recognition device 10.

The speech recognition device 10 performs speech recognition processing on an input speech signal to estimate the speech content. The speech recognition device 10 estimates a speech section based on the acoustic characteristics of the input speech signal.
The speech recognition device 10 uses a first model to determine, for each frame, an emphasis feature that emphasizes speech components of acoustic features of an input speech signal. The speech recognition device 10 uses a second model to determine a sequence of hidden state features (sometimes referred to herein as a “hidden state feature sequence”) based on a sequence of target features (sometimes referred to herein as a “target feature sequence”). The speech recognition device 10 samples sample values of the target feature sequence multiple times using a third model that indicates a probability distribution of a target feature sequence corresponding to the sequence of emphasis features (sometimes referred to herein as an “emphasis feature sequence”) and the sequence of acoustic features (sometimes referred to herein as an “acoustic feature sequence”), and determines an expected value of the hidden state feature sequence from the sample values of the hidden state feature sequence. The speech recognition device 10 uses a fourth model to determine speech content for each utterance section based on the expected value of the hidden state feature sequence.

The speech recognition device 10 may apply the determined utterance content to various processes, or may be part of an electronic device that has another primary function. The various processes may include, for example, identifying a voice command and executing a process instructed by the identified voice command, document creation, editing, etc. In this application, the term "model" primarily refers to a mathematical model. A speech section is a section that includes the sound of a single utterance, that is, a section in which sound is uttered continuously without interruption. Speech sound refers to a human voice and does not necessarily have to be the sound being spoken at that time. The input voice signal may be recorded in advance or synthesized. The input voice signal may also be input via wired or wireless connection from a device other than the microphone 20.

The speech recognition device 10 includes a control unit 110. The control unit 110 includes a feature analysis unit 112, a speech enhancement unit 114, a speech segment processing unit 116, a sampling processing unit 118, a hidden state processing unit 120, a speech processing unit 122, and a model training unit 130. The speech recognition device 10 illustrated in FIG. 1 performs Missing Data-Automatic Speech Recognition (MD-ASR). In MD-ASR, uncertainty in speech enhancement (SE) is utilized. The uncertainty in speech enhancement is represented by a probabilistic evidence model.

The feature analysis unit 112 acquires an input audio signal input from the microphone 20. The input audio signal is a digital signal representing a time series of signal values sampled at a predetermined sampling frequency. The sampling frequency is, for example, 16 kHz. The feature analysis unit 112 calculates an acoustic feature for each analysis window having a predetermined window length. The analysis window is a section in which the acoustic characteristics of the audio signal are analyzed at one time. For example, a Hanning window can be used as the analysis window. The window length corresponds to the period to be analyzed, i.e., a frame. The window length is, for example, 512 samples. The feature analysis unit 112 moves the analysis window section by a predetermined hop length at regular time intervals. The hop length corresponds to the period by which the analysis window is moved at one time. The hop length may be any positive real number equal to or less than the window length. For example, the hop length is 128 samples. The feature analysis unit 112 can use, as the acoustic feature, features that represent frequency characteristics, such as Short Time Fourier Transform (STFT) parameters and Mel filter bank.
The feature analysis unit 112 outputs the acoustic feature amount calculated for each frame to the speech enhancement unit 114 , the speech segment processing unit 116 and the sampling processing unit 118 .

The speech enhancement unit 114 performs speech enhancement (SE) on the acoustic features input from the feature analysis unit 112 using a first model, and calculates acoustic features in which speech components are emphasized as enhancement features. Emphasizing speech components means relatively emphasizing speech components more than other components. Noise suppression processing may be applied as speech enhancement processing. For example, a deep neural network (DNN) can be applied as the first model. The speech enhancement unit 114 outputs the calculated enhancement features to the sampling processing unit 118. Specific examples of the speech enhancement unit 114 will be described later.

The speech section processing unit 116 determines speech sections based on the acoustic features for each frame acquired from the feature analysis unit 112. The speech section processing unit 116 can use a known voice detection method (VAD: Voice Activity Detection) to detect speech sections. The speech section processing unit 116 determines for each frame whether or not the frame is a speech section. For example, the speech section processing unit 116 uses a preset speech section determination model to determine as a speech section a frame where the speech section probability calculated based on the acoustic features is equal to or greater than a predetermined speech section probability threshold, and determines as a non-speech section a frame where the speech section probability is less than the threshold.

The speech section processing unit 116 can determine a series of non-speech sections that continues for a predetermined number of consecutive frames or more as a non-speech section. The speech section processing unit 116 can also determine a section consisting of one or more frames sandwiched between non-speech sections before and after the series of non-speech sections as a speech section. As a result, even a speech section that includes a temporary silent section can be detected as a series of speech sections without being divided by silent sections.
The speech segment processing unit 116 outputs the determined speech segment to the sampling processing unit 118. Note that the speech segment determination model may be implemented as a part of the first model or the third model.

The speech section processing unit 116 may analyze the acoustic characteristics of the input speech signal independently for each frame, without acquiring acoustic features from the feature analysis unit 112. The speech section processing unit 116 may analyze, for example, power and the number of zero-crossings as acoustic characteristics. The number of zero-crossings is the number of times the signal value in a frame in the time domain changes from a positive value to a negative value, or from a negative value to a positive value. The speech section processing unit 116 may determine as a speech section a frame in which the analyzed power is greater than a predetermined power threshold and the number of zero-crossings is within a predetermined range (e.g., 300 to 1000 times per second), and may determine other frames as non-speech sections.

The sampling processing unit 118 receives the speech feature for each frame from the feature analysis unit 112 and receives the emphasized feature for each frame from the speech enhancement unit 114 .
The sampling processing unit 118 receives an utterance section from the speech section processing unit 116. The sampling processing unit 118 arranges the emphasis features for each frame in the utterance section in that order to form an emphasis feature sequence. The sampling processing unit 118 arranges the speech features for each frame in the utterance section in that order to form a speech feature sequence.

The sampling processing unit 118 determines a probability distribution of a target feature sequence for the emphasis feature sequence and acoustic feature sequence within the speech section according to the third model, and samples the sample values of the target feature sequence N times using pseudo-random numbers according to the probability distribution. The target feature is a value estimated as the realization of the emphasis feature obtained by performing speech emphasis processing on the acoustic feature. This is because speech emphasis processing does not necessarily emphasize only the speech component, depending on whether non-speech components are suppressed excessively or insufficiently.

Therefore, the model learning unit 130 according to this embodiment obtains sample values of N target feature sequences using, as the probability distribution related to the third model, a probability distribution indicating the relationship between a target feature sequence and a set of known emphasis feature sequences and acoustic feature sequences. Expected values based on these sample values are statistically more likely to be actual values related to the target features. N is a predetermined integer equal to or greater than 2. N is, for example, 10 to 200. The third model corresponds to an utterance segment-specific evidence model, which will be described later, and can be regarded as a type of probabilistic evidence model.
In addition, when the emphasis features and the acoustic features are each short-time Fourier transform coefficients, the target features of each frame forming the target feature sequence may be converted into a Mel filter bank in the sampling processing unit 118 or the hidden state processing unit 120.

The sampling processing unit 118 outputs sample values of the target feature sequence to the hidden state processing unit 120 .
The sampling processing unit 118 receives sample values of the hidden state feature sequence as a response to the sample values of the target feature sequence from the hidden state processing unit 120. The sampling processing unit 118 determines an expected value obtained by averaging the sample values of the hidden state feature sequence between samples as the hidden state feature sequence for that utterance section. The sampling processing unit 118 outputs the determined hidden state feature sequence to the utterance processing unit 122. A specific example of the sampling processing unit 118 will be described later.

The hidden state processing unit 120 receives sample values of the target feature sequence from the sampling processing unit 118 for each sample. The hidden state processing unit 120 uses a second model to calculate sample values of the hidden state feature sequence for the sample values of the target feature sequence. The second model may be, for example, a model equivalent to a shared encoder network that forms part of a known CTC (connectionist temporal classification)/attention architecture. A parameter set trained to provide a hidden state feature sequence for a speech feature sequence of known clean speech may be applied to the second model. A specific example of the hidden state processing unit 120 will be described later.

The speech processing unit 122 receives the hidden state feature sequence from the sampling processing unit 118. Using the fourth model, the speech processing unit 122 calculates, for the hidden state feature sequence, a posterior probability indicating the likelihood that each candidate (hypothesis) of speech content in that speech section has been uttered. The speech processing unit 122 searches for the candidate of speech content with the highest calculated posterior probability as the recognition result for that speech section.

The fourth model may have, for example, a CTC/attention architecture. The CTC/attention architecture includes a CTC decoder network and an attention decoder network. The speech content or its candidates is configured using a label string including one or more labels. A label can be a character, a syllable, a word, or any other unit related to the representation of the speech content. The label string may be expressed using text. When searching for a recognition result from a set of speech information candidates, for example, a well-known beam search technique can be used. A specific example of the speech processing unit 122 will be described later.
The control unit 110 may store the speech information, which is the recognition result determined by the speech processing unit 122, or may use it for other processing, or may output it to other devices.

The model learning unit 130 learns the second and fourth models using pre-configured training data. In this disclosure, "model learning" or "learning a model" means determining parameter sets to be used in model-based calculations. The training data includes multiple different data sets, each of which includes known input data and output data and is configured by associating these data sets. In learning a certain model, the model learning unit 130 recurrently updates the parameter sets so that the difference between the calculated value obtained by calculating the input value that constitutes the input data and the output value that constitutes the output data corresponding to that input data is reduced (minimized) for the entire training data. When the difference falls below a predetermined threshold, or when the number of updates reaches a predetermined number, the model learning unit 130 stops model learning at that point and sets the obtained parameter sets in the functional units associated with each model.

The model learning unit 130 learns the second and fourth models simultaneously. Each data set constituting the training data used for learning includes, as input data, speech features derived from a speech signal in a certain speech section, and, as output data, speech information indicating known speech content in that speech section. When learning the second and fourth models, input data derived from a speech signal indicating clean speech may be used. This output data may be represented as a vector value in which the posterior probability for the candidate speech information that provides the correct answer is 1, and the posterior probability for other speech information is 0.

For example, binary cross entropy can be used as a loss function that indicates the magnitude of the difference between the calculated value and the output value. When updating the parameter set, backpropagation based on steepest gradient or stochastic gradient descent, or a variant thereof (e.g., Adam Optimizer), can be used.

The first model used in the voice enhancement process can be trained independently of the second and fourth models used in the voice recognition process. The training of the first model will be described later along with a specific example of the voice enhancement unit 114. Depending on the configuration of the third model described below, training may not be necessary. Some or all of the parameters of the third model may be derived from calculated values obtained during the training process of the first model.

(Example of hardware configuration)
Next, an example of the hardware configuration of the speech recognition device 10 according to this embodiment will be described. Fig. 2 is a schematic block diagram showing an example of the hardware configuration of the speech recognition device 10 according to this embodiment. The speech recognition device 10 may be configured as a speech recognition system including dedicated components (e.g., integrated circuits) forming a set of one or more functional units illustrated in Fig. 1. The speech recognition device 10 may also be configured as a part or the entirety of a general-purpose computer system as a speech recognition system.

The speech recognition device 10 includes, for example, a processor 152, a drive unit 156, an input unit 158, an output unit 160, a ROM (Read Only Memory) 162, a RAM (Random Access Memory) 164, an auxiliary storage unit 166, and an interface unit 168. The processor 152, drive unit 156, input unit 158, output unit 160, ROM 162, RAM 164, auxiliary storage unit 166, and interface unit 168 are interconnected using a bus BS (base line).

The processor 152, for example, reads programs and various data stored in the ROM 162, executes the programs, and controls the operation of the speech recognition device 10. The number of processors 152 in the speech recognition device 10 is not limited to one, but may be multiple. The processor 152 is, for example, a CPU (Central Processing Unit). When there are multiple processors 152, the processing according to this embodiment may be shared among the multiple processors 152. Furthermore, the multiple processors 152 do not necessarily all need to be the same type; some may be different. The multiple processors 152 may include at least one GPU (Graphics Processing Unit) in addition to a CPU. Note that in this embodiment, "executing a program" includes the meaning of executing processing instructed by various instructions (commands) written in the program.

The processor 152 executes a predetermined program to realize some or all of the functions of the above-mentioned control unit 110, such as the feature analysis unit 112, speech enhancement unit 114, speech section processing unit 116, sampling processing unit 118, hidden state processing unit 120, speech processing unit 122, and some or all of the functions of the model learning unit 130.

The storage medium 154 stores various types of data and is, for example, a portable storage medium such as a magneto-optical disk, a flexible disk, or a flash memory.
The drive unit 156 is, for example, a device that reads various data from the storage medium 154 and/or writes various data to the storage medium 154 .

The input unit 158 receives input data from various input source devices and outputs the input data to the processor 152 .
The output unit 160 outputs the output data input from the processor 152 to various devices as output destinations.

The ROM 162 stores, for example, a program to be executed by the processor 152 .
The RAM 164 is used as a main storage medium that functions as a work area for temporarily storing various data and programs used by the processor 152, for example.
The auxiliary storage unit 166 is a storage medium such as a hard disk drive (HDD) or a flash memory.

The interface unit 168 connects to other devices and allows various types of data to be input and output. The interface unit 168 includes, for example, a communication module that connects to a network via wired or wireless means.

(Specific example of a voice enhancement unit)
Next, a specific example of the voice enhancement unit 114 of the voice recognition device 10 according to this embodiment will be described.
The speech enhancement unit 114 uses a preset first model to calculate an enhancement feature (feature vector) s _t for an acoustic feature (noisy observed spectrum) x _t for each frame t. Here, the speech enhancement unit 114 calculates, as a denoised acoustic feature (denoised spectrum) y ^∼ _t , the logarithm of the multiplied value obtained by multiplying the acoustic feature x _t by its soft mask m(x ^∼ _t ) element by element. Symbols such as ~ and ^ added above letters constituting formulas such as formula (1) will be written adjacent to the letters in this document, such as x ^∼ _t and y ^∼ _t .

As exemplified in Equation (1), a prediction error ^{n ty} _may be added to the logarithmic value of the noise-removed acoustic feature y ^∼ _t . In Equation (1), the small circle in the second term on the right side indicates element-by-element multiplication. log(...) indicates the logarithmic value of .... |...| indicates the absolute value of .... ^{x ∼} _t indicates a concatenated vector formed by concatenating the acoustic feature x _t-k of frame t-k to the acoustic feature x _t-k of frame t+k across 2k+1 frames. The soft mask m(x ^∼ _t ) is modeled by a function that takes x ^∼ _t as an input value and gives a natural number between 0 and 1 as an output value for each element. The prediction error _n ^ty is obtained by sampling using pseudorandom numbers according to a multidimensional Gaussian function N(0,Λ _yt ⁻¹ ) as the probability density. Λ _yt ⁻¹ denotes the inverse matrix of the precision matrix Λ _yt . The precision matrix Λ _yt is a diagonal matrix with D rows and D columns having D variances λ _y,t,1 ² , ..., λ _y,t,D ² as diagonal components. D is an integer value indicating the number of dimensions of the enhancement feature s _t . The speech enhancement unit 114 may also calculate the variances λ _y,t,1 ² , ..., λ _y,t,D ² as other part of the output values of the DNN. Note that the number of dimensions F of the acoustic feature x _t and the noise-reduced acoustic features y ^∼ _t each corresponds to the frame length.

Next, the speech enhancement unit 114 calculates the function value of a feature extraction function f(y ^∼ _t ) for the noise-removed acoustic feature y ^∼ _t as an enhancement feature s _t . As exemplified in Equation (2), a prediction error n _t ^s may be added to the function value of the enhancement feature s _t . The prediction error n _t ^s is obtained by sampling using pseudo-random numbers according to a multidimensional Gaussian function N(0,Λ _s,t ⁻¹ ) as a probability density. Λ _st ⁻¹ indicates the inverse matrix of the precision matrix Λ _s,t . The precision matrix Λ _s,t is a diagonal matrix with D rows and D columns having D variances λ _s,t,1 ² , ..., λ _s,t,D ² as diagonal elements. In this embodiment, the soft mask m(x ^∼ _t ) and the feature extraction function f(y ^∼ _t ) are each realized as a part (subset) of a DNN constituting a first model. In addition, the variances λ _y,t,1 ² , ..., λ _y,t,D ² , λ _s,t,1 ² , ..., λ _s,t,D ² may also be calculated as output values from the DNN.

Next, the training of the first model will be described. The model training unit 130 trains the first _model so as to increase (maximize) the log-likelihood log p(s _t |x ^{∼ t) of the probability density function p(s t |x ∼} _{t )} ^of _the enhancement feature s _t with respect to the combined vector x ^∼ _t of the acoustic feature x t. The probability density p(s _t |x ^∼ _t ) is the convolution integral value of the conditional probability p(y ^∼ _t |x ^∼ _t ) of the noise-reduced acoustic feature y ^∼ _t , given the combined vector x ^∼ _t as a condition, and the conditional probability (s _t |y ^∼ _t , x ^∼ _t ) of the enhancement feature s _t with respect to the set of ^the combined vector x ^∼ _t and the noise-reduced acoustic feature y ∼ _t , as shown in Equation (3). Therefore, it is generally difficult to analytically derive the log-likelihood log p(s _t |x ^∼ _t ).

In this embodiment, as shown in Equation (4), the lower limit of the logarithmic likelihood log p(s _t |x ^∼ _t ) is approximated to the conditional variational posterior probability q(y^ _t |x ^∼ _t ) of the noise-reduced acoustic feature y^ _t , where the noise-reduced acoustic feature y^ _t is conditioned on the combined vector x ^∼ _t , and is utilized as the expected value of the logarithmic value of the product of the conditional logarithmic likelihood conditional probability p(y^ _t |x ^∼ _t ) and the conditional probability (s _t |y^ _t , x ^∼ _t ). In Equation (4), E(...) indicates the expected value of.... However, for simplicity, in Equation (4), the noise-reduced acoustic feature y ^{^} _t to which the prediction error n _t ^y is not added is applied instead of the noise-reduced acoustic feature y ∼ _t in Equation (3). This is because noise components other than speech components are also known in the learning stage.

Therefore, the model training unit 130 only needs to be able to train the first model so as to maximize the value defined on the right side of equation (4). More specifically, the model training unit 130 can train the first model so as to minimize the cost function _Jse exemplified in equation (5). The cost function _Jse corresponds to a value obtained by inverting the sign of the sum of the logarithm of the conditional probability p(s _t |y^ _t , x ^∼ _t ), log p _{(s t |y − t | x} ^∼ _t ⁾ _, and the logarithm of the conditional probability of the logarithmic feature ^{y −} _t , conditioned on the combined vector x ^∼ _t . Equation (5) is derived assuming that the conditional variational _posterior probability q(y^ _t |x ^∼ _t ) on the right side of equation (4) is a delta function. The logarithmic feature y ^- _t corresponds to the logarithmic value of the absolute value of the noise-removed acoustic feature y^ _t . The conditional probability logarithm log p(s _t |y ⁻ _t |x ^∼ _t ) facilitates soft mask training. Here, the conditional probabilities p(s _t |y^ _t , x ^∼ _t ) and p(s _t |y ⁻ _t |x ^∼ _t ) may be assumed to be multidimensional Gaussian functions. In this case, the model training unit 130 can define the multidimensional Gaussian function N(f(y^ _t ), _Λ s,t −1 (y^ _t , z ^∼ _t )) by defining the mean value of the trained conditional probability p(s t |y^ _t , x ^∼ _t ) as the function value f(y^ _t ) and determining its variance. Λ _{s,t −1} ^{( y} ^ _t , z ^∼ _t ) denotes the inverse matrix Λ _s,t ⁻¹ (y^ _t , z ^∼ _t ) of the precision matrix. This function value f(y^ _t ) is obtained in a calculation process based on the first model. The model learning unit 130 can calculate the precision matrix Λs _,t by performing LU decomposition on the correlation matrix of the emphasis feature, which is the output from the first model.

(Specific example of sampling processing unit)
Next, a specific example of the sampling processing unit 118 according to this embodiment will be described, including the relationship between the second model and the fourth model.
The sampling processing unit 118 determines the hidden state feature sequence h1 _:T' based on an utterance-wise evidence model as the third model, and can cause the utterance processing unit 122 to calculate the posterior probability of utterance content candidates based on the hidden state feature sequence h1 _:T' . The hidden state feature sequence h1 _:T' can be estimated from the emphasis feature sequence z1 _:T in the utterance section by using the second model forming an encoder network, as shown in equation (6). T' indicates the number of frames of the hidden state feature in the utterance section. The number of frames T' depends on the configuration of the encoder network forming the third model, and may be equal to or less than the number of frames T of the hidden state feature sequence z1 _:T in the utterance section.

Estimation of the hidden state feature sequence h1 _:T' using the utterance-specific evidence model is formulated as shown in equation (7). According to equation (7), the expected value of the hidden state feature sequence _h1:T' (s1 _:T ) for the emphasis feature sequence s1: _T in an utterance section is expressed as the convolution integral value of the hidden state feature sequence _{h1 :T'} (z1 _:T ) for the target feature sequence z1 _:T and the probability distribution ε(z1 _:T ;Θ) by the utterance-specific evidence model for the target feature sequence z1:T (hereinafter sometimes referred to as the "model probability distribution"), where Θ denotes a parameter set of the model probability distribution.

Integrating the target feature sequence in an utterance section is generally difficult because it requires an extremely large amount of calculation. This is because the model probability distribution is defined in a high-dimensional space (D×T dimensions) representing the target feature sequence. The sampling processing unit 118 according to this embodiment applies a Monte-Carlo-like approximation and randomly extracts sample values {z 1 _: T ⁽ⁿ⁾ } _n of the target feature sequence z _1:T multiple times using pseudo-random numbers in accordance with the model probability distribution ε(z _1:T ; Θ).

3, the sampling processing unit 118 causes the hidden state processing unit 120 to estimate sample values {h _1:T' ⁽ⁿ⁾ } _n of the hidden state feature sequence corresponding to sample values of the extracted emphasis feature sequence z _1:T . The sampling processing unit 118 calculates the average value of the sample values {h _1:T' ⁽ⁿ⁾ } _n of the hidden state feature sequence as the expected value E[h _1:T' ] of the hidden state feature sequence. The sampling processing unit 118 outputs the calculated expected value E[h _1:T' ] of the hidden state feature sequence to the utterance processing unit 122, which then estimates the utterance content using the fourth model.

3, when the fourth model includes the attention decoder network 122a and the CTC decoder network 122c, the expected value E[h1 _:T' ] of the hidden state feature sequence is used in both networks. The attention decoder network 122a and the CTC decoder network 122c each calculate the posterior probabilities _patt (c1 _:L |E[ _h1:T' ]) and _pctc ( _c1:L |E[h1 _:T' ]) for each utterance candidate. The utterance content estimation unit 122d of the utterance processing unit 122 then calculates the weighted sum of the logarithmic value of the posterior probability _patt (c1 _:L |E[h1 _:T' ]) and the logarithmic value of the posterior probability _pctc (c1 _:L |E[h1 _:T' ]) as shown in Equation (8) as the speech recognition score J _asr for each utterance candidate. The utterance content estimation unit 122d can estimate, as the utterance content, the candidate utterance content that maximizes the speech recognition score J _asr . In equation (8), w ₁ and w ₂ are each a predetermined real number between 0 and 1, and are normalized so that their sum becomes 1.

In the above example, it is assumed that the expected value E[h1 _:T' ] of the hidden state feature sequence is input to both the attention decoder network 122a and the CTC decoder network 122c, but this is not limiting. As illustrated in Figure 5, the expected value E[h1 _:T' ] of the hidden state feature sequence may be input to the attention decoder network 122a, and the sample value {h1 _:T' ⁽ⁿ⁾ } _n of the hidden state feature sequence may be input to the CTC decoder network 122c. In this case, the CTC decoder network 122c outputs a sample value _pctc (c1 _:L |{ _h1:T' ⁽ⁿ⁾ } _n ) of the posterior probability for each sample and utterance candidate. The utterance content estimation unit 122d can calculate the average value among samples of the posterior probability sample value p _ctc (c _1:L | {h _1:T' ⁽ⁿ⁾ } _n ) as the expected value E[p _ctc (c _1:L | h _1:T' )]. The utterance content estimation unit 122d can calculate the speech recognition score J asr by substituting the expected value E[p _ctc (c _1:L | h _1:T' )] for the posterior probability p _ctc (c _1:L | E[h _1:T' ]) in equation (8). Therefore, the utterance content estimation unit 122d can estimate the utterance content candidate that maximizes the speech recognition score _{J asr as} the utterance content.

5, the estimation of the posterior probability p _ctc by the CTC decoder network using the utterance-specific evidence model is based on the relationship illustrated in equation (9). Equation (9) indicates that the expected value of the posterior probability p ctc (c _1:L | h _{1:T' ) by the CTC decoder network of the label sequence c 1:L that is a candidate for the utterance content of the hidden state feature sequence h 1:T ' in the} utterance section corresponds to the convolution integral value of the posterior probability p _ctc (c _1:L | h 1: _T' (z _1:T )) by the CTC decoder network of the label sequence c _1:L, which is conditioned on the hidden state feature sequence h _1:T' (z _1:T ) for the target _feature sequence z 1:T, and the model probability density distribution ε(z _1:T ; Θ).

Next, an example of a third model according to this embodiment will be described.
In a first example of the third model, the model probability distribution according to the third model is expressed using a latent variable indicating the correspondence between a set of speech feature sequences and emphasis feature sequences and a target feature sequence. Using this latent variable, the model probability distribution is expressed in a lower dimension. More specifically, under the assumption that the target feature sequence is a weighted sum of the emphasis feature sequence and the acoustic feature sequence, the model probability distribution is expressed as a probability distribution in which the ratio between the emphasis feature sequence and the acoustic feature sequence is a latent variable. In this case, the sampling unit 118 extracts a sample value of the ratio from the model probability distribution during sampling. The sampling unit 118 can calculate the weighted sum of the emphasis feature sequence and the acoustic feature sequence as a sample value of the target feature sequence according to the sample value of the extracted ratio.

Fig. 4 is a graph showing the correspondence between the emphasis feature sequence, acoustic feature sequence, target feature sequence, and ratio α in this example. In Fig. 4, s _t,d , u _t,d , and z _t,d represent the elements of the emphasis feature, acoustic feature, and target feature related to frame t and dimension d, respectively. T and D represent the number of frames in the speech section and the number of dimensions of each feature, respectively. α represents the ratio. Each arrow indicates the relationship between the information indicated at its start point and the information indicated at its end point, in that order. That is, Fig. 4 shows that the target feature sequence is given from the entire emphasis feature sequence, the entire acoustic feature sequence, and ratio α.
The model probability distribution ε(z _1:T |α) is expressed as the probability distribution p(α) of the ratio α and the conditional probability distribution ε _U (z _1:T |α) of the target feature sequence z _1:T conditioned on the ratio α, as exemplified in equation (10).

The respective ratios of the enhancement feature sequence and the acoustic feature sequence may be non-negative real values, and their sums may be normalized to 1. In this case, the weighted sum corresponds to a weighted average, and the ratio forming the latent variable can be expressed by a single variable. When one variable α indicates the ratio for the acoustic feature sequence, the ratio for the enhancement feature sequence is determined to be 1-α. The minimum and maximum values forming the range of the ratio α may be set to 0 and 1, respectively, and the probability distribution p(α) may be assumed to be a uniform distribution in which the probability density for the ratio α is 1 and the probability density for other ratios α is 0. In this case, the conditional probability distribution ε _U (z _1:T |α) is expressed as a Dirac's delta function (sometimes referred to as a "delta function" in this application) for the difference value between the weighted sum of the enhancement feature sequence and the acoustic feature sequence from the target feature sequence. Under these assumptions, Equation (10) can be transformed into Equation (11). In equation (11), ε _U denotes a uniform distribution over the range [0, 1]. ε _δ denotes a delta function, which is defined for a scalar value for each frame and dimension. The multiplication across frame t and dimension d shown in equation (11) is performed to obtain a delta function for the entire target feature within the speech section.

Next, a second example of the third model according to this embodiment will be described. The model probability distribution according to the second example has a first probability distribution indicating the likelihood that the target feature sequence will be equal to the emphasis feature sequence, and a second probability distribution which is a probability distribution in which the target feature sequence will diverge from the emphasis feature sequence. The first probability distribution indicates the degree to which the emphasis features obtained by the speech emphasis unit 114 are used as is in the speech recognition process. The second probability distribution indicates the degree to which the emphasis features deviate from the emphasis features obtained by the speech emphasis unit 114. This allows for consideration of the degree to which the emphasis features obtained by the speech emphasis unit 114 can be trusted as true emphasis features.

The model probability distribution may be expressed as a weighted average of the first probability distribution and the second probability distribution. In the model probability distribution ε _p−,U− (z _1:T ; Θ) exemplified in equation (12), the first probability distribution is the product of the posterior distributions p _se (z _t ) of the target feature for each frame t, and is based on the assumption that the posterior probability p _se (z _t ) of the target feature corresponds to the posterior probability p _se (s _t ) of the enhancement feature. As the posterior probability p _se (s _t ), the conditional probability p(s _t | y^ _t , x ^∼ _t ) obtained by learning the first model can be used.

In equation (12), π is a weighting coefficient (mixture weight) for the first probability distribution. 1-π is a weighting coefficient for the second probability distribution. In this example, normalization is performed so that the sum of the weighting coefficients for the first probability distribution and the second probability distribution is 1. The weighting coefficient π is a real value greater than 0 and less than 1. The weighting coefficient π may be determined in advance. A uniform distribution may be applied as the second probability distribution ε _U (z _1:T ). The value range of each target feature that provides a uniform distribution may be determined in advance for each element value.

The model training unit 130 may determine some or all of the various parameters constituting the third model by performing model training. In this case, a mixed signal for a certain speech section is input as input data to the speech recognition device 10, and the probability distribution of the emphasis features derived from the output data obtained from the speech enhancement unit 114 can be derived as the probability distribution of the target features. The mixed signal is created by mixing a known speech signal and a non-speech signal. The probability distribution of the target features can be obtained in the process of training the first model. The acoustic features for the speech signal before mixing are ideal emphasis features, and the emphasis features actually obtained by speech enhancement involve errors and have a statistical distribution.
The model learning unit 130 can then recursively determine various parameters so as to reduce the magnitude of the difference between the probability distribution for each example of the third model and the derived probability distribution. As a loss function that indicates the magnitude of the difference, for example, the Wasserstein metric can be used.

Next, a third example of the third model according to this embodiment will be described. The model probability distribution illustrated in Equation (13) is also expressed as a weighted average of the first probability distribution and the second probability distribution. The first probability distribution is a delta function for the difference between the target feature sequence and the emphasis feature sequence. In other words, the first probability distribution indicates the degree to which the target feature sequence becomes the emphasis feature sequence. ε _δ is defined for scalar values for each frame and dimension, as in the example of Equation (11). The multiplication across frame t and dimension d shown in Equation (13) is due to the fact that the delta function is defined for each element constituting the target feature of each frame in the speech section. Note that the second probability distribution shown in Equation (13) is the same as that shown in Equation (12).

Next, a first example of the speech recognition process according to this embodiment will be described. Fig. 6 is a flowchart showing the first example of the speech recognition process according to this embodiment. The example of Fig. 6 is based on the premise that the speech processing unit 122 has the configuration illustrated in Fig. 3.
(Step S102) The feature analysis unit 112 analyzes the acoustic feature quantity indicating the acoustic characteristics of the input speech signal for each frame.
(Step S104) The speech emphasizing unit 114 performs speech emphasizing processing on the acoustic features acquired for each frame using the first model, and determines emphasized features in which the speech components are emphasized.
(Step S106) The speech section processing unit 116 uses a known voice detection method based on the acoustic characteristics of each frame to determine a speech section consisting of a plurality of frames.

The sampling processing unit 118 constructs an emphasis feature sequence consisting of emphasis features within the determined speech section, and constructs a speech feature sequence consisting of speech features. The sampling processing unit 118 determines a probability distribution of a target feature sequence corresponding to the constructed emphasis feature sequence and speech feature sequence according to a third model. The sampling processing unit 118 repeats the processes of steps S108 and S110 N times, and after the N repetitions are completed, proceeds to the process of step S112. N is a predetermined integer value greater than or equal to 2. In this application, N may be referred to as the "number of samples."

(Step S108) The sampling processing unit 118 samples the sample values of the target feature sequence using the determined probability distribution.
(Step S110) The sampling processing unit 118 causes the hidden state processing unit 120 to calculate sample values of the hidden state feature sequence for sample values of the target feature sequence sampled using the second model.

(Step S112) The sampling processing unit 118 calculates the average value of the sample values of the N hidden state feature sequence as the expected value of the hidden state feature sequence.
(Step S114) The utterance processing unit 122 uses the fourth model to calculate the posterior probability of each utterance candidate with respect to the expected value of the hidden state feature sequence. The utterance processing unit 122 estimates the utterance candidate with the highest calculated posterior probability as the utterance content in the utterance section. Then, the processing in FIG. 6 ends.

Next, a second example of the speech recognition processing according to this embodiment will be described, focusing on the differences from the first example. Regarding the commonalities with the first example, the explanation of the first example will be used unless otherwise specified. Fig. 7 is a flowchart showing the second example of the speech recognition processing according to this embodiment. The example of Fig. 7 is based on the premise that the speech processing unit 122 has the configuration illustrated in Fig. 5.
The process in FIG. 7 includes steps S102 to S112 and steps S122 to S128.

After steps S102 to S106 are completed, the sampling unit 118 determines the probability distribution according to the third model as described above. The sampling unit 118 repeats steps S108, S110, and S122 N times, and after the N repetitions are completed, the processing proceeds to step S112.
(Step S122) The sampling processing unit 118 outputs the sample values of the hidden state feature sequence to the CTC decoder network 122c of the utterance processing unit 122, and calculates the sample values of the posterior probability for each candidate utterance content as CTC posterior probability sample values.

After the process of step S112 is completed, the process proceeds to step S124.
(Step S124) The attention decoder network 122a of the speech processing unit 122 calculates the expected value of the posterior probability for the expected value of the hidden state feature sequence for each candidate utterance content as the attention posterior probability expected value.
(Step S126) The utterance content estimation unit 122d of the utterance processing unit 122 calculates the average value of the CTC posterior probability sample values for each candidate utterance content as the CTC posterior probability expected value.
(Step S128) The speech processing unit 122 calculates the weighted average of the expected attention posterior probability and the expected CTC posterior probability for each candidate utterance content as a speech recognition score. The speech processing unit 122 estimates the candidate utterance content with the highest calculated speech recognition score as the utterance content for the utterance section. Then, the processing in FIG. 7 ends.

The above explanation mainly focuses on the case where the sampling processing unit 118 acquires sample values of N target feature sequences obtained by sampling N times from a model probability distribution. If the model probability distribution is expressed as a weighted average of a first probability distribution and a second probability distribution, the sampling processing unit 118 may sample sample values of the πN-th target feature sequence from the first probability distribution instead of the model probability distribution as a first-type target feature sequence, and sample values of the (1-π)N-th target feature sequence from the second probability distribution as a second-type target feature sequence.

The sampling processing unit 118 causes the hidden state processing unit 120 to acquire a total of N sample values of the hidden state feature sequence for each of the πN sample values of the first type target feature sequence and the (1-π)N sample values of the second type target feature sequence. The sampling processing unit 118 can calculate the average value of the acquired N sample values of the hidden state feature sequence as the expected value of the hidden state feature sequence. When N is sufficiently large, the obtained expected value approximates the average value of the sample values of the hidden state feature sequence based on the sample values of the target feature sequence sampled from the model probability distribution. The calculated expected value of the hidden state feature sequence is used to estimate the speech content in the speech processing unit 122, as described above.

Furthermore, when the speech processing unit 122 has the configuration illustrated in FIG. 5, the sample values of the N hidden state feature sequences may be used by the CTC decoder network 122c to calculate N posterior probability sample values for each candidate utterance content. The average value of the obtained N posterior probability sample values is used as the expected value of the posterior probability.

The processing of steps S108 and S110 illustrated in FIG. 6, or the processing of steps S108, S110, and S122 illustrated in FIG. 7, may be repeated for each sample as described above, or may be performed in parallel across samples. Parallel processing may be shared using multiple computational resources provided by processor 152. The computational resource units may be software-defined or hardware-defined. For example, the repeated processing or parallel processing may be performed by a GPU, and other processing may be performed by a CPU. When parallel processing is performed by a GPU, data related to the processing of one or more predetermined number of samples may be divided using GPU mini-batches. This utilizes the capabilities of the computational resources in the computer system and speeds up processing.

(Experimental Example)
Next, an example of an experiment performed on the speech recognition device 10 according to this embodiment will be described. In order to evaluate the effectiveness of this embodiment, the speech recognition rate according to this embodiment was compared with the speech recognition rate according to other methods.

The experiments used speech data extracted from the Corpus of Spontaneous Japanese (CSJ) and non-speech data extracted from the ProSoundEffects (PSE) sound effects corpus. Approximately 230 hours of speech data from academic presentations was used as training data. Three official CSJ evaluation sets, eval1, eval2, and eval3, were used as test sets. The total training data was 5 hours. Each piece of speech data was divided into utterances. The test set was used for a series of speech recognition processes. The speech recognition processes were performed both with and without speech enhancement. The PSE sound effects corpus includes non-speech data such as environmental sounds, animal calls, and musical sounds. Noise was applied to the training data for the first, second, and fourth speech enhancement models, as well as the test set. Separate evaluation sets were also prepared for the test and training data. The sampling frequency of the speech data was 16 kHz.

Clean and reverberant speech were used as training data for the second and fourth models for speech recognition processing. The reverberant speech was generated by performing a convolution operation on the clean speech using an impulse response measured in a laboratory. By using reverberant speech as training data, the second and fourth models were trained to be robust against reverberation and noise, and were used for evaluation.

A mixture of clean speech and non-speech data was used as training data for the first model for speech enhancement. The mixture data was generated by adding randomly selected non-speech data from the PSE sound effects corpus to each piece of speech data from CSJ. The signal-to-noise ratio (SNR) for the mixture of speech and non-speech data was randomly selected for each piece of speech data to take one of five values: -5, 0, 5, 10, or 15. The test set was also generated by mixing clean speech and non-speech data, similar to the training data for the first model. However, non-speech data that was not used to generate the training data for the first model was mixed with speech data. The SNR was also randomly selected for each piece of speech data to take one of four values: -5, 0, 5, or 10.

ESPnet (End-to-End Speech Processing Toolkit) was used as the second and fourth models for speech recognition. ESPnet is an example of an open-source transformer encoder-decoder type speech recognition model. In ESPnet, the frame length is 512 points and the shift length is 128. An 80-dimensional Mel filter bank is applied as the speech feature to be added to the second model.

The first model for speech enhancement was developed using the PyTorch library. PyTorch is an example of an open-source machine learning library. Neural networks with identical configurations were used to infer the soft mask m and the precision matrices Λ _y,t and Λ _s,t . Each neural network included an 80-dimensional filter bank network, a power activation function, an absolute activation function, concatenation of 32 frames before and after the central frame, layer-wise normalization, three-layer fully connected networks with a sigmoid function, and a dropout layer. The dimensionality of the hidden layer was set to 2048. The sigmoid function was applied to the last layer of the neural network for the mask. However, the last layer of the other neural networks did not apply any functions and simply output the output value from the previous layer.

The first model was trained using Adam optimization, a type of stochastic gradient descent, with gradient clipping applied. The parameter set constituting the first model was updated for each utterance. The learning rate was set to 1.0 × 10 ⁻⁴ . The parameter set was updated 50 epochs. This number of updates yielded the best performance on the evaluation set.

The parameter settings for the fourth model were those described in the publicly known ESPnet CSJ recipe, except for the sampling parameters related to utterance content estimation. Specifically, the beam size was set to 20, and the weighting coefficients for the CTC decoder network, attention decoder network, and language model were set to 0.3, 0.7, and 0.3, respectively. The number of samples N was set to four: 16, 32, 64, and 128, and the weighting coefficient π for the first probability distribution was set to 0.25.

As experimental results, the character error rate (CER) was compiled for each combination of speech enhancement method, test set, and SNR. The CER is equivalent to the ratio of the total number of inserted words, replaced words, and deleted words to the number of correct words. A smaller CER indicates better speech recognition performance. In compiling the experimental results, the character sequence that was the recognition result for each utterance was compared with existing text consisting of character strings to detect the presence or absence of inserted words, replaced words, and deleted words. The total number of words in the test sets eval1, eval2, and eval3 was 115,745.

FIG. 8 is a table illustrating experimental results. Each row indicates a method. "No processing" indicates that no speech enhancement processing was performed. "Clean model" indicates the case where clean speech was used, and rows without "clean model" indicate the case where reverberant speech was used. "Speech enhancement only" indicates the case where enhanced features obtained by speech enhancement were used directly in speech recognition processing. "Frame-specific model" indicates the case where target features obtained using a probabilistic evidence model configured for each frame, as in the comparative example shown in FIG. 9, were used in speech recognition processing. Here, target features were sampled for each frame using the model probability distribution shown in Equation (14). The model probability distribution in Equation (14) is given by the product of the posterior probability p _se (z _t ) of the target features for each frame. This model probability distribution corresponds to the first probability distribution shown in the first term on the right-hand side of Equation (12). As mentioned above, the posterior probability p _se (z _t ) is expressed using a multidimensional Gaussian function.

In the comparative example of Figure 9, the expected value may also be calculated by averaging sample values of hidden state features output from the encoder, or by averaging sample values of the posterior probability or speech recognition score output from the decoder. In the experiment, the sample values of hidden state features were averaged and their expected value was calculated. The calculated hidden state features are input to the decoder.

Returning to Figure 8, "uniform" in this embodiment corresponds to the first example of the third model above, i.e., when the model probability distribution of equation (11) is used. "Delta + uniform" corresponds to the third example of the third model above, i.e., when the model probability distribution of equation (13) is used. "Gaussian + uniform" corresponds to the second example of the third model above, i.e., when the model probability distribution of equation (12) is used.

For the frame-specific model and this embodiment, the experimental conditions include the number of samples N and the type of expected value. "enc" in the "CTC" column indicates that the expected value obtained on the encoder side is used, i.e., the expected value of the hidden state feature sequence is input to the CTC decoder network. "prob" indicates that the expected value obtained on the decoder side is used, i.e., the speech recognition score is calculated using the expected value of the CTC posterior probability based on the sample value of the posterior probability from the CTC decoder network. "Atten" indicates the attention decoder network. In both experiments, "enc," i.e., the expected value of the hidden state feature sequence, was input to the attention decoder network.

In Figure 8, eval1, eval2, and eval3 in each column indicate test sets. For each test set, the CER for four different SNRs is shown. The average column indicates the average CER across test sets and SNRs.

According to the experimental results illustrated in Figure 8, the CER obtained by this embodiment was significantly smaller than the CER obtained by other methods. The CER increased in the following order: this embodiment, speech enhancement only, frame-specific model, and no processing. Among these embodiments, the CER was lowest overall when Gaussian + uniform, the number of samples N was 128, and both CTC and Atten were set to enc. However, when using test set eval1 with an SNR of 10 dB, the CER was lowest when Gaussian + uniform, the number of samples N was 128, and CTC and Atten were set to prob and enc, respectively. When using test set eval3 with an SNR of -5 dB, the CER was lowest when uniform, the number of samples N was 16, and both CTC and Atten were set to enc, at the same rate as when both CTC and Atten were set to enc. For the test set eval3, when the SNR was 10 dB, the CER was lowest when uniform, the number of samples N was 128, and CTC and Atten were set to prob and enc, respectively.

In the CER according to this embodiment, no significant differences were observed between the model probability distribution (Examples 1 to 3), the number of samples N, and the expected value patterns. The differences between these are significantly smaller than the differences with the CER obtained for speech enhancement alone. The lack of significant differences due to the number of samples indicates that sufficient performance can be obtained even with a relatively small number of samples, and that further expansion of the second model does not result in an extreme increase in the amount of computation. Considering that both models share the commonality of using a uniform function for the model probability distribution, this indicates that using a uniform function makes it possible to explain the distribution of target features using speech features, and to effectively correct enhancement features.

Furthermore, the fact that the CER associated with the frame-specific model is higher than the CER associated with speech enhancement alone indicates that frame-specific processing actually has a negative impact on speech recognition. This supports the idea that one of the reasons for this is that the continuity of the target feature sequence is not maintained because target features are sampled independently between frames.

The speech recognition device 10 according to this embodiment may be realized by modifying it as follows: The speech recognition device 10 illustrated in FIG. 1 is separate from the microphone 20, but may be configured to include the microphone 20.
The first model is not limited to the neural network used in the above experiment, and may be configured using other types of learning models, such as a recurrent neural network (RNN).
The second model serving as the encoder and the fourth model serving as the decoder are not limited to transformers, and may be configured using other types of learning models such as RNNs and conformers.

The learning of the first to fourth models may be performed by a device separate from the speech recognition device 10, and the parameter sets of each model obtained by learning may be set in the speech recognition device 10. The parameter sets may be provided in association with a program for realizing the functions of the speech recognition device 10. Furthermore, the model learning unit 130 may be omitted from the speech recognition device 10.

In this embodiment, sound source separation processing may be applied instead of the speech enhancement processing by the speech enhancement unit 114. Sound source separation processing is processing for extracting source-specific components from multiple sound sources from an input speech signal. The processing for extracting a speaker's speech component as a source-specific component by sound source separation processing can also be considered as one form of speech enhancement processing for emphasizing that speech component relative to other components. The speech period processing unit 116 performs speech period detection for each source-specific component, and can simultaneously count the number of sound sources for which a speech period is detected as the number of speakers. Even if the number of counted speakers is two or more, the sampling processing unit 118, hidden state processing unit 120, and speech processing unit 122 only need to function for the source-specific components for each detected speech period.

As described above, the speech recognition device 10 according to this embodiment includes a speech section processing unit 116 that determines a speech section based on acoustic characteristics of an input speech signal, a speech enhancement unit 114 that determines, for each frame, an emphasis feature in which speech components are emphasized among the acoustic features of the input speech signal using a first model, and a hidden state processing unit 120 that determines a hidden state feature sequence that is a sequence of hidden state features based on a target feature sequence that is a sequence of target features using a second model. The speech recognition device 10 also includes a sampling processing unit 118 that samples sample values of a target feature sequence multiple times (e.g., N times) using a third model that indicates a probability distribution of an emphasis feature sequence that is a sequence of emphasis features within an utterance section and a target feature sequence corresponding to the acoustic feature sequence that is a sequence of the acoustic features, and determines an expected value of the hidden state feature sequence from the sample values of the hidden state feature sequence, and a speech processing unit 122 that determines the content of an utterance in the speech section based on the expected value of the hidden state feature sequence using a fourth model.
With this configuration, sample values of multiple target feature sequences corresponding to the emphasis feature sequence and the acoustic feature sequence within an utterance section are obtained, and the expected value of the target feature sequence is obtained from the sample values of the multiple target feature sequences. The speech content is determined based on the expected value of the hidden state feature sequence obtained from the expected value of the target feature sequence. Since the target feature sequence can express the continuity of acoustic features as a change trend within an utterance section, discontinuity in acoustic features between frames due to random sampling can be avoided. Therefore, a decrease in speech recognition rate due to discontinuity in acoustic features can be avoided. Furthermore, sampling the target feature sequence within the utterance section can suppress an increase in processing volume due to high dimensionality.

Alternatively, the target feature sequence may be a weighted sum of the emphasis feature sequence and the acoustic feature sequence, and the probability distribution of the target feature sequence may be a probability distribution of the ratio between the emphasis feature sequence and the acoustic feature sequence. The sampling processing unit 118 may sample a sample value of the ratio using a third model, and calculate a sample value of the target feature by combining the emphasis feature sequence and the acoustic feature sequence based on the sample value of the ratio.
This configuration allows the probability distribution of the target feature sequence to be expressed as the ratio of the emphasis feature sequence to the acoustic feature sequence, thereby reducing the amount of processing required for sampling while maintaining speech recognition accuracy.

Furthermore, the probability distribution of the target feature sequence may have a first probability distribution (e.g., a delta function) that indicates the likelihood that the target feature sequence will be equal to the emphasis feature sequence, and a second probability distribution (e.g., a uniform function) that is a probability distribution that the target feature sequence will diverge from the emphasis feature sequence.
With this configuration, the first probability distribution can be used to quantify the degree to which an emphasized feature sequence resulting from emphasis of speech components is adopted as a target feature sequence, and the second probability distribution can be used to quantify the degree to which the emphasized feature sequence deviates from the target feature sequence. Speech recognition accuracy can be maintained by sampling that takes into account differences in the reliability of the emphasized feature sequence depending on the usage environment.

Furthermore, the sampling processing unit 118 may determine the first probability distribution based on a posterior probability distribution (for example, a multidimensional Gaussian function) of the emphasis feature for each frame in the speech section.
According to this configuration, the accuracy of speech recognition can be maintained by sampling the target feature sequence taking into consideration the error of the emphasized feature for each frame as well as the continuity of the target feature in the speech section.

Furthermore, the sampling processing unit 118 may use the first probability distribution to sample some (e.g., πN) sample values of the target feature series as sample values of the first-type target feature series, and use the second probability distribution to sample other (e.g., (1-π)N) sample values of the target feature series as sample values of the second-type target feature series, and may determine, as the expected value of the hidden state feature series, the average value of the sample values of the hidden state feature series for the sample values of the first-type target feature series and the sample values of the hidden state feature series for the sample values of the second-type target feature series.
According to this configuration, the first probability distribution and the second probability distribution are used for each sample when sampling sample values of a target feature sequence. Addition of the first probability distribution and the second probability distribution for each sample is avoided, thereby reducing the amount of processing. Furthermore, parallel processing across samples allows for effective use of computational resources.

The fourth model may include an attention decoder (e.g., attention decoder network 122a) and a CTC decoder (e.g., CTC decoder network 122c). The attention decoder may calculate a first posterior probability (e.g., attention posterior probability expected value) for each utterance content candidate for an expected value of the hidden state feature sequence, and the CTC decoder may calculate a second posterior probability sample value (e.g., CTC posterior probability sample value) for each utterance content candidate for a sample value of the hidden state feature sequence, and may calculate an expectation of the second posterior probability sample value for each utterance content candidate as the second posterior probability (e.g., CTC posterior probability expected value), and may determine the utterance content based on a score (e.g., a speech recognition score) obtained by combining the first posterior probability and the second posterior probability.
According to this configuration, the CTC decoder receives samples of the hidden state feature sequence and outputs sample values of the second posterior probability. By including the CTC decoder processing, which is performed independently of the attention decoder, in the processing for each sample, it is possible to further improve the utilization of computational resources.

One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to the above, and various design modifications can be made without departing from the spirit of the present invention. In other words, the above disclosure includes specific examples, and various modifications are apparent from the specification, drawings, and claims, so the present invention should not be limited to the scope of the disclosure.

10...Speech recognition device, 20...Microphone, 110...Control unit, 112...Feature analysis unit, 114...Speech enhancement unit, 116...Speech segment processing unit, 118...Sampling processing unit, 120...Hidden state processing unit, 122...Speech processing unit, 130...Model learning unit, 152...Processor, 156...Drive unit, 158...Input unit, 160...Output unit, 162...ROM, 164...RAM, 166...Auxiliary storage unit, 168...Interface unit

Claims (8)

a speech segment processing unit that determines a speech segment based on acoustic characteristics of an input speech signal;
a speech enhancement unit that determines, for each frame, an enhancement feature in which a speech component is enhanced with respect to the acoustic feature of the input speech signal using a first model;
a hidden state processing unit that determines a hidden state feature sequence that is a sequence of hidden state feature values based on a target feature sequence that is a sequence of target feature values using a second model;
sampling sample values of the target feature sequence a plurality of times using a third model indicating a probability distribution of an emphasis feature sequence that is a sequence of the emphasis feature within an utterance section and a target feature sequence that corresponds to the acoustic feature sequence that is a sequence of the acoustic feature ;
causing the hidden state processing unit to calculate sample values of a hidden state feature sequence corresponding to sample values of the target feature sequence;
a sampling processing unit that determines an expected value of the hidden state feature sequence from sample values of the hidden state feature sequence;
an utterance processing unit that determines an utterance content of the utterance section based on an expected value of the hidden state feature sequence using a fourth model. the target feature sequence is a weighted sum of the emphasis feature sequence and the acoustic feature sequence, and the probability distribution is a probability distribution of a ratio between the emphasis feature sequence and the acoustic feature sequence,
the sampling processing unit samples the sample value of the ratio using the third model;
The speech recognition device according to claim 1 , wherein the sample value of the target feature is calculated by combining the sequence of emphasis features and the sequence of acoustic features based on the sample value of the ratio. 2. The speech recognition device according to claim 1, wherein the probability distribution includes a first probability distribution indicating a probability that the target feature sequence is equal to the emphasis feature sequence, and a second probability distribution indicating a probability that the target feature sequence diverges from the emphasis feature sequence. The sampling processing unit
The speech recognition device according to claim 3 , wherein the first probability distribution is determined based on a posterior probability distribution of the emphasis feature for each frame in the speech section. The sampling processing unit
sampling a part of the sample values of the target feature sequence as sample values of a first type target feature sequence using the first probability distribution;
sampling sample values of the other target feature sequences as sample values of a second type target feature sequence using the second probability distribution;
5. The speech recognition device according to claim 3, wherein an average value of sample values of the hidden state feature sequence for sample values of the first type target feature sequence and sample values of the hidden state feature sequence for sample values of the second type target feature sequence is determined as an expected value of the hidden state feature sequence. The fourth model is
It has an attention decoder and a connectionist temporal classification (CTC) decoder,
the attention decoder calculates a first posterior probability for each candidate utterance content with respect to an expected value of the hidden state feature sequence;
the CTC decoder calculates a sample value of a second posterior probability for each candidate utterance content for the sample value of the hidden state feature sequence;
calculating an expectation value of the sample value of the second posterior probability for each candidate utterance content as the second posterior probability;
The speech recognition device according to claim 1 , wherein the speech content is determined based on a score obtained by combining the first posterior probability and the second posterior probability. To the computer
a speech segment processing step of determining a speech segment based on acoustic characteristics of the input speech signal;
a speech enhancement step of determining, for each frame, an enhancement feature in which a speech component is enhanced with respect to the acoustic feature of the input speech signal using a first model;
a hidden state processing step of determining a hidden state feature sequence that is a sequence of hidden state features based on a target feature sequence that is a sequence of target features using a second model;
using a third model indicating a probability distribution of an emphasis feature sequence, which is a sequence of the emphasis feature in the speech section, and a target feature sequence corresponding to the acoustic feature sequence, which is a sequence of the acoustic feature, sampling sample values of the target feature sequence a plurality of times using the third model;
causing the hidden state processing step to calculate sample values of a hidden state feature sequence for sample values of the target feature sequence;
a sampling processing step of determining an expected value of the hidden state feature sequence from sample values of the hidden state feature sequence;
and a speech processing step of determining the speech content of the speech section based on the expected value of the hidden state feature sequence using a fourth model . A speech recognition method in a speech recognition device, comprising:
The speech recognition device
a speech segment processing step of determining a speech segment based on acoustic characteristics of the input speech signal;
a speech enhancement step of determining, for each frame, an enhancement feature in which a speech component is enhanced with respect to the acoustic feature of the input speech signal using a first model;
a hidden state processing step of determining a hidden state feature sequence that is a sequence of hidden state features based on a target feature sequence that is a sequence of target features using a second model;
using a third model indicating a probability distribution of an emphasis feature sequence, which is a sequence of the emphasis feature in the speech section, and a target feature sequence corresponding to the acoustic feature sequence, which is a sequence of the acoustic feature, sampling sample values of the target feature sequence a plurality of times using the third model;
causing the hidden state processing step to calculate sample values of a hidden state feature sequence for sample values of the target feature sequence;
a sampling processing step of determining an expected value of the hidden state feature sequence from sample values of the hidden state feature sequence;
an utterance processing step of determining the utterance content of the utterance section based on the expected value of the hidden state feature sequence using a fourth model.