JP7825441B2 - Speech recognition device and program - Google Patents

Description

The present invention relates to a speech recognition device and program.

One of the techniques used for speech recognition processing is the CTC speech recognition model. CTC stands for "Connectionist Temporal Classification." The CTC speech recognition model is a model configured to directly learn the relationship between speech and character strings, without using phonetic symbols or other means. In other words, the CTC speech recognition model is an end-to-end speech recognition model technique. The CTC speech recognition model allows the insertion of the blank character string <blank> into the output symbol string, and allows the same symbol to appear consecutively in the output symbol string. In this way, the CTC speech recognition model matches feature frames with output symbol strings.

Non-Patent Document 1 describes a technique called Intermediate Loss Regularization. Intermediate Loss Regularization is a learning technique that involves providing a means for calculating a CTC loss function in the intermediate layer of a CTC speech recognition model, and weighting and adding the values of the CTC loss functions of multiple layers.

Non-Patent Document 2 describes the Shallow Fusion method. Shallow Fusion is a technique that utilizes a language model trained solely on text data for speech recognition. In the Shallow Fusion method, recognition results are output by linearly interpolating the inference probabilities of the speech recognition model and language model in the logarithmic domain during inference.

Jaesong Lee, Shinji Watanabe, “Intermediate Loss Regularization for CTC-based Speech Recognition,” In Proc. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6224-6228, 2021. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, Yoshua Bengio, “On Using Monolingual Corpora in Neural Machine Translation,” in arXiv:1503.03535v2, 2015.

For example, in Japanese, there are expressions such as "hashi" (bridge) and "hashi" (edge), which are different symbol sequences but have the same or similar pronunciation. The CTC speech recognition model is a model that assumes that the probabilities between output symbols (characters) are independent. In other words, the probability of an output symbol does not depend on the probabilities of other adjacent or nearby symbols. Therefore, for a pronunciation for which there are multiple candidate output symbol sequences, the CTC speech recognition model outputs a symbol sequence based on the pronunciation and symbol sequence obtained as a training result. When such a CTC speech recognition model is made to recognize speech on a topic different from the topic of the text used during training, it tends to produce a symbol sequence based on the training text. For example, when the speech to be recognized is about the weather, the CTC speech recognition model may output "optimal sound" when the recognition result should be "minimum temperature." In other words, the CTC speech recognition model has the problem of outputting a symbol sequence that is inappropriate for the topic of the text being inferred, resulting in recognition errors.

If data pairs of speech and text (correct speech recognition results) on a target topic were available, it would be possible to use that data to train a CTC speech recognition model in advance, but such speech data is not always available in advance. Even if such data were available, annotating the speech is costly. In other words, it is difficult or impossible to adapt the output symbol strings of a CTC speech recognition model in conventional technology to a desired topic.

The present invention was developed based on the above-mentioned problem recognition, and aims to provide a speech recognition device and program that can train a speech recognition model using only the text data, even when paired speech and text data is not available for the topic (domain) to be inferred, and output a symbol string appropriate for the topic (domain) during inference, if only the text is available.

[1] In order to solve the above problem, one aspect of the present invention provides a speech recognition device having a multi-layer model that receives a sequence of acoustic feature frames and outputs an output symbol sequence consisting of a sequence of symbols corresponding to the acoustic feature frames, the device comprising: an encoder unit configured to enable machine learning for each of the multi-layer models; and an adapter unit that receives a symbol sequence belonging to a target domain and outputs features to be input to a final layer model of the encoder unit; intermediate features are passed from a model in a certain layer of the multi-layer model of the encoder unit to a model in the next layer; and the encoder unit is configured to perform machine learning based on the relationship between the acoustic feature frame sequence and the output symbol sequence, and then input the features output by the adapter unit to the final layer model of the encoder unit and adapt the final layer model to the target domain using a first loss that is a loss based on the output from the final layer model.

[2] In one aspect of the present invention, the speech recognition device further includes a pseudo-symbol string generation unit that generates the symbol string to be input to the adapter unit by performing at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on the text belonging to the target region.

[3] In another aspect of the present invention, the speech recognition device further includes a replacement processing unit that replaces characters included in text belonging to the target region with other characters in accordance with an output character probability distribution obtained based on predetermined statistical data, and the pseudo-symbol string generation unit generates the symbol string based on the text belonging to the target region after the replacement processing output by the replacement processing unit.

[4] In another aspect of the present invention, in the above-described speech recognition device, the pseudo-symbol string generation unit determines the number of times the same character is repeated when repeating characters in the text based on statistics regarding the number of times the same character is repeated in an intermediate symbol string, which is a symbol string calculated based on features output from the model of a predetermined layer in the encoder unit that has been trained based on the relationship between the frame string of the acoustic features and the output symbol string, and determines the number of consecutive blank symbols to be inserted between characters in the text based on statistics regarding the number of consecutive blank symbols in the intermediate symbol string.

[5] In another aspect of the present invention, the speech recognition device further includes a symbol string generation unit that generates a symbol string to be input to the adapter unit for training the adapter unit based on the intermediate features output from a model of a predetermined layer of the encoder unit based on the acoustic features, and an adapter unit loss calculation unit that calculates a second loss, which is a loss for the features output by the adapter unit based on the symbol string generated by the symbol string generation unit, and is configured to perform machine learning on the adapter unit using the second loss calculated by the adapter unit loss calculation unit.

[6] In another aspect of the present invention, in the above-described speech recognition device, the adapter loss calculation unit calculates the second loss based on a loss based on the likelihood of a correct output symbol string given the features output from the adapter unit, and a loss representing the difference between the features output from the adapter unit and the features output from a model of the corresponding layer in the encoder unit.

[7] In another aspect of the present invention, the speech recognition device described above aggregates repeated characters in the output symbol string and deletes the blank symbols contained in the output symbol string, thereby outputting the resulting character string as the speech recognition result.

[8] Another aspect of the present invention is that, in the above-described speech recognition device, when the encoder unit adapts the model of the final layer to the target domain using the first loss, the encoder unit also performs adaptation based on a third loss obtained from a symbol sequence output by the multi-layer model of the encoder unit based on the frame sequence of acoustic features and the correct output symbol sequence corresponding to the frame sequence of acoustic features. This aspect uses a second re-training technique described below. According to this aspect, while adapting to the target domain (called the target domain), it is possible to prevent or at least suppress degradation of recognition performance in the domain where the encoder unit was originally trained (called the source domain).

[9] Another aspect of the present invention is a program for causing a computer to function as a speech recognition device, comprising: an encoder unit configured to input a sequence of acoustic feature frames and output an output symbol sequence consisting of a sequence of symbols corresponding to the acoustic feature frames; each of the multiple layer models being capable of machine learning; and an adapter unit configured to input a symbol sequence belonging to a target domain and output features to be input to a final layer model of the encoder unit; intermediate features are passed from a model in a certain layer of the multiple layer model of the encoder unit to a model in the next layer; and the encoder unit is configured to perform machine learning based on the relationship between the acoustic feature frame sequence and the output symbol sequence, and then input the features output by the adapter unit to the final layer model of the encoder unit and adapt the final layer model to the target domain using a first loss that is a loss based on the output from the final layer model.

This invention makes it possible to perform domain adaptation of a speech recognition model using only text data from the target domain.

1 is a block diagram showing a schematic functional configuration of a speech recognition device according to an embodiment of the present invention; 2 is a block diagram showing the functional configuration of the main part of a voice recognition function of the voice recognition device according to the embodiment. FIG. FIG. 2 is a block diagram showing a functional configuration for learning an adapter unit of the speech recognition device according to the embodiment. 10 is a block diagram showing a functional configuration for relearning Encoder _h (third-layer encoder unit) using a first relearning technique in the speech recognition device according to the embodiment. FIG. 10 is a block diagram showing a functional configuration for relearning Encoder _h (third-layer encoder unit) using a second relearning technique in the speech recognition device according to the embodiment. FIG. 10 is a flowchart (1/2) showing the procedure of a process for converting a character string into a pseudo CTC symbol string in the embodiment. 10 is a flowchart (2/2) showing the procedure of a process for converting a character string into a pseudo CTC symbol string in the embodiment. FIG. 2 is a block diagram showing an example of the internal configuration of the voice recognition device according to the embodiment. FIG. 13 is a block diagram showing the functional configuration of a portion for performing replacement processing according to a fifth modified example in the speech recognition device of the embodiment.

Next, one embodiment of the present invention will be described with reference to the drawings. Below, we will first explain the technical aspects that this embodiment is based on, and then explain the configuration unique to this embodiment.

An end-to-end speech recognition model that directly learns the relationship between speech and strings of characters without using phonetic symbols or other methods is a model that converts an input acoustic feature sequence, such as a log-mel spectrogram of length T, into an output symbol sequence (string of characters) of length L. This input acoustic feature sequence x is expressed by equation (1) below. The output symbol sequence y is expressed by equation (2) below.

D in the above equation (1) is the number of dimensions of the acoustic feature x[t] for one frame. V in the above equation (2) is the set of output symbols (vocabulary).

Shallow Fusion (Non-Patent Document 2), a conventional technique, linearly interpolates the inference probability of a speech recognition model and the inference probability of a language model in the logarithmic domain during inference. The recognition result y (hat) obtained by the Shallow Fusion technique is expressed by equation (3) below.

In equation (3), P _AM (y|x) is the inference probability of the output symbol string of the speech recognition model (the probability of the output symbol string y given the input acoustic feature sequence x). Also, P _LM (y) is the inference probability of the output symbol string of the language model (the probability of the output symbol string y). This language model P _LM (y) can be trained using only text data. On the other hand, the speech recognition model P _AM (y|x) is trained using paired data of speech and text.

A feature of this embodiment is that it is configured to train a speech recognition model P _AM (y|x) instead of a language model P _LM (y) using only text data. In other words, the speech recognition device of this embodiment adapts a speech recognition model to topics contained in the text data used for training.

Next, an overview of the CTC speech recognition model will be described. The CTC speech recognition model itself is a method belonging to existing technology. The CTC speech recognition model outputs one symbol for each frame of input features. The difference between the frame length of the features and the sequence length between output symbols is absorbed by allowing the insertion of blank symbols <blank> and the continuation of output symbols (characters). As an example, when associating the symbol string "weather" with four frames of acoustic features, there are the following 17 alignments. Note that here, the blank symbols <blank> are replaced with "_". Furthermore, the separators between symbols are represented by slashes.
01: _/_/Heaven/Ki 02: _/Heaven/_/Ki 03: _/Heaven/Ki/_
04: Heaven/_/_/Ki 05: Heaven/_/Ki/_
06: Heaven/Air/_/_
07: _/Heaven/Heaven/Ki 08: Heaven/_/Heaven/Ki 09: Heaven/Heaven/_/Ki 10: Heaven/Heaven/Ki/_
11: _/Heaven/Ki/Ki 12: Heaven/_/Ki/Ki 13: Heaven/Ki/_/Ki 14: Heaven/Ki/Ki/_
15: Heaven/Ki/Ki/Ki 16: Heaven/Heaven/Ki/Ki 17: Heaven/Heaven/Heaven/Ki

The symbol string output by the CTC speech recognition model as an inference result can be converted into a recognition result by deleting blank symbols <blank> and compressing consecutive identical symbols into a single character. In other words, each of the 17 alignments above is converted into the recognition result "weather". However, when the correct character is "ii" in "ii tenki" (good weather), for example, and the correct recognition result contains consecutive identical characters, a blank symbol <blank> must be inserted between the "ii" characters.

The training of the CTC speech recognition model is as follows. In the recognition process using the CTC speech recognition model, the probability of alignment a of an output symbol string is calculated for a feature frame string h extracted from a neural network. Here, the feature frame string h is expressed by the following equation (4). Also, the alignment a is expressed by the following equation (5). Note that T _h in equations (4) and (5) is the length of the string.

The alignment a of the output symbol sequence given the input feature frame sequence h is expressed by equation (6) below.

The sum of the probabilities of all possible alignments of the output symbol string y is calculated using equation (7) below:

β ⁻¹ (y) in equation (7) is the set of all possible alignments for the output symbol string y and feature string h. The CTC loss function L is calculated by taking the logarithm of the above P(y|h) using the following equation (8).

The CTC speech recognition model is trained by updating the values of the internal parameters of the neural network to minimize the loss function value L above.

Each of the functions of the speech recognition device of this embodiment can be realized, for example, by a computer and a program. Furthermore, each of these functions has a storage means as needed. The storage means is, for example, a program variable or memory allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk drive or solid-state drive (SSD) may also be used as needed. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

The speech recognition device 1 of this embodiment is configured based on Intermediate CTC (Non-Patent Document 1). The outline of the speech recognition model in this embodiment is as follows. The speech recognition model in this embodiment is an improvement on the open source Espnet (https://github.com/espnet/espnet) provided on Github. In Intermediate CTC, the value of the CTC loss function is calculated even in the intermediate layers of the neural network, and the sum of the weighted values for each layer is used as the final loss function for model training.

1 is a block diagram showing a schematic functional configuration of a speech recognition device 1 according to this embodiment. As shown in the figure, the speech recognition device 1 includes a speech supply unit 10, a convolutional neural network unit 20, a first-layer encoder unit 30 (Encoder _l 30), a second-layer encoder unit 40 (Encoder _m 40), a third-layer encoder unit 50 (Encoder _h 50), a target region text supply unit 110, a pseudo-CTC symbol string generator 120, an adapter unit 130, a CTC symbol string generator 210, a mean squared error loss calculator 220, and a CTC loss calculator 230.

The speech supply unit 10 supplies speech data to be subjected to recognition processing by the speech recognition device 1. The speech supply unit 10 passes the speech data to the convolutional neural network unit 20. The speech supply unit 10 supplies, as speech data, a sequence of acoustic features represented by, for example, a vector.

The convolutional neural network unit 20 performs convolution processing on the audio data passed from the audio supply unit 10. The convolutional neural network unit 20 passes the feature value (h ₀ described later) obtained as a result of the convolution processing to the first layer encoder unit 30.

The first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50 are a three-layer encoder. Each of the first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50 is implemented using a self-attention neural network. The speech recognition device 1 has a function (CTC loss calculation unit) for calculating losses for the outputs of each of the first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50, but this function is omitted in this diagram. The loss calculation function will be explained in Figure 2.

The first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50, which are configured with models in multiple layers, may be collectively referred to as the "encoder unit." The encoder unit has models in multiple layers that input a sequence of acoustic feature frames and output an output symbol sequence consisting of a sequence of symbols corresponding to the frames of the acoustic features, and each of the models in the multiple layers is configured to be capable of machine learning. Intermediate features (h _l and h _m , described below) are passed from a model in a certain layer to a model in the next layer in the multiple layer models of the encoder unit. Note that this encoder unit is configured to perform machine learning based on the relationship between the frame sequence of the acoustic features and the output symbol sequence, and then input the features output by the adapter unit to the final layer model (third-layer encoder unit 50) of the encoder unit, and adapt the final layer model (third-layer encoder unit 50) to the target domain using a first loss, which is a loss based on the output from the final layer model (third-layer encoder unit 50).

The first layer encoder unit 30, also called Encoder _l 30, calculates a feature h _l based on the feature h ₀ received from the convolutional neural network unit 20 and passes it to the second layer encoder unit 40. The second layer encoder unit 40, also called Encoder _m 40, calculates a feature h _m based on the feature h _l received from the first layer encoder unit 30 and passes it to the third layer encoder unit 50. The third layer encoder unit 50, also called Encoder _h 50, calculates a feature h _h based on the feature h _m received from the second layer encoder unit 40 and outputs it.

Each of the feature quantities h ₀ , h ₁ , h _m , and h _h is a series of vectors.

The target domain text supply unit 110 supplies text data for adapting the Encoder _h 50. The target domain text supply unit 110 supplies text for the target domain so that the Encoder _h 50 outputs text that matches the target domain during inference. The target domain text supply unit 110 passes the text for the target domain to the pseudo-CTC symbol string generation unit 120.

The pseudo-CTC symbol string generation unit 120 generates a pseudo-CTC symbol string based on the text string passed from the target area text supply unit 110. The pseudo-CTC symbol string generation unit 120 is also simply referred to as the "pseudo-symbol string generation unit." The method for generating a pseudo-CTC symbol string will be described in detail later with reference to flowcharts, etc. Multiple CTC symbol strings may correspond to one text string. In other words, the pseudo-CTC symbol string generation unit 120 outputs multiple pseudo-CTC symbol strings corresponding to one text string. In other words, the pseudo-CTC symbol string generation unit 120 converts one text string into multiple pseudo-CTC symbol strings. The pseudo-CTC symbol string generation unit 120 can generate multiple pseudo-CTC symbol strings corresponding to each text string included in the text data supplied by the target area text supply unit 110.

The pseudo-CTC symbol string generation unit 120 generates a symbol string to be input to the adapter unit 130 by at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on text belonging to a target domain. The pseudo-CTC symbol string generation unit 120 may determine the number of times the same character is repeated when repeating characters in the text, based on statistics regarding the number of times the same character is repeated in an intermediate symbol string, which is a symbol string calculated based on features output from the model of a predetermined layer in the encoder unit that has been trained based on the relationship between the frame sequence of the acoustic features and the output symbol string. The pseudo-CTC symbol string generation unit 120 may also determine the number of consecutive blank symbols to insert between characters in the text based on statistics regarding the number of consecutive blank symbols in the intermediate symbol string.

When adapting the Encoder _h ·50, the adapter unit 130 converts each pseudo-CTC symbol string passed from the pseudo-CTC symbol string generation unit 120 into a feature to be input to the Encoder _h ·50. In other words, the adapter unit 130 inputs a symbol string (pseudo-CTC symbol string) belonging to a target region, and outputs, based on that symbol string, a feature to be input to the final layer model (Encoder _h ·50) of the encoder unit.

The adapter unit 130 is realized using a self-attention neural network and is configured to be capable of learning. In other words, the adapter unit 130 learns. When training the adapter unit 130, the adapter unit 130 generates features similar to those output by the Encoder _m ·40 based on a symbol string passed from a CTC symbol string generation unit 210 (described later). As a result of training the adapter unit 130, it is expected that the features output by the adapter unit 130 will approach the features output by the Encoder _m ·40. When training the adapter unit 130, parameters of the neural network inside the adapter unit 130 are adjusted based on losses calculated by a mean squared error loss calculation unit 220 and a CTC loss calculation unit 230 (described later). In other words, the adapter unit 130 is configured to perform machine learning using a "second loss" (described later).

When training the adaptor unit 130, the CTC symbol string generation unit 210 obtains alignment a _l (hat) based on the feature h _l output from the Encoder _l 30. The CTC symbol string generation unit 210 passes the obtained a _l (hat) to the adaptor unit 130. In other words, the CTC symbol string generation unit 210 generates a symbol string to be input to the adaptor unit 130 for training the adaptor unit 130 based on intermediate features output from a model of a predetermined layer (Encoder _l 30) of the encoder unit, based on acoustic features. The CTC symbol string generation unit 210 is also simply referred to as a "symbol string generation unit."

The mean squared error loss calculation unit 220 and the CTC loss calculation unit 230 each calculate a loss related to the feature value h _m (hat) output from the adaptor unit 130 when training the adaptor unit 130. In other words, the mean squared error loss calculation unit 220 and the CTC loss calculation unit 230 are each part of the function of the adaptor unit loss calculation unit that calculates the loss for training the adaptor unit 130. Details of the losses calculated by the mean squared error loss calculation unit 220 and the CTC loss calculation unit 230 will be described later. The losses calculated by the mean squared error loss calculation unit 220 and the CTC loss calculation unit 230 are used to update the parameters of the neural network inside the adaptor unit 130.

[Voice recognition function]
2 is a block diagram showing the functional configuration of the main parts of the speech recognition function of the speech recognition device 1 according to this embodiment. As shown in the figure, the speech recognition device 1 includes a convolutional neural network unit 20, an Encoder _l 30, a CTC loss calculation unit 32, an Encoder _m 40, a CTC loss calculation unit 42, an Encoder _h 50, and a CTC loss calculation unit 52.

As shown in the figure, the speech recognition model is configured to have four neural networks: Convolution (convolutional neural network unit 20), Encoder _l 30 (low-level encoder), Encoder _m 40 (mid-level encoder), and Encoder _h 50 (high-level encoder). Each of these four neural networks is configured with multiple layers. Note that Encoder _l 30, Encoder _m 40, and Encoder _h 50 are each self-attention networks. This speech recognition model calculates CTC loss function values L _l , L _m , and L _h from the outputs of Encoder _l 30, Encoder _m 40, and Encoder _h 50, respectively. The feature value _h0 , which is the output of the convolution, is as shown in the following equations (9) and (10).

In equation (9), _Th is the sequence length of the feature. _Dh is the number of dimensions of each vector included in _h0 . As shown in equation (10), _h0 is the output of convolution when an acoustic feature x is input.

The output h _l from Encoder _l 30, the output h _m from Encoder _m 40, and the output h _h from Encoder _h 50 are as shown in equation (11) below, and are calculated by equations (12), (13), and (14), respectively.

That is, the feature amount _h1 is the amount output by the Encoder _l 30 when the feature amount _h0 is input. The feature amount _hm is the amount output by the Encoder _m 40 when the feature amount _h1 is input. The feature amount _hh is the amount output by the Encoder _h 50 when the feature amount _hm is input.

The loss function values _Ll , _Lm , and _Lh for the outputs from Encoder _l 30, Encoder _m 40, and Encoder _h 50 are calculated by the following formulas (15), (16), and (17), respectively. The CTC loss calculation unit 32 calculates the loss _Ll based on the feature value _hl output from Encoder _l 30. The CTC loss calculation unit 42 calculates the loss _Lm based on the feature value _hm output from Encoder _m 40. The CTC loss calculation unit 52 calculates the loss _Lh based on the feature value _hh output from Encoder _h 50.

The objective function in the base model is calculated, for example, as the average of the loss function values _Ll , _Lm , and _Lh , using the following equation (18).

That is, in the Intermediate CTC that is the premise of this embodiment, learning is performed by updating the internal parameters of the neural network so as to minimize Loss ₁ calculated by equation (18).

Next, a configuration for retraining a speech recognition model using text data on a target topic in the speech recognition device 1 of this embodiment will be described. In this embodiment, a text string is converted into a pseudo-CTC symbol string, and the resulting CTC symbol string is converted into a feature frame by a neural network adapter (described later). In this embodiment, this feature frame is used to retrain the Endocer _h of the intermediate CTC speech recognition model (the model on which this embodiment is based) described above, thereby adapting the neural network to the target topic.

[Adapter Learning]
3 is a block diagram showing a configuration for training the adapter unit 130 (also simply referred to as "Adapter") of the speech recognition device 1 according to this embodiment. The Adapter plays a role in converting pseudo-CTC symbol strings into features to be input to the Encoder _h 50 during re-training of the speech recognition model, which will be described later. As shown in the figure, the speech recognition device 1 includes a convolutional neural network unit 20, an Encoder _l 30, an Encoder _m 40, an Encoder _h 50, the adapter unit 130, a CTC symbol string generation unit 210, a mean squared error loss calculation unit 220, and a CTC loss calculation unit 230.

The functions of the components shown in Fig. 3 have been described above. That is, the CTC symbol string generation unit 210 generates an alignment a _l (hat) based on the feature h _l output from the Encoder _l 30. The adapter inputs the alignment a _l (hat) and outputs an estimated feature value h _m (hat). The adapter unit 130 is trained based on the losses calculated by the mean squared error loss calculation unit 220 and the CTC loss calculation unit 230. Through training, the adapter unit 130 is expected to output a feature close to the feature h _m output by the Encoder _m 40.

Here, the details of the training of the neural network Adapter will be explained. The speech recognition device 1 inputs the acoustic feature x to the trained Intermediate CTC speech recognition model (FIG. 3), and calculates the feature _hm using the following equations (19) and (20).

The process of calculating h _m is the same as the calculations already explained in equations (10), (12), and (13). The alignment a _l (hat) with the highest likelihood for the feature h _l is calculated by the following equation (21).

The speech recognition device 1 trains the adapter so that the CTC alignment sequence approaches the output feature frame _hm of the encoder _m . In other words, the speech recognition device 1 updates the internal parameters of the neural network of the adapter. When the alignment _al (hat) is input to the adapter, the output _hm (hat) is calculated by the following equation (22).

The speech recognition device 1 trains the Adapter so as to bring _hm (hat) in equation (22) closer to _hm . Specifically, the speech recognition device 1 updates the internal parameters of the neural network of the Adapter based on the loss function value _Lm calculated by the following equation (23) and the loss function value _Lmse calculated by equation (24). The CTC loss calculation unit 230 calculates the loss Lm based on the likelihood of y, which is the correct output symbol string, given _hm (hat), which is the output from the Adapter. The mean square error loss calculation unit 220 calculates the loss _Lmse , which represents the difference between _hm (hat), which is the output from the Adapter, and the feature, which is the output from Encoder _m ·40 (the model of the layer corresponding to the Adapter), based on _equation (24).

The loss function value L _mse shown in equation (24) is the mean square error loss. An example of the objective function used by the speech recognition device 1 when training the Adapter is calculated by the following equation (25).

In equation (25), α is a hyperparameter. The speech recognition device 1 updates the internal parameters of the neural network of the Adapter so as to minimize the loss Loss ₂ expressed in equation (25). This loss Loss ₂ may be conveniently referred to as the "second loss." The speech recognition device 1 trains the Adapter so as to update parameters for layers other than the final layer of the multi-layer neural network in the Adapter. When training the Adapter, the speech recognition device 1 reuses the parameter values of the trained final layer of Encoder _m ·40 as the parameters of the final layer of the Adapter. In this way, the parameters of the layers other than the final layer of the Adapter are adjusted (updated) so as to simulate the output of Encoder _m ·40. By training the Adapter in this way, it is expected that the Adapter will become a model that converts a CTC symbol string into an output feature frame _hm of the Encoder _m .

[Relearning Encoder _h.50 ]
Next, the retraining of the Encoder _h -50 will be described. In the retraining of the Encoder _h -50, a text string containing the topic of the speech to be inferred is used as training data to improve the recognition accuracy of the topic to be inferred. Hereinafter, the domain to which the topic of the speech to which the Intermediate CTC speech recognition model and Adapter are trained belongs is referred to as the source domain, and the domain to which the topic of the speech to be inferred belongs is referred to as the target domain. As a method for retraining the Encoder _h -50, either a first retraining method or a second retraining method described below may be used. In this embodiment, either the first retraining method or the second retraining method can be selected as desired depending on the application, etc.

[Relearning Encoder _h.50 : First Relearning Method]
FIG. 4 is a block diagram showing the functional configuration of the speech recognition device 1 according to this embodiment for relearning the Encoder _h 50 using the first relearning method. The first relearning method performs domain adaptation using only text from the target domain. Re-training the Encoder _h 50 is performed on the premise that the above-described training of the Adapter has been completed. As shown in the figure, the speech recognition device 1 includes the Encoder _h 50, a CTC loss calculation unit 52, a pseudo-CTC symbol string generation unit 120, and an adapter unit 130. Note that FIG. 4 also shows the convolutional neural network unit 20, the Encoder _l 30, and the Encoder _m 40.

As shown in the figure, a configuration including the pseudo-CTC symbol string generator 120, adapter unit 130, Encoder _h 50, and CTC loss calculator 52 may be referred to as a model adaptation device 2. The model adaptation device 2 adapts at least the encoder of the recognition result output layer (only Encoder _h 50 in this embodiment) of a speech recognition model including a convolutional neural network unit 20, Encoder _l 30, Encoder _m 40, and Encoder _h 50 to a target domain. A feature of this embodiment is that the model adaptation device 2 does not require speech data of the target domain, but operates based only on text data in that domain.

The outline of the functions of each unit shown in FIG. 4 has already been described. That is, the trained adapter unit 130 outputs a feature value similar to the feature value h _m that would be output by the Encoder _m -40, based on the pseudo-CTC symbol string output by the pseudo-CTC symbol string generation unit 120. The Encoder _h -50 outputs a feature value h _h,i (hat) based on the output from the adapter unit 130. The Encoder _h -50 adjusts the parameters of its internal neural network based on the loss calculated by the CTC loss calculation unit 52. For convenience, the loss calculated by the CTC loss calculation unit 52 for the re-training of the Encoder _h -50 may be referred to as the "first loss." That is, the first loss is a loss based on the output from the Encoder _h -50 obtained by inputting the feature value output by the Adapter into the model of the Encoder _h -50.

The details of the relearning of Encoder _h -50 using the first relearning method will be described with reference to Figure 4. The method for training the Adapter has been described above, but here, the Encoder _h -50 is relearned using a trained Adapter. When relearning Encoder _h -50, only text strings containing the topic of the speech to be recognized are used as training data. In other words, when relearning Encoder _h -50, training data of text strings of the target topic (domain) is used.

In relearning Encoder _h ·50 using the first relearning method, the speech recognition device 1 first generates N pseudo CTC symbol strings based on text strings of the target domain, which are training data, using a pseudo CTC symbol string conversion method described later. These N pseudo CTC symbol strings are a _i (hat) expressed by the following equation (26).

The CTC loss calculation unit 52 of the speech recognition device 1 calculates the loss function value L _h,i for the above a _i (hat) using the following equations (27) and (28).

That is, L _h,i which is the i-th loss function value (i = 1, ..., N) is calculated based on h _h,i (hat) obtained by inputting the feature output by the Adapter by inputting the i-th CTC symbol string a _i (hat) into Encoder _h 50. The objective function when adaptive learning of Encoder _h 50 is performed by the first re-learning method is calculated by the following equation (29) as the average of N loss function values.

In the process of relearning Encoder _h , the speech recognition device 1 updates the internal parameters of the neural network so as to minimize the value of Loss ₃ expressed in equation (29). However, when relearning Encoder _h ·50, the speech recognition device 1 fixes the parameters of the trained Adapter. In other words, in the stage of relearning Encoder _h ·50, the speech recognition device 1 does not update the internal parameters of the Adapter, but updates the internal parameters of Encoder _h ·50.

Upon completing such retraining, the Encoder _h.50 neural network is expected to be adapted to the topic of the text strings used for retraining.

[Encoder _h.50 Re-learning: Second Re-learning Method]
5 is a block diagram showing a functional configuration for the speech recognition device 1 according to this embodiment to re-train Encoder _h 50 using the first re-training method. As shown in the figure, the speech recognition device 1 includes Encoder _h 50, a CTC loss calculation unit 52, a pseudo-CTC symbol string generation unit 120, an adapter unit 130, a convolutional neural network unit 20, Encoder _l 30, and Encoder _m 40. Here, the configuration including Encoder _h 50, the CTC loss calculation unit 52, the pseudo-CTC symbol string generation unit 120, the adapter unit 130, the convolutional neural network unit 20, Encoder _l 30, and Encoder _m 40 functions as a model adaptation device 2 that executes processing using the second re-training method. In the second retraining method, domain adaptation is performed using not only the target domain text but also the source domain speech-text pair when retraining Encoder _h ·50. When the second retraining method is used, it is expected to be effective in reducing the deterioration of speech recognition accuracy in the source domain.

In relearning Encoder _h ·50 using the second relearning method, the speech recognition device 1 generates _Nt pseudo CTC symbol strings based on text strings in the target domain, which are training data, using a pseudo CTC symbol string conversion method described later. These _Nt pseudo CTC symbol strings are at _,i (hat) expressed by the following equation (30).

The CTC loss calculation unit 52 of the speech recognition device 1 calculates the loss function value L _t,i for the above a _t,i (hat) by the following equations (31) and (32).

Furthermore, a string of _Ns acoustic features is input to the trained Intermediate CTC speech recognition model to calculate a loss function value L _s,j . This string of acoustic features is expressed by the following formula (33). Furthermore, the loss function value L _s,j is calculated by formulas (34), (35), (36), and (37).

That is, the convolutional neural network unit 20 performs convolution on the sequence _xj of acoustic features and outputs Convolution( _{xj )} . The Encoder _l 30 outputs hsl _,j based on Convolution(xj). The Encoder _m 40 outputs hsm _{,j based on hsl ,} j. The Encoder _h 50 outputs hsh _{,j based on hsm ,} j.

The objective function in the second relearning method is Loss _st expressed in the following equation (38).

In other words, Loss _st is the average of N _t loss function values L _t,i plus the average of N _s loss function values L _s,j as a regularization term to suppress degradation of source domain recognition accuracy. The model adaptation device 2 re-trains Encoder _h・50 by updating the model parameters so as to minimize Loss _st . However, at this time, the Adapter parameters are not trained but are fixed. Once re-training is complete, Encoder _h・50 becomes a neural network adapted to the topic of the text string without reducing its processing ability for source domain speech.

The loss calculated by the above formula (37) is conveniently referred to as the "third loss." That is, the third loss is a loss obtained (calculated) from a symbol string output by a model consisting of multiple layers (in this embodiment, the first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50) of the encoder unit based on a frame sequence of acoustic features, and a correct output symbol string corresponding to the frame sequence of acoustic features. In other words, in the second relearning method described here, when adapting the final-layer model to a target domain using the first loss, the encoder unit also performs adaptation based on the above-mentioned third loss. That is, relearning is performed based on the loss Loss _st expressed by formula (38). The second relearning method makes it possible to prevent or at least suppress degradation of recognition performance in the domain (source domain) where the encoder unit was originally trained, while adapting to a target domain.

When comparing the first and second relearning methods described above, using the first relearning method is expected to improve speech recognition accuracy in the target domain more than using the second relearning method. However, when using the first relearning method, speech recognition accuracy in the source domain deteriorates. When using the second relearning method, the improvement in speech recognition accuracy in the target domain is smaller, but it is expected to have the effect of reducing deterioration in speech recognition accuracy in the source domain.

[Generation of pseudo CTC symbol strings]
Next, a method for converting a text character string into a pseudo CTC symbol string will be described. The pseudo CTC symbol string was used for relearning the above-mentioned Encoder _h.50 . The speech recognition device 1 generates a set of pseudo CTC symbol strings based on character strings included in the training text data.

Consider the case where a certain text character string is converted into a pseudo CTC symbol string. For example, if the length of the CTC symbol string is limited to 6, there are nine possible CTC symbol strings corresponding to the character string "Nii Tenki" (good weather).
1: _/ ...
6: I/I/_/I/Heaven/Ki 7: I/_/I/I/Heaven/Ki 8: I/_/I/Heaven/Ki 9: I/_/I/Heaven/Ki/Ki

In the pseudo-CTC symbol string above, "_" is a simplified representation of the blank symbol <blank>. In a CTC symbol string, wherever the same symbol appears consecutively, such as in "ii", a blank symbol <blank> is always inserted between them. Note that the separator between symbols is represented by a slash.

The position of the blank symbol <blank> can be considered a position (time period) in a feature frame where no pronunciation is present. As shown in the example above, there are many CTC symbol strings corresponding to a certain string (here, "Iiteki" [good weather]). In other words, training a neural network using all CTC symbol string patterns corresponding to a certain string can require an enormous amount of time. On the other hand, there are also CTC symbol strings that are possible but unlikely to occur in reality. For example, if a CTC symbol string of length 10 is generated for the string "Iiteki" [good weather], it is unlikely to occur in reality a feature frame in which there is a long period between "I" and "I" with no pronunciation, such as "I/__/__/__/__/__/__/i/ten/ki," or in which a specific symbol repetition (pronunciation interval) is longer than a predetermined length, such as "i/__/i/ten/ki/ki/ki/ki/ki/ki." Therefore, when converting a text string to a CTC symbol string, it is possible to limit the symbol strings to only those with patterns that are considered to occur in reality.

Therefore, the speech recognition device 1 of this embodiment converts a _i (hat), which is a CTC symbol string calculated based on the feature frame h _l output from the trained Intermediate CTC Encoder _l , into a pseudo CTC symbol string that is likely to occur based on statistics. Specifically, the speech recognition device 1 converts the CTC symbol string into a pseudo CTC symbol string in accordance with a probability density function generated from statistics of the number of consecutive identical symbols and the number of consecutive blank symbols <blank>.

To obtain statistics for generating a pseudo-CTC symbol string, for example, speech data used in training the Intermediate CTC is used. The speech data is input to the trained Intermediate CTC, and statistics are obtained based on a _l (hat) output from Encoder _l . Specifically, the number of times the same symbol (excluding blank symbols) appears consecutively n times in a _l (hat) is defined as N _c (n), and the number of times the blank symbol <blank> appears consecutively n times is defined as N _b (n). Based on these count data, the probability density function P _c (n) of the number of consecutive occurrences of the same symbol (excluding blank symbols) and the probability density function P _b (n) of the number of consecutive occurrences of blank symbols <blank> are calculated using the following equations (39) and (40), respectively. In other words, in this embodiment, the probability density functions are generated based on statistics regarding the symbol string actually output from the encoder based on speech.

An example of calculation for finding a probability density function is as follows. Assume that speech data contains two linguistic expressions, "nice weather" and "tomorrow." If the CTC symbol strings output by Encoder ₁ for the speech "nice weather" and "tomorrow" are "_/_/i/_/i/i/i/_/weather/_/_/_" and "_/min/min/_/_/_/day/day/_," respectively, the number of consecutive symbols is as follows: Note that the blank symbol <blank> is represented as "_".

The number of occurrences of each consecutive symbol (character) other than the blank symbol is as follows:
_Nc (0)=0
_Nc (1) = 3: (corresponding to "i", "ten", and "ki")
_Nc (2) = 2: (applies to "Morning/Morning" and "Sun/Sun")
_Nc (3) = 1: (corresponds to "i/i/i")

Thus, the probability P _c (n) is:
P _c (0)=0/6=0
P _c (1) = 3/6 = 1/2
P _c (2) = 2/6 = 1/3
P _c (3) = 1/6

On the other hand, the number of occurrences of each consecutive blank symbol <blank> is as follows:
N _b (0) = 1: (Since there is no "_" between "heaven" and "ki", count N _b (0))
N _b (1) = 4: (applies between "i" and "i", between "i" and "ten", before "akari", after "hi")
N _b (2) = 1: (applies before "i")
N _b (3) = 2: (after "ki" (spirit)), between "myo" (light) and "hi" (day)

Therefore, the probability P _b (n) is:
P _b (0) = 1/8
P _b (1) = 4/8 = 1/2
P _b (2) = 1/8
P _b (3) = 2/8 = 1/4

The above example shows a probability value calculated based on two pieces of speech data, "nice weather" and "tomorrow." However, the actual statistics are created based on all of the speech data used in training Intermediate CTC. This allows us to understand the probability (trend) of the number of consecutive symbols (other than blank symbols, or blank symbols) that appear consecutively in the CTC symbol strings corresponding to the speech data used in training.

Figures 6 and 7 are flowcharts showing the processing steps for conversion into a pseudo-CTC symbol string. Figures 6 and 7 are connected by a connector, and together form a single flowchart. The processing of this flowchart is executed by the pseudo-CTC symbol string generator 120. Below, the processing steps for generating a CTC symbol string will be explained with reference to this flowchart.

In the process of this flowchart, a text string c is converted into a CTC symbol string _ai (hat). By repeating the process of this flowchart, multiple (N) CTC symbol strings _ai (hat) (where i = 1, ..., N) can be created. The text string c is expressed by the following equation (41). In equation (41), V is a set of output symbols (i.e., characters) that do not contain blank symbols, and J is the length (number of characters) of the string c.

First, in step S1 of FIG. 6, the pseudo-CTC symbol string generator 120 sets (initializes) a _i (hat) to a null symbol string.

The next step S2 is the start of Loop 1. Here, the pseudo-CTC symbol string generation unit 120 controls the conditions for Loop 1. In this flowchart, j is a variable that serves as an index for the iterations in Loop 1. The initial condition for Loop 1 is j = 1. The condition during the iteration of Loop 1 is "+1" (i.e., j = j + 1). The condition for determining whether Loop 1 ends is "J" (the loop is terminated when j is such that the condition j <= J is no longer satisfied). In other words, the processing of Loop 1 is repeated from j = 1 to j = J, with j being incremented by 1. The end point of Loop 1 is Step S10.

Next, in step S3, the pseudo CTC symbol string generator 120 selects the number of consecutive blank symbols <blank> _Nb probabilistically in accordance with _Pb (n). That is, the pseudo CTC symbol string generator 120 selects the number of consecutive blank symbols _Nb based on previously obtained statistics and in accordance with a random element. In the process of step S3, _Nb can be any of 0, 1, 2, ... as long as the probability is not zero.

Next, in step S4, the pseudo-CTC symbol string generator 120 determines whether the following conditions are true or false for the current character position j: (j ≠ 1) and (c[j-1] == c[j]). Note that if j = 1, the condition is determined to be false, so c[j-1] may not be referenced. Alternatively, it may be assumed that c[0] is any arbitrary symbol. If the above condition is true (step S4: YES), the process proceeds to step S5. If the above condition is false (step S4: NO), the process jumps to step S8 in Figure 7. The branching based on the condition determination in step S4 is intended to execute the processes of steps S5 to S7 only for locations where the same character appears consecutively in the input string.

Next, when the process proceeds to step S5, the following occurs. That is, step S5 is the start point of loop 2. Here, the pseudo-CTC symbol string generation unit 120 controls the conditions related to loop 2. The condition for executing the processing inside loop 2 is ( _Nb = 0). If the initial value of _Nb ( _Nb determined in step S3) is not 0, the processing inside loop 2 is never executed. If the initial value of _Nb is 0, the processing of loop 2 is repeated while the value of _Nb is 0. When the value of _Nb is no longer 0, loop 2 is exited. The end point of loop 2 is step S7.

Step S6 is the only step in loop 2. In step S6, the pseudo CTC symbol string generator 120 probabilistically selects the number _Nb of consecutive blank symbols <blank> in accordance with _Pb (n). Note that each time _Nb is selected, it is selected based on a different probabilistic factor. In other words, as the selection of _Nb is repeated, it is expected that a value other than 0 will be selected as _Nb .

Step S7 is the end point of Loop 2. If the condition (N _b = 0) is true, the processing of Loop 2 is repeated. If the condition (N _b = 0) is false, Loop 2 is exited and the process proceeds to the next step S8 in Fig. 7. In other words, at the point when Loop 2 is exited, a value other than 0 has been selected as _Nb .

In step S8 of FIG. 7, the pseudo-CTC symbol string generator 120 concatenates N _b blank symbols <blank> to the rear (usually, to the right) of the current symbol string a _i (hat) based on the value of N _b that has already been determined.

Next, in step S9, the pseudo CTC symbol string generator 120 probabilistically selects the number of consecutive characters (symbols other than blank symbols) _Nc in accordance with the predetermined _Pc (n). That is, the pseudo CTC symbol string generator 120 selects the number of consecutive characters _Nc based on previously obtained statistics and in accordance with a random element. That is, _Nc can be any of 0, 1, 2, ... as long as the probability is not zero. Next, the pseudo CTC symbol string generator 120 concatenates _Nc consecutive characters c[j] ( _Nc is the value selected in this step) to the rear (usually, to the right) of the current symbol string _ai (hat).

Step S10 is the end point of Loop 1. As per the above condition, the value of j is incremented by 1 (incremented by 1), and the processing of Loop 1 is repeated as long as the resulting value of j is 1, 2, ..., J. When the condition for repetition is no longer met, that is, when incrementing J causes the value of j to reach J+1, Loop 1 is exited and processing proceeds to the next step, S11. In other words, when Loop 1 is exited, processing of characters c[1] through c[J] in the source string is completed.

Next, in step S11, the pseudo CTC symbol string generator 120 probabilistically selects the number _Nb of consecutive blank symbols <blank> in accordance with the predetermined _Pb (n). That is, the pseudo CTC symbol string generator 120 selects the number _Nb of consecutive symbols based on previously obtained statistics and in accordance with a random element. That is, _Nb can be any of 0, 1, 2, ... as long as the probability is not zero. Next, the pseudo CTC symbol string generator 120 concatenates _Nb consecutive blank symbols <blank> ( _Nb is the value selected in this step) to the end of the current symbol string _ai (hat).

Of the processes shown in this flowchart, step S1 can be considered initialization processing. Steps S3 through S8 can be considered first-phase processing. The first-phase processing determines the number of consecutive blank symbols <blank> and concatenates that number of blank symbols <blank> as part of the string to be generated. Step S9 can be considered second-phase processing. The second-phase processing determines the number of consecutive characters (other than blank symbols) and concatenates that number of characters as part of the string to be generated. The first and second phases are repeated for each character (starting from the beginning) contained in the source text string (loop 1 processing controlled by steps S2 and S10). Step S11 can be considered third-phase processing. The third-phase processing follows loop 1 above, i.e., determines the number of consecutive blank symbols <blank> following the last character of the input string and concatenates that number of blank symbols <blank> as part of the string to be generated.

Here, we will explain the processing of the above flowchart using an actual example. As an example, let's assume that the source text string is "Tomorrow will be a nice day."

<First Phase Processing>
In the process of step S3 of the above-mentioned flowchart, _Nb, which is the number of consecutive blank symbols <blank>, is selected probabilistically according to the calculated _Pb (n). Here, it is assumed that the selected _Nb is 3. In the above example, the probability that _Nb = 3 is selected is _Pb (3) = 1/4. In the process of step S8 of the above-mentioned flowchart, when three blank symbols <blank> are concatenated to _ai (hat), _ai (hat) = "_/_/_". Here, the slashes are inserted as separators between symbols for convenience.

<Second Phase Processing>
In the process of step S9 of the above-mentioned flowchart, _Nc, which is the number of consecutive symbols (i.e., characters) other than blanks, is selected probabilistically according to the calculated _Pc (n). Assume that the selected _Nc is 2. Here, the probability that _Nc = 2 will be selected is _Pc (2) = 1/3 in the above example. Then, a sequence of _Nc characters (i.e., two in this case) "明" is concatenated with _ai (hat). Then, _ai (hat) = "_/_/_/明/明". The slashes are inserted as separators between symbols for convenience.

<Repeat the first and second phases>
The above first and second phase processes are repeated for each of the remaining characters in the input character string: "hi", "ha", "i", "i", "ten", and "ki".

<Third Phase Processing>
After the above-mentioned repetitive processing is completed, the third phase processing is performed, and consecutive blank symbols <blank> are added to the end of the CTC symbol string.

If the same character is repeated in an input character string (such as "ii" in "tomorrow will be a good day"), the number _Nb of blank symbols between the consecutive characters ("ii") will not be 0. This is guaranteed by the processing in steps S5 to S7 in the flowchart.

In this way, based on the character string "Tomorrow will be a nice day," a CTC symbol string such as "__/__/__/min/min/__/day/day/__/__/ha/hi/__/i/__/ten/ten/ki/__/_" is output. Note that since each of the N CTC symbol strings a _i (hat) is generated based on a probabilistic element (the number of consecutive characters or the number of consecutive blank symbols <blank>), it is fully expected that the symbol strings will be different from one another except for pairs that match by chance.

[Processing during inference]
Next, a description will be given of the inference process of the speech recognition device 1 after model training has been completed. Note that, as a prerequisite for executing inference, re-training of Encoder _h using text data in the target region has also been completed.

For input speech x to be recognized, the speech recognition device 1 calculates output feature quantities _hh using the following equation (42): Furthermore, the speech recognition device 1 obtains a CTC symbol string a (hat) based on the feature quantities _hh using the following equation (43):

Then, similar to conventional CTC speech recognition techniques, the speech recognition device 1 combines successive occurrences of the same character contained in the obtained CTC symbol string a (hat) into a single character and deletes blank symbols <blank> to obtain text as the speech recognition result. In other words, the speech recognition device 1 combines repeated characters in the output symbol string (a (hat)) and outputs the character string obtained by deleting blank symbols contained in the output symbol string as the speech recognition result.

Figure 8 is a block diagram showing an example of the internal configuration of the speech recognition device 1 of this embodiment. The speech recognition device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, RAM 902, input/output port 903, input/output devices 904 and 905, etc., and bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations. RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input/output port 903 is a port through which the central processing unit 901 exchanges data with external input/output devices, etc. The input/output devices 904 and 905 are input/output devices. Input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input/output port via the bus 906.

At least some of the functions of the speech recognition device 1 in the above-described embodiment can be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs over networks like the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within the computer systems that serve as servers or clients in such cases. The program may be designed to implement some of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already stored in the computer system.

In addition, backpropagation based on the calculated loss can be used to train neural networks (update parameters).

The above describes an embodiment, but the present invention can also be implemented in the following modified forms. Note that multiple embodiments and modified forms may be combined to the extent possible.

[Modification 1]
In the above embodiment, the encoder is implemented with three layers (a first layer encoder unit 30 (Encoder _l 30), a second layer encoder unit 40 (Encoder _m 40), and a third layer encoder unit 50 (Encoder _h 50)). The number of layers of the encoder may be other numbers. For example, the number of layers of the encoder may be four or more. Even in such cases, adaptation based on the text of the target region should be possible for at least the layers including the final layer (high-level side) of the encoder.

[Modification 2]
Instead of implementing the entire speech recognition device 1 shown in Fig. 1, only the model adaptation device 2 shown in Fig. 4 may be implemented. In this case, the model adaptation device 2 can still adapt the speech recognition model. In other words, the model adaptation device 2 can adapt the speech recognition model based on the text in the target domain so that it outputs a character string as a speech recognition result that matches the target domain.

[Modification 3]
A different Encoder _h 50 may be retrained in advance for each topic (field, area). In this case, by selecting the Encoder _h 50 that matches the topic of the speech recognition target each time, a character string of the speech recognition result that matches the topic will be output. Conversely, the Encoder _h may be trained each time using text on the target topic.

[Modification 4]
In the above embodiment, the pseudo CTC symbol string generator 120 generates a pseudo CTC symbol string for relearning the Encoder _h -50. As a modified example, instead of providing the pseudo CTC symbol string generator 120, a symbol string (corresponding to a pseudo CTC symbol string) belonging to a target region may be provided from outside. In this case as well, the Adapter calculates and outputs features to be input to the Encoder _h -50 based on the symbol string.

[Modification 5]
As a fifth modification, before converting a text string into a pseudo-CTC symbol string, a character substitution process may be performed on the text string. This substitution process has the effect of making the text string resemble the error pattern of the string output by Encoder _l 30. Then, a pseudo-CTC symbol string is generated based on the text string after this substitution process. In other words, Encoder _h 50 can be retrained via Adapter based on the CTC symbol string corresponding to such a string that may contain an error. This is expected to further improve recognition performance.

Figure 9 is a block diagram showing the functional configuration of the portion of the speech recognition device 1 that performs the replacement process according to Variation 5. As shown in the figure, the speech recognition device 1 (model adaptation device 2) according to Variation 5 includes a replacement processing unit 115. The replacement processing unit 115 performs character replacement processing in the text data (character string) supplied by the target area text supply unit 110. The replacement processing unit 115 passes the text data (character string) after replacement to the pseudo-CTC symbol string generation unit 120. In Variation 5, the pseudo-CTC symbol string generation unit 120 generates a pseudo-CTC symbol string based on the text string passed from the target area text supply unit 110, using the character string after the replacement processing unit 115 has performed character replacement.

The method for calculating the output character probability distribution used in Modification 5 is as follows: The speech data used to train the Intermediate CTC is input to the trained Intermediate CTC. The alignment a _l (hat), which is the highest probability output by Encoder _l 30 for each speech data, is expressed by the following equation (44).

Consider the case where the position of a _l (hat) is t=t' and the following equation (45) holds.

In the case of the above equation (45), the probability that Encoder _1-30 outputs each symbol v (v is an element of V) at the position t=t' is as shown in the following equation (46). In other words, the probability expressed by equation (46) is the probability that symbol s is replaced by symbol v, and does not depend on t.

For each t in each piece of training data, when the symbol at that position is s (s is an element of V as described above), the probability of the above formula (46) is calculated, and the average of the calculated probabilities is defined as the output character probability distribution P _s (v). Once this output character probability distribution is determined, the value expressed by the following formula (47) can be considered to be the probability that when a certain character s is the correct character, Encoder ₁ 30 will erroneously output a different character s'.

For every s (where s is an element of V), an output character probability distribution P _s (v) can be calculated and used in the text string replacement process.

The text string replacement process in Variation 5 is performed as follows: For each character c[j] contained in text string c, the character is replaced according to the output character probability distribution (obtained above) expressed by equation (48) below.

For example, if the text string c is "Weather forecast," then c[1] = heaven. Furthermore, the character substitution is as shown in equations (49) and (50) below.

In this example, a substitution process from "heaven" to "heaven" is performed with a probability of 0.8. That is, the character "heaven" remains unchanged, and the original character string c remains unchanged as a result of this substitution. On the other hand, a substitution process from "heaven" to "dot" is performed with a probability of 0.1. That is, since "heaven" is replaced with a dot, the original character string c changes to "Tenki Yoho Desu" (This is a weather forecast). By performing the above substitution process on all c[j] included in character string c, a character string intended to resemble the error in the character string output by Encoder _l -30 can be obtained. Then, the CTC symbol string obtained based on this substitution process can be used to re-learn Encoder _h -50.

In other words, in variant 5, the replacement processing unit 115 performs processing to replace characters included in text belonging to the target region (text supplied from the target region text supply unit 110) with other characters in accordance with an output character probability distribution obtained based on predetermined statistical data. Furthermore, the pseudo CTC symbol string generation unit 120 performs processing to generate a CTC symbol string based on the text belonging to the target region after the replacement processing output by the replacement processing unit 115.

Embodiments of the present invention (including variations) have been described in detail above with reference to the drawings, but the specific configuration is not limited to these embodiments and includes designs that do not deviate from the gist of the present invention.

As described above, in this embodiment (including the modified examples), the end-to-end speech recognition model can be adapted using only text data. In other words, it is possible to domain-adapt the speech recognition model using only text data (without speech data) and improve the recognition accuracy of the target domain.

[Verification of the effect of the configuration]
A demonstration experiment was conducted to verify the effectiveness of the speech recognition device 1 according to the embodiment described above. The first speech used in the demonstration experiment was the CSJ corpus, which contains speeches presented at conferences on phonetics. Using this CSJ corpus, training of the Intermediate CTC and training of the Adapter were performed. Then, Encoder _h of the trained Intermediate CTC was retrained using weather forecast text data using the first and second retraining methods. Character error rates (CERs) were calculated for the weather forecast evaluation set and the CSJ corpus evaluation set, which were the second speech, using Encoder _h before and after retraining using the weather forecast text data. Character error rates were also calculated with and without the text string replacement process (the process described in Variation 5).

The experimental specifications common to both the first and second relearning methods are as follows:
Feature quantity: 80 dimensions of log-mel spectrogram + 3 dimensions of pitch + Δ + ΔΔ, total of 249 dimensions Character types: 3260 types

The model used is as follows:
Convolution: Convolutional neural network, 2 layers Encoder _l : Conformer, 6 layers Encoder _m : Conformer, 3 layers Encoder _h : Conformer, 3 layers Adapter: Conformer, 6 layers Number of pseudo-CTC symbol strings to be generated N: 64

Below are the individual experimental specifications and experimental results of the first and second relearning methods for relearning Encoder _h ·50.

The training data for the first retraining method (domain adaptation using only text from the target domain) is as follows:
1) Intermediate CTC study: CSJ Corpus 2) Adapter study: CSJ Corpus 3) Encoder _re -study: 1,000 weather forecast texts

The evaluation data for the first relearning technique is as follows.
1) eval1, eval2, and eval3 from the CSJ evaluation set
2) Two hours of weather forecast evaluation audio. However, this does not include the audio of the 1,000 sentences of weather forecast text used for relearning Encoder _h .

The learning parameters for the first re-learning method are as follows:
1) Training of Intermediate CTC: A model trained on the epoch with the lowest validation loss out of 100 epochs was used. 2) Training of Adapter: A model trained on the epoch with the lowest validation loss out of 50 epochs was used. 3) Retraining of Encoder _h : A model trained on the epoch with the greatest improvement in CER (Character Error Rate) out of 20 epochs was used. The number of pseudo-CTC character strings N was 64.

Table 1 below shows the comparison results of CER (character error rate) before and after re-training Encoder _h (in the case of the first re-training method).

As shown in Table 1 above, for the weather forecast evaluation speech, when the text string replacement process was not performed, the CER improved by 3.2%, from 18.7% before Encoder _h 's relearning to 15.5%. When the text string replacement process was performed (in the case of Variation 5), the CER improved by 5.0%, from 18.7% before Encoder _h 's relearning to 13.7%. Conversely, for the speech of eval1, eval2, and eval3 in the CSJ corpus, the CER naturally deteriorated after Encoder _h was relearned. The reason why recognition accuracy deteriorated after Encoder _h 's relearning in the evaluation data of the CSJ corpus was because Encoder _h was re-adapted from the topic of the original CSJ corpus to the topic of weather forecasts.

A specific example of an improvement to the weather forecast evaluation voice (when a text string replacement process is performed) is as follows.
Evaluation voice utterance: "This is the predicted minimum temperature."
Recognition result of Encoder _h before relearning: "Predicted optimal fundamental tone"
Recognition result after Encoder _h retraining: "This is the predicted minimum temperature."

As described above, the demonstration experiment described here demonstrated that Encoder _h was retrained using only weather forecast text data, and that it was possible to output symbol strings appropriate for the topic of weather forecasts during inference.

The training data for the second retraining method (domain adaptation using text in the target domain and a speech-text pair in the source domain) is as follows:
1) Intermediate CTC training: CSJ Corpus 2) Adapter training: CSJ Corpus 3) Encoder _retraining : 1,000 weather forecast texts and the CSJ Corpus

The evaluation data for the second relearning method is as follows.
1) eval2 of the CSJ evaluation set
2) Two hours of weather forecast evaluation audio. However, this does not include the audio of the 1,000 sentences of weather forecast text used for relearning Encoder _h .

The learning parameters for the second re-learning method are as follows:
1) Training of Intermediate CTC: Use the model trained at the epoch with the lowest validation loss out of 100 epochs. 2) Training of Adapter: Use the model trained at the epoch with the lowest validation loss out of 50 epochs. 3) Retraining of Encoder _h : Use the model trained at the epoch with the greatest improvement in CER out of 20 epochs (after retraining Encoder _h using the second retraining method in Table 2 below).
However, regarding epochs, one epoch was counted when the weather forecast text had been learned once.
The number of pseudo CTC character strings _Nt is 64.
The number of CSJ speeches _Ns to be learned simultaneously with the pseudo-CTC character strings is 64.

Table 2 below shows the comparison results of CER before and after retraining Encoder _h (in the case of the second retraining method). Note that the results after retraining shown in Table 2 are for the case where the text string replacement process (transformation rate 5) was performed.

In the results shown in Table 2, the CER for the CSJ evaluation set (CSJ eval2), which is the source domain, is 5.2% both before relearning Encoder _h and after relearning Encoder _h using the second relearning method. In other words, the recognition accuracy in the source domain is not degraded by relearning using the second relearning method. On the other hand, the CER for the weather forecast evaluation set (weather forecast), which is the target domain, is improved from 18.7% before relearning to 15.3% after relearning using the second relearning method.

In Table 2, the CERs after relearning Encoder _h using the first relearning method (for 4 epochs and 5 epochs) are compared. This is because stopping relearning using the first relearning method, which uses only text data as training data, midway is considered as a way to improve speech recognition accuracy in the target domain without degrading speech recognition accuracy in the source domain. In other words, when using the first relearning method, a CER equivalent to the CER (15.3%) of the weather forecast evaluation set when using the second relearning method is achieved after 4 epochs (CER is 15.5%) or 5 epochs (CER is 15.1%). The CER of the CSJ evaluation set (CSJ eval2) after 4 epochs of relearning using the first relearning method is 5.8%. Furthermore, after 5 epochs of relearning using the first relearning method, the CER for the CSJ evaluation set (CSJ eval2) was 6.1%. In other words, for both 4 and 5 epochs, the CER for the CSJ evaluation set (CSJ eval2) deteriorated from 5.2% before relearning. In other words, it was confirmed that the second relearning method is more effective than the first relearning method for comparison in terms of not degrading source domain speech recognition accuracy.

As described above, the first relearning method and the second relearning method can be used appropriately. Furthermore, it has been shown that the relearning method according to this embodiment is effective when using either the first relearning method or the second relearning method. That is, assume a case where it is desired to improve the recognition accuracy of a pre-trained general-purpose speech recognition model for speech on a desired topic. For example, assume that for a topic about a sport that is just starting to become popular, no speech-text pairs are available, and only the text is available. In such a situation, recognition performance can be improved by performing adaptive training (encoder _h relearning) according to this embodiment using text on the topic of the target sport.

The present invention can be used, for example, for speech recognition processing and model adaptation processing for speech recognition. Such inventions can be used in a wide range of industries. However, the scope of use of the present invention is not limited to the examples given here.

1 Speech recognition device 2 Model adaptation device 10 Speech supply unit 20 Convolutional neural network unit 30 First layer encoder unit (Encoder ₁ )
32 CTC loss calculation unit 40 Second layer encoder unit (Encoder _m )
42 CTC loss calculation unit 50 Third layer encoder unit (Encoder _h )
52 CTC loss calculation unit 110 target region text supply unit 115 replacement processing unit 120 pseudo CTC symbol string generation unit (pseudo symbol string generation unit)
130 Adapter unit 210 CTC symbol string generator (symbol string generator)
220 Mean square error loss calculation unit (adapter unit loss calculation unit)
230 CTC loss calculation unit (loss calculation unit for adapter unit)
901 Central processing unit 902 RAM
903 Input/output port 904, 905 Input/output device 906 Bus

Claims (8)

an encoder unit having a model consisting of multiple layers, which receives a sequence of frames of acoustic features and outputs an output symbol string consisting of a sequence of symbols corresponding to the frames of the acoustic features, and each of the models in the multiple layers is configured to be capable of machine learning;
an adapter unit that receives a symbol string belonging to a target region and outputs a feature quantity to be input to a model in the final layer of the encoder unit;
a pseudo-symbol string generator that generates the symbol string to be input to the adapter unit by performing at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on the text belonging to the target region;
Equipped with
an intermediate feature is passed from a model in a certain layer to a model in a next layer in the multiple layer models of the encoder unit;
the encoder unit is configured to perform machine learning based on a relationship between a frame sequence of the acoustic features and the output symbol sequence, and then input the features output by the adapter unit to a model of the final layer of the encoder unit, and adapt the model of the final layer to the target domain using a first loss that is a loss based on an output from the model of the final layer.
Voice recognition device. a replacement processing unit that performs processing to replace characters included in the text belonging to the target region with other characters in accordance with an output character probability distribution obtained based on predetermined statistical data;
Furthermore,
the pseudo-symbol string generation unit generates the symbol string based on the text belonging to the target region after the replacement process output by the replacement processing unit.
The speech recognition device according to claim 1 . The pseudo-symbol string generation unit
determining the number of times the same character is repeated when repeating characters in the text based on statistics regarding the number of times the same character is repeated in an intermediate symbol string, which is a symbol string calculated based on features output from the model of a predetermined layer in the encoder unit that has been trained based on the relationship between the frame string of the acoustic features and the output symbol string;
determining the number of consecutive blank symbols to be inserted between characters in the text based on statistics regarding the number of consecutive blank symbols in the intermediate symbol string;
3. The speech recognition device according to claim 1 . a symbol string generation unit that generates a symbol string to be input to the adapter unit for training the adapter unit based on the intermediate features output from a model of a predetermined layer of the encoder unit based on the acoustic features;
an adapter unit loss calculation unit that calculates a second loss, which is a loss for a feature output by the adapter unit, based on the symbol string generated by the symbol string generation unit;
Furthermore,
The adapter section is configured to perform machine learning on the adapter section using the second loss calculated by the adapter section loss calculation unit.
4. A speech recognition device according to claim 1. The adapter loss calculation unit
a loss based on the likelihood of a correct output symbol string given the feature values output from the adapter unit;
a loss representing the difference between a feature output from the adapter unit and a feature output from a model of a corresponding layer in the encoder unit;
Calculating the second loss based on
5. The speech recognition device according to claim 4 . The speech recognition device
aggregating repeated characters in the output symbol string and deleting blank symbols contained in the output symbol string, and outputting the resulting character string as a speech recognition result;
3. The speech recognition device according to claim 1 . When adapting the model of the final layer to the target region using the first loss, the encoder unit also performs adaptation based on a third loss obtained from a symbol string output by the model consisting of multiple layers included in the encoder unit based on the frame sequence of the acoustic features and the correct output symbol string corresponding to the frame sequence of the acoustic features.
A speech recognition device according to any one of claims 1 to 6 . an encoder unit having a model consisting of multiple layers, which receives a sequence of frames of acoustic features and outputs an output symbol string consisting of a sequence of symbols corresponding to the frames of the acoustic features, and each of the models in the multiple layers is configured to be capable of machine learning;
an adapter unit that receives a symbol string belonging to a target region and outputs a feature quantity to be input to a model in the final layer of the encoder unit;
a pseudo-symbol string generator that generates the symbol string to be input to the adapter unit by performing at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on the text belonging to the target region;
Equipped with
an intermediate feature is passed from a model in a certain layer to a model in a next layer in the multiple layer models of the encoder unit;
the encoder unit is configured to perform machine learning based on a relationship between a frame sequence of the acoustic features and the output symbol sequence, and then input the features output by the adapter unit to a model of the final layer of the encoder unit, and adapt the model of the final layer to the target domain using a first loss that is a loss based on an output from the model of the final layer.
A program that enables a computer to function as a speech recognition device.