JP7800738B2 - Speech recognition device, speech recognition method, and program - Google Patents

Description

This disclosure relates to a speech recognition device, a speech recognition method, and a program.

Speech recognition is a technology that converts human speech into a sequence of tokens (such as a sequence of words, subwords, or character strings) using a computer. The introduction of neural network (NN) technology has dramatically improved the accuracy of speech recognition. Conventional speech recognition systems consist of modules such as an acoustic analysis unit, an acoustic model unit, a language model unit, and a search unit, with each module (mainly the acoustic model unit and the language model unit) incorporating individual NN-based modeling. In recent years, however, end-to-end (E2E) speech recognition technology, which models a speech recognition system using a single NN, has been actively researched, and it has been reported that it has achieved speech recognition accuracy that exceeds human capabilities in telephone conversation speech recognition tasks. E2E speech recognition systems use a single NN, but the encoder, decoder, and other components are modularized within the system (see, for example, non-patent document 1).

E2E speech recognition systems (models) are trained to directly capture the conversion from speech to token sequences using paired data of speech and its corresponding token sequences (text). While it is possible to train highly accurate E2E speech recognition models using large amounts of paired data, there is a limit to the amount of paired data that can actually be prepared. This is because transcribing the token sequences corresponding to speech is usually done manually, which is costly in terms of both time and money.

On the other hand, text data alone can be obtained more easily and in larger quantities than paired data. Therefore, a method has been proposed in which a language model (a model that calculates the probability of occurrence of token sequences) trained using large amounts of text data prepared separately from paired data is integrated with an E2E speech recognition model as external knowledge (external language model). The most commonly used integration method is shallow fusion (SF). SF is a simple method that weights and adds the scores (logarithmic probability) output by the main E2E speech recognition model and the scores (also logarithmic probability) output by the external language model during the search process, yet has been shown to be highly effective (see, for example, non-patent documents 1 and 2).

Since SF is performed sequentially according to the extension of speech recognition hypotheses from the beginning of a sentence in beam search, a conventional forward language model (FLM) is used as the external language model. Below, "hypothesis" and "token sequence" are used interchangeably. Note that beam search is a process that simultaneously extends and expands multiple hypotheses while retaining the hypothesis that is likely to yield the highest score for the input speech and pruning less likely hypotheses.

On the other hand, there is a method called rescoring of speech recognition hypotheses. Rescoring involves re-scoring (re-evaluating) multiple hypotheses obtained in the form of an N-best list or lattice using an external rescoring model (usually a language model) to obtain a final speech recognition result (one best hypothesis). In this method, a backward language model (BLM) is often used in addition to an FLM, and its effectiveness has been confirmed (see, for example, Non-Patent Document 3). A backward language model (BLM) is a language model trained using data (reversed text data) in which the normal word order used to train an FLM is reversed. The model calculates the probability of occurrence of hypotheses in the reverse word order, from the end of the sentence to the beginning of the sentence.

BLM is used in rescoring because it can evaluate the probability of a hypothesis from a different perspective than FLM, and because the two models are complementary, using them together can assign a more accurate language score to a hypothesis. Another reason is that rescoring is performed on hypotheses for which the search process has been completed, i.e., when the token sequence from the beginning of the sentence to the end (or from the end to the beginning) has been completely obtained (referred to as a "complete hypothesis" in this specification), BLM, which uses the end of the sentence as the starting point for score calculation, is easily applicable. Conversely, it is difficult to apply BLM to hypotheses in the middle of the search process whose end of the sentence has not yet been determined (referred to as a "partial hypothesis" in contrast to a complete hypothesis; also, when simply referring to a "hypothesis," it refers to a "partial hypothesis"). Furthermore, because FLM uses the beginning of the sentence as the starting point for score calculation, it can also be easily applied to partial hypotheses.

Figure 1 shows an overview of the speech recognition process in an E2E speech recognition model. In this specification, the E2E speech recognition model is an attention-based encoder-decoder (AED) model, and the search algorithm is a label (token) synchronous beam search (see, for example, Non-Patent Document 4). However, this disclosure is also applicable to other types of E2E speech recognition models and search algorithms. As shown in Figure 1, input speech is first input to the encoder in the E2E speech recognition model and converted into a hidden state vector sequence X. It is rare for the speech waveform to be input directly as input speech; more commonly, a filter bank feature sequence extracted from the speech waveform is input as input speech.

Next, X is input to the decoder in the E2E speech recognition model, and a complete hypothesis of length T {<sos>, w^ _1:T , <eos>} = {<sos>, w ₁ , w ₂ , ... , w _T , <eos>} that satisfies the following formula (1) is found by a search process, and this is output as the final speech recognition result (one best complete hypothesis). Note that in the main text of this specification, "w^" is originally intended to be written as "^" directly above "w", but due to limitations in the functionality of word processing software, it is not possible to write it directly above, so for convenience it is written as "w^".

Here, <sos> and <eos> are special symbols representing the start of a sentence (or sequence) and the end of a sentence (or sequence), respectively. The FLM and BLM score calculations are performed using <sos> and <eos> as the starting points. In this specification, <sos> is excluded from the calculation of the hypothesis length. γ is a reward (γ≧0) that is added each time a hypothesis is extended, and its introduction prevents a complete hypothesis that is too short (T is too small) from becoming the final speech recognition result. In addition, during the search process in the decoder, score calculations are performed in the logarithmic probability domain to prevent numerical underflow.

Figure 2 shows an overview of the speech recognition process in an E2E speech recognition model, where an FLM is used as an external language model using a conventional SF. The process from obtaining X from input speech by the encoder is the same as in Figure 1, but the difference from Figure 1 is that the decoder's search process takes into account not only the decoder score (see equation (1)) but also the language score calculated by the FLM (FLM score), as shown in equation (2) below.

where α is the weight for the FLM score (α≧0).

FIG. 3 shows the details of the search process with SF in the decoder of FIG. 2 , in which a token w _t is concatenated to a hypothesis {<sos>, w _1:t-1 } of length t-1 at time t, and the score calculation is performed when extending the hypothesis to a hypothesis {<sos>, w _{1 :t-1 } of length t. In this specification, extending a hypothesis by one token is considered to be the passage of one time. It is assumed that the score s1 of the hypothesis {<sos>, w 1:t-1} } given X has already been calculated at the previous time t-1. At t=0, the score s1 is initialized to 0. When concatenating w _t to {<sos>, w _1:t-1 }, the decoder score s2 of w _t given X and {<sos>, w _1:t-1 } and the FLM score s3 of w _t given {<sos>, w _1:t-1 } are calculated. The above three scores are weighted and added as shown in the following equation (3) to obtain the score s4 of {<sos>, w _1:t } when X is given.

As mentioned above, FLM uses <sos> as the starting point for score calculation, so FLM score calculation can be performed in synchronization with the extension of the hypothesis. In addition, the FLM score for the hypothesis {<sos>, w _1:t } can be written as the following equation (4).

In other words, the FLM score can be calculated cumulatively. In other words, once calculated, the FLM score for token w _t given hypothesis {<sos>, w _1:t-1 } remains constant throughout the search process for that input speech.

The above explains the details of score calculation when extending a single type of token in a single hypothesis by concatenating them, but in reality, the number of types of tokens that can be concatenated to a single hypothesis is equal to the total number of types, i.e., the vocabulary size. In what follows, the vocabulary size is assumed to be V. Note that the vocabulary includes <eos> but excludes <sos>.

Figure 4 shows the calculation of scores (s2 and s3) when, at time t, V types of tokens _wv (v = 1, ..., V) are concatenated into one hypothesis {<sos>, _w1:t-1 } and expanded into V hypotheses {<sos>, _w1:t-1 , _wv } (v = 1, ..., V) in a search process incorporating SF. During this expansion, it is necessary to calculate the decoder score s2 (see Figure 3) for the vocabulary size, which can be performed in a batch using a softmax function (see, for example, Non-Patent Document 4). Regarding the FLM score s3 (see Figure 3), when a neural network-based LM is used as the FLM, V FLM scores s3 can be calculated in a batch using a softmax function, just like the decoder score s2. Furthermore, when a classical count-based n-gram LM is used as the FLM, the FLM score can be obtained by table reference (however, when an n-gram LM is used, long hypotheses cannot be taken into account, so for example, in the case of a 3-gram LM (a language model that calculates the probability of a chain of three tokens), the hypothesis {<sos>, w _1:t-1 }, which is the condition for score calculation, is degenerated to {w _t-2,t-1 } (a chain of two tokens). For score calculation of FLM based on NN and n-gram, see, for example, non-patent document 5).

The above describes score calculation when V types of tokens are concatenated into one hypothesis and expanded into V hypotheses. However, in actual beam search, multiple hypotheses are each expanded into V hypotheses, and these are then pruned. Figure 5 shows the overall processing procedure for beam search incorporating SF. The beam width is B (V>>B). Note that in Figure 5, the single best complete hypothesis (the single complete hypothesis with the highest score) is output as the speech recognition result, but it may be desirable to output N best complete hypotheses (N complete hypotheses sorted in descending order from highest score based on score s4). In this case, in S94, rather than selecting only the single best complete hypothesis, the (B-b+N) complete hypotheses are sorted in descending order based on score s4, as in step S92, and the top N complete hypotheses are selected and the others are pruned.

Z. T¨uske, G. Saon, K. Audhkhasi, and B. Kingsbury, “Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard,” in Proc. Interspeech, 2020, pp. 551-555. C. Gulcehre et al., “On using monolingual corpora in neural machine translation,” arXiv:1503.03535v2 [cs.CL]. A. Ogawa, N. Tawara, M. Delcroix, and S. Araki, “Lattice rescoring based on large ensemble of complementary neural language models,” in Proc. ICASSP, 2022, pp. 6517-6521. S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid CTC/Attention architecture for end-to-end speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 11, no. 8, pp. 1240-1253, Dec. 2017. M. Sundermeyer, R. Schl¨uter, and H. Ney, “LSTM neural network for language modeling,” in Proc. Interspeech, 2012, pp. 194-197. Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” arXiv:1206.5533v2 [cs.LG].

Since SF can also be considered a type of rescoring using an external language model, if BLM could be applied to SF in addition to FLM, it would be possible to assign more accurate language scores to hypotheses during the search process, thereby achieving more accurate beam search. However, because SF extends and develops speech recognition hypotheses from the beginning of a sentence, FLM is used as the language model, and it is difficult to apply BLM to hypotheses during the search process where the end of the sentence has not been determined.

Therefore, the purpose of this disclosure is to provide a speech recognition device that can incorporate BLM into SF.

The speech recognition device of the present invention includes a decoder score calculation unit, a complete hypothesis generation unit, a BLM score calculation unit, a symbol removal unit, an ISF score calculation unit, and an N-best complete hypothesis output unit.

Let 1,...,t be time, _w1 , _w2 ,..., _wt-1 , _wt be tokens at each time in a speech recognition hypothesis. When concatenating token wt to a hypothesis _w1:t-1 = { _w1 , _w2 ,...,wt _-1 } of length t- ₁ and extending it to a hypothesis w1 _:t of length t, the decoder score calculation unit calculates the decoder score in the speech recognition model for token _wt in calculating the ISF score. If token _wt is not at the end of a sentence, the complete hypothesis generation unit temporarily concatenates the special symbol <eos>, which indicates the end of a sentence, to generate a complete hypothesis. The BLM score calculation unit reverses the word order of the complete hypothesis and calculates the BLM score. If token _wt is not at the end of a sentence, the symbol deletion unit deletes the temporarily concatenated special symbol <eos>. The ISF score calculation unit calculates the ISF score for the hypothesis _{w1:t of} length t based on the ISF score for the hypothesis w1 _{:t-1 of length t} -1, the decoder score for the token _wt , the BLM score for the hypothesis _{w1:t of length t} , and the BLM score for the hypothesis w1 _: t-1 of length t-1. The decoder score calculation unit, complete hypothesis generation unit, BLM score calculation unit, symbol deletion unit, and ISF score calculation unit perform their respective processes while incrementing t by 1 until no partial hypotheses remain. The N-best complete hypothesis output unit outputs the N-best complete hypotheses based on the ISF scores.

The speech recognition device of the present invention makes it possible to incorporate BLM into SF.

FIG. 1 is a diagram showing an overview of speech recognition processing in an E2E speech recognition model. FIG. 1 is a diagram showing an overview of speech recognition processing in an E2E speech recognition model when FLM is used. FIG. 3 is a diagram showing an example of score calculation when concatenating a token w _t with a hypothesis of length t−1, as a detail of the search process in the decoder of FIG. 2 . A diagram showing an example of how scores (s2 and s3) are calculated when expanding one hypothesis into V hypotheses. 10 is a flowchart showing the processing procedure of a conventional beam search in which SF is introduced. FIG. 1 is a diagram showing an overview of speech recognition processing in an E2E speech recognition model when BLM is used. FIG. 7 is a diagram showing an example of score calculation when concatenating a token w _t with a hypothesis of length t−1, as a detailed view of the search process in the decoder of FIG. 6 . A diagram showing an example of how scores (s2 and s6) are calculated when expanding one hypothesis into V hypotheses. FIG. 1 is a block diagram illustrating an example of the functional configuration of a speech recognition device according to a first embodiment. FIG. 2 is a block diagram showing an example of the functional configuration of an N-best complete hypothesis output unit according to the first embodiment. 4 is a flowchart showing the operation of the voice recognition device according to the first embodiment. 4 is a flowchart showing a beam search operation of the speech recognition device according to the first embodiment. FIG. 10 is a diagram showing an overview of speech recognition processing in an E2E speech recognition model when both FLM and BLM are used. FIG. 14 is a diagram showing an example of score calculation when concatenating a token w _t with a hypothesis of length t−1, as a detailed view of the search process in the decoder of FIG. 13 . FIG. 10 is a block diagram showing an example of the functional configuration of a speech recognition device according to a second embodiment. 10 is a flowchart showing the operation of the voice recognition device according to the second embodiment. 10 is a flowchart showing a beam search operation of the speech recognition device according to the second embodiment. FIG. 10 is a block diagram showing an example of the functional configuration of a speech recognition device according to a fourth embodiment. FIG. 13 is a block diagram showing an example of the functional configuration of an N-best complete hypothesis output unit according to the fourth embodiment. 10 is a flowchart showing the operation of the voice recognition device according to the fourth embodiment. 10 is a flowchart showing a beam search operation of the speech recognition device according to the fourth embodiment. FIG. 10 is a block diagram showing an example of the functional configuration of a speech recognition device according to a fifth embodiment. FIG. 13 is a block diagram showing an example of the functional configuration of an N-best complete hypothesis output unit according to a fifth embodiment. 10 is a flowchart showing the operation of the voice recognition device according to the fifth embodiment. 13 is a flowchart showing a beam search operation of the speech recognition device according to the fifth embodiment. FIG. 13 is a block diagram showing an example of the functional configuration of a speech recognition device according to a sixth embodiment. FIG. 20 is a block diagram showing an example of the functional configuration of an N-best complete hypothesis output unit according to a sixth embodiment. 13 is a flowchart showing a beam search operation of the speech recognition device according to the sixth embodiment. FIG. 13 is a diagram showing an example of a token string generated by the language model learning device according to the ninth embodiment. FIG. 13 is a block diagram showing an example of the functional configuration of a language model learning device according to ninth and tenth embodiments. 13 is a flowchart showing the operation of the language model learning device according to the ninth and tenth embodiments. FIG. 23 is a diagram showing an example of a token string generated by the language model learning device of the tenth embodiment. FIG. 23 is a diagram showing an example of a token sequence generated by the language model learning device of the eleventh embodiment. FIG. 12 is a block diagram showing an example of the functional configuration of a language model learning device according to an eleventh and twelfth embodiments. 13 is a flowchart showing the operation of the language model learning device according to the eleventh and twelfth embodiments. FIG. 23 is a diagram showing an example of a token string generated by the language model learning device of the twelfth embodiment. 10A and 10B are diagrams showing the effect of improving speech recognition accuracy by the speech recognition device described in the embodiment. FIG. 2 is a diagram showing an example of the functional configuration of a computer.

The following describes in detail the embodiments of the present disclosure. Components having the same functions are assigned the same numbers, and duplicate explanations will be omitted.

Figure 6 shows an overview of speech recognition processing in an E2E speech recognition model, where a BLM is used as an external language model using the ISF of this disclosure. In Figure 2, the search process in the decoder of the E2E speech recognition model takes into account the language scores calculated using the FLM using the SF, but in Figure 6, the language scores calculated using the BLM using the ISF are instead taken into account.

Figure 7 shows the details of the search process in the decoder of Figure 6, illustrating the score calculation when, at time t, a token _wt is concatenated to a hypothesis _w1:t-1 = { _w1 , _w2 ,..., wt _-1 } of length t-1 and extended to a hypothesis w1 _:t of length t. It is assumed that the score s5 of w1 _:t-1 given X has already been calculated at the previous time t-1 (at t = 0, the score s5 is initialized to 0). When concatenating _wt to _w1:t-1 , the decoder score s2 of _wt given X and {<sos>, _w1:t-1 } is calculated. Here, the complete hypothesis {<sos>, w1 _:t , <eos>} is generated by temporarily concatenating <eos> to {<sos>, _w1:t } (not concatenated if _wt is <eos>). For this complete hypothesis, the word order is reversed to {<eos>, w _t:1 , <sos>}, and the BLM score s6 is calculated. After calculating the BLM score s6, the <eos> temporarily linked to {<sos>, w _1:t } is deleted (it is not deleted if w _t is <eos>). The weighted sum of these three scores yields the score s8 for {<sos>, w _1:t } given X. In this process, the BLM score s7 for the reversed temporary complete hypothesis {<eos>, w _t-1:1 , <sos>}, which was included in score s5 calculated at the previous time t-1, is subtracted. (The BLM score s7 is stored as information accompanying {<sos>, w _1:t-1 } during the search process at time t-1.) That is, the score s8 is obtained using the following formula (5):

Here, β is the weight for the BLM score (β ≥ 0). The two BLM scores (s6 and s7) in the third term on the right-hand side of equation (5) can be rewritten as equations (6) and (7) below.


As described above, when BLM is used for SF, the score calculation is not cumulatively calculated for hypotheses as when FLM is used for SF, but is performed iteratively for temporarily generated and inverted complete hypotheses (iterative SF: ISF). In other words, as is clear from equation (6), in the search process for a certain input speech, the "BLM score for w _τ given {<eos>, w _t:τ+1 }" calculated once changes every time the hypothesis is extended, i.e., every time the score calculation condition {<eos>, w _t:τ+1 } changes.

Figure 8 shows the calculation of scores (s2 and s6) when V types of tokens _wv (v=1, ..., V) are concatenated to one hypothesis {<sos>, w1 _:t-1 } at time t, and the resulting V hypotheses {<sos>, _w1:t-1 , _wv } (v=1, ..., V) are expanded. The decoder score s2 is calculated collectively for the vocabulary size (V) using the softmax function, as in Figure 4, but the BLM score s6 is calculated individually for each vocabulary size.

The functional configuration of the speech recognition device of Example 1 incorporating the above-mentioned processing will be described below with reference to Figure 9. As shown in the figure, the speech recognition device 1 of this example includes a decoder score calculation unit 11, a complete hypothesis generation unit 12, a BLM score calculation unit 13, a symbol deletion unit 14, an ISF score calculation unit 15, and an N-best complete hypothesis output unit 16.

As shown in FIG. 10, the N-best complete hypothesis output unit 16 includes a pruning unit 161, a hypothesis classification unit 162, a hypothesis selection unit 163, and a termination processing unit 164.

The detailed operation of each component is explained below with reference to Figures 11 and 12.

In calculating the ISF score when concatenating a token _wt to a hypothesis _w1:t -1 of length t-1 and extending it to a hypothesis w1 _:t of length t, the decoder score calculation unit 11 calculates the decoder score in the speech recognition model for the token _wt based on the hidden state vector sequence X and the hypothesis w1 _:t-1 of length t-1 (S11).

If the token w _t is not at the end of the sentence, the complete hypothesis generation unit 12 temporarily connects <eos> to {<sos>, w _1:t } to generate a complete hypothesis {<sos>, w _1:t , <eos>} (S12).

The BLM score calculation unit 13 reverses the word order of the complete hypothesis and calculates the BLM score (S13).

The symbol deletion unit 14 deletes the temporarily concatenated special symbol <eos> when the token _wt is not at the end of the sentence (S14).

The ISF score calculation unit 15 calculates the ISF score P ^isf ({<sos>,w _1:t-1 }|X) for the hypothesis w _{1:t of length t-1} based on the ISF score P _isf ( _{ <sos>,w _1:t-1 }|X) for the hypothesis w 1:t-1 of length t-1, the decoder score P _dec (w t |X,{<sos>,w _{1 :t-1} }), the BLM score P _blm ({<eos>,w _t:1 ,<sos>}), and the BLM score P ^blm ({<eos>,w t-1:1 ,<sos>}) for the hypothesis w _{1: t-1} of length t-1, for example using equation (5) (S15).

The decoder score calculation unit 11, full hypothesis generation unit 12, BLM score calculation unit 13, symbol deletion unit 14, and ISF score calculation unit 15 perform their respective processes while incrementing t by 1 until no partial hypotheses remain. Details will be provided later, so repeated branching and other details have been omitted from the diagram.

The N best complete hypothesis output unit 16 outputs the N best complete hypotheses based on the ISF score (S16).

The above is the general operation of each component. Below, with reference to Figure 12, we will describe the detailed processing procedure for beam search (beam width = B) for multiple (b) hypotheses, and then further expand each hypothesis at time t to V hypotheses for a total of bV hypotheses.

The flowchart in this figure is similar to the flowchart in Figure 5, but differs in that scores s2 and s6 are calculated instead of scores s2 and s3 in hypothesis development (S11-S15), and that pruning (S161) is performed based on score s8 instead of score s4.

The decoder score calculation unit 11, the complete hypothesis generation unit 12, the BLM score calculation unit 13, the symbol removal unit 14, and the ISF score calculation unit 15 execute the aforementioned steps S11 to S15 and calculate scores s2, S6, and S8.

The pruning unit 161 sorts the hypotheses in descending order based on the score s8, keeps the top B hypotheses, and prunes the other hypotheses (S161).

The hypothesis classification unit 162 classifies hypotheses as to whether they are complete hypotheses or not (S162).

The hypothesis selection unit 163 selects the 1(N) best complete hypotheses from the previous 1(N) best complete hypotheses = the 1(N) complete hypotheses with the highest scores (if any) and the newly obtained (B-b) complete hypotheses (total of (B-b+1(N)) complete hypotheses), and prunes the others (S163).

In the subsequent termination determination, if b=0, i.e., no partial hypotheses remain, the termination processing unit 164 outputs the 1(N) best complete hypotheses (S164) and the processing terminates (END).

In the subsequent termination determination, if b≠0, i.e., partial hypotheses remain, the speech recognition device 1 increments t, executes steps S11 to S15, S161, S162, and S163 again, and executes the termination determination again.

The speech recognition device of Example 2 performs SF and ISF simultaneously. Figure 13 shows an overview of speech recognition processing in an E2E speech recognition model, in which FLM is used as an external language model using conventional SF, and BLM is used as an external language model using ISF.

14 shows the details of the search process in the decoder of FIG. 13 , illustrating the score calculation when, at time t, a token w _t is concatenated to a hypothesis w _1:t-1 ={w ₁ , w ₂ , ..., w _t-1 } of length t-1 and extended to a hypothesis w _1:t of length t. It is assumed that the score s9 of w _1:t-1 given X has already been calculated at the previous time t-1 (at t=0, the score s9 is initialized to 0). When concatenating w _t to _{w 1:t-1} , the decoder score s2 of w _t given X and {<sos>, w _1:t-1 } and the FLM score s3 of w _t given {<sos>, w _1:t-1 } are calculated. Here, <eos> is temporarily concatenated to {<sos>, w _1:t } (it is not concatenated if w _t is <eos>), and a complete hypothesis {<sos>, w _1:t , <eos>} is temporarily generated. For this complete hypothesis, the word order is reversed to {<eos>, w _t:1 , <sos>}, and the BLM score s6 is calculated. After calculating the BLM score s6, the <eos> temporarily concatenated to {<sos>, w _1:t } is deleted (it is not deleted if w _t is <eos>). The score s10 of {<sos>, w _1:t } when X is given is obtained by weighting and adding the above four scores, but in this case, the BLM score s7 for the inverted temporary complete hypothesis {<eos>, w _t-1:1 , <sos>}, which was included in the score s9 calculated at the previous time t-1, is subtracted (the BLM score s7 is stored as information accompanying {<sos>, w _1:t-1 } in the search process at time t-1). In other words, the score s10 is obtained by the following equation (8).

The functional configuration of a speech recognition device according to a second embodiment incorporating the above-described processing will be described below with reference to Fig. 15. As shown in Fig. 15, the speech recognition device 2 of this embodiment includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 13, a symbol deletion unit 14, an SF+ISF score calculation unit 25, and an N-best complete hypothesis output unit 16. The components other than the FLM score calculation unit 22 and the SF+ISF score calculation unit 25 operate in the same way as those of the first embodiment.

The detailed operation of each component will be described below with reference to Fig. 16. Step S11 is executed in the same manner as in the first embodiment. The FLM score calculation unit 22 calculates the FLM score P _flm (w _t |{<sos>, w _1:t-1 )}) of the token w _t at time t based on the symbol <sos> and the hypothesis w _1:t-1 of length t-1 (S22). Steps S12 to S14 are executed in the same manner as in the first embodiment. The SF+ISF score calculation unit 25 calculates the SF+ISF score P ^sf+isf ({<sos>,w _1:t-1 }|X) for the hypothesis w _1:t-1 of length t-1 based on the SF+ISF score P sf+ _isf ({<sos>,w _1:t-1 }|X), the decoder _score P _dec (w _t | _X ,{<sos>,w _1:t-1 }), the FLM score P _flm ( _{w t} |{<sos>,w _{1:t-1 }), the BLM score P blm ({<eos>,w t:1 ,} <sos>}) and the BLM score P ^blm ({<eos>,w _{t-1:1 ,} <sos>}), for example using equation (8) (S25).

Figure 17 shows the overall processing procedure for beam search (beam width = B) incorporating SF and ISF. The flowchart in this figure is similar to the flowchart in Figure 12, but differs in that scores s2, s3, s6, and s10 are calculated in hypothesis development (S11, S22, S12, S13, S14, S25), and that pruning (S161) is performed based on score s10 instead of score s8.

If α = 0 and β > 0 in equation (8), SF will not be performed and only ISF will be performed (equivalent to Example 1). If α > 0 and β = 0 in equation (8), only SF will be performed and ISF will not be performed. If α = 0 and β = 0 in equation (8), neither SF nor ISF will be performed.

As described above, the speech recognition devices described in Examples 1 and 2 can incorporate BLM into SF, allowing for more accurate language scores to be assigned to hypotheses during the search process, thereby enabling more accurate beam searches. However, the above-mentioned method clearly shows that the computational cost of ISF using BLM is high. In the following Examples 3 to 6, speech recognition devices that reduce this computational cost are disclosed.

In the methods of Examples 1 and 2, the BLM score s6 is calculated individually for each vocabulary size, resulting in high computational costs (see Figure 8). Therefore, in Example 3, the BLM score s6 is calculated using batch calculations that bundle together hypotheses equal to the vocabulary size (for details on batch calculations, see, for example, Non-Patent Document 6). In the label (token) synchronous beam search assumed in this specification (see, for example, Non-Patent Document 4), batch calculations are particularly effective (no wasted calculations) because the lengths of each hypothesis during the search process are basically uniform, but batch calculations can also be introduced into other search algorithms.

In other words, in the speech recognition device of Example 3, the BLM score calculation unit is characterized by calculating the BLM score s6 by batch calculation that bundles together hypotheses equivalent to the vocabulary size, thereby reducing the cost of calculating the BLM score.

In the above-described embodiments, the ISF-based BLM score s6 was calculated every time the hypothesis was extended by one token (one time elapsed), but in the fourth embodiment, the ISF-based BLM score s6 is calculated every time the hypothesis is extended by I tokens (I time elapsed). In other words, the ISF-based BLM score s6 is calculated only when the hypothesis length (time) t satisfies t%I=0 (% is the modulo operator). Hereinafter, I is referred to as the interval (I≧1). In the present embodiment, Equation (7) in the first embodiment is changed to the following Equation (9).

The functional configuration of a speech recognition device according to the fourth embodiment, which incorporates the above-described processing, will be described below with reference to Fig. 18. As shown in Fig. 18, the speech recognition device 4 of this embodiment includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 43, a symbol deletion unit 14, an SF+ISF score calculation unit 25, and an N-best complete hypothesis output unit 46. The components other than the BLM score calculation unit 43 and the N-best complete hypothesis output unit 46 operate in the same way as those of the second embodiment.

As shown in Figure 19, the N-best complete hypothesis output unit 46 includes a pruning unit 161, a hypothesis classification unit 162, a complete hypothesis score calculation unit 463, a hypothesis selection unit 163, and a termination processing unit 164, and the components other than the complete hypothesis score calculation unit 463 operate in the same way as in Examples 1 and 2.

The detailed operation of each component will be described below with reference to Figure 20. Steps S11, S22, and S12 are executed in the same manner as in Example 2. The BLM score calculation unit 43 calculates the BLM score only when I is an arbitrary interval and the hypothesis length t satisfies t%I=0 (% is the modulo operator) (S43). Steps S14 and S25 are executed in the same manner as in Example 2.

Figure 21 shows the overall processing procedure for beam search that introduces ISF based on SF and interval I. The beam width is B. The flowchart in this figure is similar to the flowchart in Figure 17, but differs in that step S43 is performed instead of step S13 in hypothesis expansion, reducing the frequency of BLM score calculations; in step S161, pruning is performed based on score s10 when t%I=0, and based on scores other than the BLM score s9+s2+α×s3 when t%I≠0; and step S463 is inserted between steps S162 and S163, and score s10 is calculated for the complete hypothesis when t%I≠0.

As mentioned above, the effectiveness of rescoring (N-best rescoring and lattice rescoring) for complete hypotheses has been confirmed (see, for example, Non-Patent Document 3). Therefore, in this embodiment, in step S463, if t%I ≠ 0 for the complete hypothesis, the score s10 is calculated. If t%I = 0, that is, if the score s10 has already been calculated in the hypothesis development (S11, S22, S12, S43, S14, S25), there is no need to calculate it again. By introducing interval I, the calculation cost of ISF is reduced to approximately 1/I compared to before its introduction.

If I = 1 in equation (9), this embodiment is equivalent to embodiment 2. If α = 0 and β > 0 in equation (8), SF is not performed and only ISF is performed in interval I. If α > 0 and β = 0 in equation (8), ISF is not performed and only SF is performed. If α = 0 and β = 0 in equation (8), neither SF nor ISF is performed.

In beam search, it is necessary to retain the hypotheses (multiple hypotheses) that are likely to yield the highest score for the input speech, and to prune other hypotheses with lower likelihood at an early stage. If highly likely hypotheses are mistakenly pruned early in the search process, it becomes difficult to ultimately obtain highly accurate speech recognition results. Therefore, it is considered important to assign highly accurate scores to hypotheses at an early stage in the search process rather than at a later stage. Therefore, in Example 5, ISF is performed only when the hypothesis length t is shorter than L (t≦L).

The functional configuration of a speech recognition device according to Example 5, which incorporates the above-described processing, will be described below with reference to Figure 22. As shown in the figure, the speech recognition device 5 of this example includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 53, a symbol deletion unit 14, an SF+ISF score calculation unit 25, and an N-best complete hypothesis output unit 56. The components other than the BLM score calculation unit 53 and the N-best complete hypothesis output unit 56 operate in the same way as in Example 2.

As shown in Figure 23, the N-best complete hypothesis output unit 56 includes a pruning unit 161, a hypothesis classification unit 162, a complete hypothesis score calculation unit 563, a hypothesis selection unit 163, and a termination processing unit 164, and the components other than the complete hypothesis score calculation unit 563 operate in the same way as in Examples 1 and 2.

The detailed operation of each component will be described below with reference to Figure 24. Steps S11, S22, and S12 are executed in the same manner as in Example 2. The BLM score calculation unit 53 calculates the BLM score only when the length of the hypothesis t is shorter than L (t <= L), where L is an arbitrary working hypothesis length (S53). Steps S14 and S25 are executed in the same manner as in Example 2.

Figure 25 shows the overall processing procedure for beam search incorporating ISF based on SF and ISF implementation hypothesis length L. The beam width is B. The flowchart in this figure is similar to the flowchart in Figure 17, but differs in that step S53 is performed instead of step S13 in hypothesis expansion, reducing the frequency of BLM score calculations; in step S161, pruning is performed based on score s10 if t≦L, and based on scores other than the BLM score s9+s2+α×s3 if t>L; and step S563 is inserted between steps S162 and S163, where score s10 is calculated for the complete hypothesis if t>L.

If L = ∞, this embodiment becomes equivalent to embodiment 2. If α = 0 and β > 0 in equation (8), SF is not performed and only ISF is performed with the ISF execution hypothesis length L. If α > 0 and β = 0 in equation (8), ISF is not performed and only SF is performed. If α = 0 and β = 0 in equation (8), neither SF nor ISF is performed.

Example 6 is a variation of Example 2, in which pruning is performed in two stages. The functional configuration of a speech recognition device of Example 6 incorporating two-stage pruning processing will be described below with reference to Figure 26. As shown in the figure, the speech recognition device 6 of this example includes a decoder score calculation unit 11, an FLM score calculation unit 22, a complete hypothesis generation unit 12, a BLM score calculation unit 13, a symbol deletion unit 14, an SF+ISF score calculation unit 25, and an N-best complete hypothesis output unit 66. The components other than the N-best complete hypothesis output unit 66 operate in the same way as in Example 2.

As shown in Figure 27, the N-best complete hypothesis output unit 66 includes a first pruning unit 661, a second pruning unit 662, a hypothesis classification unit 162, a hypothesis selection unit 163, and a termination processing unit 164, and the components other than the first pruning unit 661 and the second pruning unit 662 operate in the same way as in Example 2.

Figure 28 shows the overall processing procedure for beam search when pruning is performed in two stages, a modification of Example 2. The beam width for the first stage of pruning is M, and the beam width for the second stage of pruning is B (M>B). The flowchart in this figure is similar to the flowchart in Figure 17, but differs in that only S11 and S22, which have low costs in hypothesis development, are executed; in step S661, a first pruning unit 661 executes pruning to leave the top M hypotheses based on a score other than the BLM score, s9 + s2 + α × s3; the BLM score calculation unit 13 calculates BLM scores only for the top M hypotheses obtained in the first stage of pruning, and other components execute pre-processing and post-processing (S12, S13, S14, S25), thereby reducing the frequency of BLM score calculation; and in step S662, after BLM score calculation, a second pruning unit 662 executes pruning to leave the top B hypotheses based on the score S10 through a second stage of pruning.

In this embodiment, in the hypothesis development and first-stage pruning (S11, S22, S661), pruning is performed based on the computationally low decoder score s2 and FLM score s3, leaving M hypotheses. In the BLM score calculation and second-stage pruning (S12, S13, S14, S25), pruning is performed in a second stage that also takes into account the computationally high BLM score s6, leaving B hypotheses. In other words, rather than calculating BLM scores for all of the very large number bV hypotheses as in Example 2, BLM scores are calculated only for a limited number of M hypotheses, reducing computational costs (bV>>M>B). Possible methods for setting M include, for example, M = B × B.

If M=bV, this embodiment becomes equivalent to embodiment 2. If α=0 and β>0 in equation (8), SF will not be performed (first-stage pruning will be performed based only on the decoder score s2), and only ISF will be performed. If α>0 and β=0 in equation (8), ISF will not be performed and only SF will be performed (first-stage pruning will not be performed (there is no point in performing it) and only second-stage pruning will be performed based on the score s9+s2+α×s3). If α=0 and β=0 in equation (8), neither SF nor ISF will be performed (first-stage pruning will not be performed and only second-stage pruning will be performed based on the score s9+s2).

Based on Example 1, it is also possible to modify it to perform two or more of the following: batch calculation (Example 3), ISF based on interval I (Example 4), ISF based on ISF execution hypothesis length L (Example 5), and two-stage pruning (Example 6).

Based on Example 2, it is also possible to modify it to perform two or more of the following: batch calculation (Example 3), ISF based on interval I (Example 4), ISF based on ISF execution hypothesis length L (Example 5), and two-stage pruning (Example 6).

In the following, a manually transcribed token sequence that is grammatically and order-correct and begins with <sos> and ends with <eos> is referred to as a "complete token sequence." BLM is trained using a complete token sequence with reversed word order, but as described above, in the ISF disclosed herein, BLM is applied to a partial hypothesis (a partial hypothesis is temporarily concatenated with <eos> to form a complete hypothesis, to which BLM is applied). Therefore, as shown in Figure 29, the device of Example 9 generates (augmented) a token sequence assuming that BLM will be applied to ISF, and uses the generated token sequence (hereinafter also referred to as a degenerate token sequence) to train an ISF-specific BLM.

With reference to Figure 30, the functional configuration of a language model training device that generates the above-mentioned token sequence and trains a language model will be described. As shown in the figure, the language model training device 900 of Example 9 (and Example 10, described below) includes a word order reversal unit 901, a degenerate token sequence generation unit 902, an FLM language model training unit 903, a BLM language model training unit 904, and an ISF-specific BLM language model training unit 905.

The operation of each component is explained below with reference to Figure 31. The word order reversal unit 901 obtains a complete token sequence used for FLM training ({<sos>, a, b, c, d, e, f, g, h, i, j, <eos>} in the example of FIG. 29 ) and reverses the word order of the complete token sequence (S901). The degenerate token sequence generation unit 902 generates a degenerate token sequence by deleting I tokens (I is a natural number; in this embodiment, I=1) that are located after the special symbol <eos>, which indicates the end of a sentence, from the complete token sequence with the reversed word order (S902). The FLM language model training unit 903 generates an FLM language model based on the complete token sequence used for FLM training (S903). The BLM language model training unit 904 trains a BLM language model based on the token sequence obtained by reversing the word order of the complete token sequence (S904). The ISF-specific BLM language model training unit 905 trains an ISF-specific BLM language model based on the degenerate token sequence (S905).

In this embodiment, the interval I is set to 1 (Example 4), and the ISF execution hypothesis length L is set to ∞ (Example 5). By setting it in this way, it is possible to generate a reversed token sequence of any length starting from any token, and to train an ISF-specific BLM that can easily adapt to any situation in the ISF.

However, as is clear from Figure 29, if I = 1 and L = ∞, the amount of training data for the ISF-specific BLM becomes enormous. In the following tenth to twelfth examples, the amount of training data for the ISF-specific BLM is limited by appropriately setting I and L.

As shown in Figure 32, the language model learning device of Example 10 limits the amount of learning data for the ISF-specific BLM by leaving L at ∞ and setting I to a value greater than 1 (I = 2 in Figure 32).

As shown in Figure 33, the language model training device of Example 11 limits the amount of training data for the ISF-dedicated BLM by leaving I at 1 and setting L to a finite integer value greater than 1 (L = 6 in Figure 33).
The functional configuration of a language model training device according to an eleventh embodiment (and a twelfth embodiment, described later) will be described with reference to Fig. 34. As shown in the figure, a language model training device 1100 according to an eleventh embodiment includes a word order reversal unit 901, a degenerate token sequence generation unit 1102, an FLM language model training unit 903, a BLM language model training unit 904, and an ISF-specific BLM language model training unit 905. The configuration other than the degenerate token sequence generation unit 1102 is the same as that according to the ninth and tenth embodiments. The language model training device 1100 according to the present embodiment is intended to use only token sequences shorter than the working hypothesis length L for training the ISF-specific BLM language model.

With reference to Figure 35, the operation of each component of the language model training device 1100 of Example 11 that differs from the operation of the language model training device 900 of Example 9 will be described. The degenerate token sequence generation unit 1102 deletes the token placed after <eos> from the complete token sequence with reversed word order, and generates the first degenerate token sequence so that the length of the token sequence excluding <eos> and the special symbol <sos> that indicates the beginning of a sentence is equal to L, and then generates degenerate token sequences by deleting I tokens (I = 1 in this example) at a time (S1102).

As shown in Figure 36, the language model learning device of Example 12 limits the amount of learning data for the ISF-specific BLM by setting I to a value greater than 1 (I = 2 in Figure 36) and L to a value greater than 1 (L = 6 in Figure 36).

The speech recognition device of Example 13 implements Examples 1 to 8 using the ISF-specific BLM trained in Examples 9 to 12.

<Experimental Results>
Figure 37 shows the improvement in speech recognition accuracy (reduction in word error rate [%]) achieved by the speech recognition device described in the examples. A neural language model (LSTMLM) based on long short-term memory (LSTM) is used as the language model. Comparing Method 0 with Method 1 confirms the effectiveness of SF using conventional FLM. Comparing Method 0 with Method 2 or 3 confirms the effectiveness of ISF using BLM by the speech recognition device described in the examples. Comparing Method 2 with Method 3 shows that the ISF-specific BLM of Example 9 is more suitable for ISF than the standard BLM. Comparing Methods 1, 2, and 3 with Method 4 confirms the effectiveness of combining SF using FLM and ISF using (ISF-specific) BLM.

<Additional Notes>
The device disclosed herein may, for example, be a single hardware entity that includes an input unit to which a keyboard or the like can be connected, an output unit to which an LCD display or the like can be connected, a communication unit to which a communication device (e.g., a communication cable) capable of communicating with an external device can be connected, a CPU (which may also include a central processing unit, cache memory, registers, etc.), RAM and ROM as memory, an external storage device such as a hard disk, and buses connecting these input unit, output unit, communication unit, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing recording media such as a CD-ROM. A physical entity equipped with such hardware resources includes a general-purpose computer.

The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required to process these programs (this is not limited to an external storage device; for example, the programs may be stored in ROM, which is a read-only storage device). In addition, data obtained by processing these programs is stored appropriately in RAM, external storage device, etc.

In a hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate. As a result, the CPU realizes the specified functions (each component represented as a "... unit," "... means," etc., above).

The present disclosure is not limited to the above-described embodiments, and modifications may be made as appropriate without departing from the spirit of the present disclosure. Furthermore, the processes described in the above embodiments may not only be executed chronologically in the order described, but may also be executed in parallel or individually depending on the processing capacity of the device executing the processes or as needed.

As mentioned above, when the processing functions of the hardware entities (apparatuses of the present disclosure) described in the above embodiments are realized by a computer, the processing content of the functions that the hardware entities should have is described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 10020 of the computer 10000 shown in Figure 38, and operating the control unit 10010, input unit 10030, output unit 10040, etc.

The program describing this processing can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memory. Specifically, for example, magnetic recording devices include hard disk drives, flexible disks, and magnetic tapes; optical disks include DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable)/RWs (Rewritable); magneto-optical recording media include MOs (Magneto-Optical discs); and semiconductor memories include EEP-ROMs (Electrically Erasable and Programmable-Read Only Memory).

This program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Furthermore, this program may be distributed by storing it in a storage device of a server computer and transferring it from the server computer to other computers via a network.

A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, each time a program is transferred from a server computer to the computer, the computer may execute the process in accordance with the received program. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function simply by issuing execution instructions and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (such as data that does not directly instruct the computer but has properties that define computer processing).

In addition, in this form, a hardware entity is configured by executing a specified program on a computer, but at least part of these processing contents may also be realized by hardware.

Claims (8)

Let 1,...,t be time instants, and _w1 , _w2 ,..., wt _-1 , _wt be tokens at each time instant in a speech recognition hypothesis. In calculating the ISF score when concatenating token _wt to a hypothesis _w1:t-1 = { _w1 , _w2 ,..., wt _-1 } of length t-1 and extending it to a hypothesis w1 _:t of length t,
a decoder score calculation unit for calculating a decoder score in a speech recognition model for the token _wt ;
a complete hypothesis generation unit that generates a complete hypothesis by temporarily concatenating a special symbol <eos> that indicates the end of a sentence when the token _wt is not the end of a sentence;
a BLM score calculation unit that calculates a BLM score by reversing the word order of the complete hypothesis;
a symbol deletion unit that deletes the temporarily concatenated special symbol <eos> when the token _wt is not the end of a sentence;
an ISF score calculation unit that calculates an ISF score for a hypothesis w _1:t of length _t based on an ISF score for a hypothesis w _1:t -1 of length t-1, the decoder score for the token w t, the BLM score for a hypothesis w _1: t of length t, and the BLM score for a hypothesis w _{1:t-1 of length t-1} ;
the decoder score calculation unit, the complete hypothesis generation unit, the BLM score calculation unit, the symbol deletion unit, and the ISF score calculation unit perform each process while increasing t by 1 until there are no more partial hypotheses;
A speech recognition device including an N-best perfect hypothesis output unit that outputs N-best perfect hypotheses based on ISF scores. 2. The speech recognition device according to claim 1,
further comprising an FLM score calculation unit that calculates an FLM score of a token _wt at time t based on a special symbol <sos> that indicates the beginning of a sentence and a hypothesis w1 _{:t-1 of length t-1} ;
The ISF score calculation unit
A speech recognition device that calculates an SF+ISF score for a hypothesis _w1:t of length _t based on an SF+ISF score for a hypothesis w1 _:t-1 of length t-1, the decoder score for the token wt, the FLM score for the token _wt , the BLM score for a hypothesis _w1:t -1 of length t, and the BLM score for a hypothesis w1 _: t-1 of length t-1. 3. The speech recognition device according to claim 2,
The BLM score calculation unit
A speech recognition device that calculates the BLM score by batch calculation of hypotheses equal to the vocabulary size. 3. The speech recognition device according to claim 2,
The BLM score calculation unit
A speech recognizer that calculates the BLM score only if the length of a hypothesis, t, satisfies t%I=0 (% is the modulo operator), where I is an arbitrary interval. 3. The speech recognition device according to claim 2,
The BLM score calculation unit
A speech recognition device that calculates the BLM score only when the length of the hypotheses t is shorter than L (t≦L), where L is an arbitrary hypothesis length. 3. The speech recognition device according to claim 2,
The BLM score calculation unit
A speech recognition device that sets the pruning beam width to M and calculates the BLM score for only the top M hypotheses based on scores other than the BLM score. A speech recognition method executed by a speech recognition device, comprising:
Let 1,...,t be time instants, and _w1 , _w2 ,..., wt _-1 , _wt be tokens at each time instant in a speech recognition hypothesis. In calculating the ISF score when concatenating token _wt to a hypothesis _w1:t-1 = { _w1 , _w2 ,..., wt _-1 } of length t-1 and extending it to a hypothesis w1 _:t of length t,
a decoder score calculation step of calculating a decoder score in a speech recognition model for the token _wt ;
a complete hypothesis generation step of generating a complete hypothesis by temporarily concatenating a special symbol <eos> representing the end of a sentence when the token _wt is not the end of a sentence;
a BLM score calculation step of calculating a BLM score by reversing the word order of the complete hypothesis;
a symbol deletion step of deleting the temporarily concatenated special symbol <eos> when the token _wt is not the end of a sentence;
an ISF score calculation step of calculating an ISF score for a hypothesis _w1:t of length _t based on an ISF score for a hypothesis w1 _:t -1 of length t-1, the decoder score for the token wt, the BLM score for a hypothesis _w1:t of length t, and the BLM score for a hypothesis w1 _{:t-1 of length t-1} ;
the decoder score calculation step, the complete hypothesis generation step, the BLM score calculation step, the symbol deletion step, and the ISF score calculation step are performed while incrementing t by 1 until no partial hypotheses remain;
A speech recognition method comprising: an N-best perfect hypothesis output step of outputting N-best perfect hypotheses based on ISF scores. A program that causes a computer to function as a speech recognition device described in any one of claims 1 to 6.