JP7828627B2 - Speech recognition device and method, and computer program - Google Patents

Description

This invention relates to a speech recognition device, and more particularly to a speech recognition device for multilingual speech.

With the widespread adoption of speech recognition technology, there are techniques that can convert speech into text with high accuracy, almost simultaneously, on an utterance-by-utterance basis, provided the language of the speech is known in advance. However, when recognizing multiple utterances whose languages are not known in advance, language identification technology is required to automatically determine the language of the speech. For example, Patent Document 1 (described below) discloses a technology that begins language identification at the start of an utterance and uses only the short initial period of the utterance for language identification. Furthermore, conventionally, the evaluation of systems that perform speech recognition simultaneously with utterance has widely used word error rate (WER) and real-time factor (RTF).

Japanese Patent Publication No. 2020-160374

Recently, international conferences, lectures, and Q&A sessions have become commonplace in the fields of politics, academia, and business. Video conferencing has increased opportunities to connect multiple countries and allow multiple speakers to participate in meetings in multiple languages. To make such events meaningful, simultaneous interpretation using machine translation is essential.

As a prerequisite for machine translation, it is necessary to accurately recognize the speech of each speaker. In meetings, for example, speeches often last for extended periods. Furthermore, long and short speeches frequently alternate, or short speeches may continue in succession. Multiple languages and speakers are often used, and these switch frequently. In such situations, unlike speech recognition of a single speaker, there are constraints: speech segment detection, language identification, and machine translation must be performed accurately in a short amount of time. Moreover, in the case of long audio recordings, non-speech is often inserted during speech, making processing even more difficult. These situations arise not only in simultaneous interpretation but also in tasks such as creating meeting minutes and subtitles.

When the language being used changes (when the speaker changes), it is necessary to identify the language in a short amount of time. However, existing language identification technologies have the problem of difficulty in accurately identifying the language from the limited information available at the beginning of an utterance. Speech recognition devices that process multiple languages continuously identify the language while the speaker is speaking, and then quickly display the speech recognition result or pass it on to post-processing such as translation. Furthermore, in order to accurately recognize and identify the language and speech of multilingual utterances, it is necessary to detect utterance segments with high accuracy. Existing utterance segment detection technologies have the problem of low accuracy for short utterances and often fail to detect utterance changes effectively.

Furthermore, when evaluating systems that perform real-time speech recognition in this manner, there is the question of whether it is appropriate to use WER and RTF as in the past.

Therefore, the objective of this invention is to provide a multilingual live speech recognition system that can perform real-time speech recognition in multiple languages with high accuracy and low latency.

Another objective of this invention is to provide metrics for appropriately evaluating multilingual live speech recognition systems.

The speech recognition device according to the first aspect of this invention includes: a speech segment detection means for detecting the start and end of a speech segment of a speech signal; a speech segment detection means for outputting a speech segment signal indicating a speech segment in response to a silence period of a predetermined time or longer after the end of a speech segment detected by the speech segment detection means; a language identification means for identifying the language of the utterance in the immediately preceding speech segment and outputting a language identifier in response to the speech segment signal; a plurality of speech recognition means for a plurality of different languages, each for performing speech recognition on the speech signal of the said speech segment in response to the detection of the start of a speech segment by the speech segment detection means; and a selection means for selecting and outputting the output of the speech recognition means for the language indicated by the identifier from among the plurality of speech recognition means in response to the identifier.

Preferably, the language identification means includes: a sub-section signal generation means that generates a sub-section audio signal of a predetermined length and shift amount from the audio signal of the utterance section in response to the detection of the end of an utterance section by the utterance section detection means; a language identification model that receives each audio signal of the sub-section and outputs information indicating which of the multiple languages the sub-section corresponds to; and a language determination means that determines the language of the audio signal of the utterance section and outputs an identifier for that language in response to the information output by the language identification model.

More preferably, each of the multiple speech recognition means includes a plurality of same-language speech recognition means that individually perform speech recognition for the language of the speech recognition means, and a switching means that, in response to an utterance segmentation signal, causes the same-language speech recognition means that is idling among the plurality of same-language speech recognition means to start speech recognition for each of the plurality of language speech recognition means.

More preferably, the speech segment detection means includes: a division means for dividing the speech signal into target segments of a predetermined length; a signal addition means for adding an additional signal to each target segment, which includes at least a first predetermined length of silent segment immediately preceding it; a sound segment detection means for distinguishing and detecting sound segments from silent segments within the target segments to which the signal addition means has been added; and a correction means for correcting sound segments by deleting sound segments corresponding to the additional signal within the sound segments detected by the sound segment detection means.

The computer program according to the second aspect of this invention causes the computer to function as follows: a speech segment detection means for detecting speech segments of an audio signal; a language identification means for identifying the language of the speech in the detected speech segment in response to the detection of a speech segment by the speech segment detection means; a plurality of speech recognition means for performing speech recognition on the audio signal of the speech segment in response to the detection of a speech segment by the speech segment detection means, for each of several different languages; and a selection means for selecting and outputting the output of the speech recognition means for the language indicated by the identification result from among the plurality of speech recognition means in response to the identification result by the language identification means.

A third aspect of this invention relates to a speech recognition method comprising the steps of: a computer detecting the start and end of a speech segment of a speech signal; the computer outputting a speech segmentation signal indicating a speech delimiter in response to a silence period of a predetermined time or longer after the end of the detected speech segment; the computer identifying the language of the utterance in the immediately preceding speech segment and outputting a language identifier in response to the speech segmentation signal; the computer initiating speech recognition of the speech signal of the speech segment using multiple speech recognition means for multiple different languages in response to the detection of the start of a speech segment; and the computer selecting and outputting the output of the speech recognition means for the language indicated by the identifier from among the multiple speech recognition means in response to the identifier.

Figure 1 is a functional block diagram of a multilingual live speech recognition device according to the first embodiment of this invention. Figure 2 shows the external appearance of the computer that implements the multilingual live speech recognition device shown in Figure 1. Figure 3 is a block diagram showing the hardware configuration of the computer shown in Figure 2. Figure 4 is a schematic diagram illustrating the method for detecting speech segments in the first embodiment of this invention. Figure 5 is a schematic diagram illustrating a method for correcting a detected speech segment according to the first embodiment of this invention. Figure 6 is a schematic diagram illustrating a language identification method in a first embodiment of the present invention. Figure 7 is a schematic diagram illustrating the method for detecting utterance segments in the first embodiment. Figure 8 is a flowchart showing the control structure of a computer program that implements multilingual live speech recognition processing performed by the multilingual live speech recognition device according to the first embodiment. Figure 9 is a flowchart showing the control structure of a program that implements preprocessing for VAD (Voice Activity Detection). Figure 10 is a flowchart showing the control structure of the program that implements VAD output correction. Figure 11 is a flowchart showing the control structure of the program that enables ASR (Automatic Speech Recognition) switching. Figure 12 is a flowchart showing the control structure of a program that implements LID (Language Identification) control. Figure 13 is a flowchart showing the control structure of the program that implements the display update process. Figure 14 shows a time chart for multilingual live speech recognition using a client-server system. Figure 15 shows the method for calculating latency according to the first embodiment. Figure 16 shows the test utterances used in the experiment of the first embodiment. Figure 17 is a diagram comparing the evaluation of the ASR unit used in the experiment with the evaluation of the multilingual live speech recognition according to the embodiment.

In the following descriptions and drawings, identical parts are assigned the same reference number. Therefore, detailed descriptions of them will not be repeated.

First Embodiment 1. Configuration (1) Overall Configuration Figure 1 shows a schematic functional configuration of the multilingual live speech recognition device 50 according to the first embodiment of this application in block diagram form. Referring to Figure 1, the multilingual live speech recognition device 50 has the function of receiving an input 52, which is a voice signal from a remote, detecting the speech interval, identifying the language for each speech interval and performing speech recognition, and outputting the result of speech recognition as a display 54.

The multilingual live speech recognition device 50 includes a FIFO (First-In, First-Out) format buffer 70 that receives and stores input 52, and a speech segment detection unit 72 that reads speech signals from this buffer 70 and detects speech segments. The input 52 includes a sequence of speech samples, each sample having time information based on a specific time. The speech segment detection unit 72 detects speech segments included in this sequence of speech samples and outputs a speech segment detection signal consisting of pairs of (start time, end time).

The multilingual live speech recognition device 50 further includes a speech segment detection unit 76 that, based on the output of the speech segment detection unit 72, considers a silent segment to be a speech segment when it continues for a predetermined length (for example, 0.5 seconds, but is not limited to this, and may be longer or shorter) and outputs a speech segment signal indicating a speech segment; an LID 78 that identifies the language of the speech using the beginning portion of the input speech signal and outputs a language identifier; and a device that can perform speech recognition of speech in multiple languages in parallel and outputs speech corresponding to the language identifier output by the LID 78. The system includes a multilingual ASR processing unit 80 for selecting and outputting recognition results, a control unit 74 for reading audio data from the buffer 70 at a position identified by information indicating the audio section from the VAD 102 and providing it to the LID 78 and the multilingual ASR processing unit 80, and a result display control unit 82 for receiving the speech recognition results output by the multilingual ASR processing unit 80 and, in response to the output of the LID discrimination result by the LID 78, outputting the speech recognition results from the multilingual ASR processing unit 80 for display 54 or outputting to an automatic translation device. Here, "utterance segment" refers not merely to the end of a single utterance, but to an utterance segment that takes into account the possibility of a change in the speaker. Therefore, when an utterance segment is detected, the ASR thread (described later), which is the input destination for the audio data, is switched. The language identification process of the utterance is also temporarily stopped and then resumed for the next utterance.

The speech segment detection unit 72 includes a preprocessing unit 100 for performing predetermined preprocessing on the audio data read from the buffer 70 to accurately recognize speech segments, and a VAD 102 for detecting the start and end times of the audio segments based on the audio data preprocessed by the preprocessing unit 100 and outputting a speech segment detection signal.

(2) Hardware Configuration Figure 2 shows the external appearance of the computer used to realize the multilingual live speech recognition device 50 shown in Figure 1. Figure 3 shows the hardware of the computer shown in Figure 2 in block diagram format.

Referring to Figure 2, this includes a computer 150 having a multilingual live speech recognition device 50, a DVD (Digital Versatile Disc) drive 182, and a USB (Universal Serial Bus) memory port 186, as well as a keyboard 154, mouse 156, and monitor 152, all connected to the computer 150, for user interaction. Of course, these are just one example of a configuration for when user interaction is required; any general hardware and software (e.g., touch panels, pointing devices in general) that can be used for user interaction can be utilized.

Referring to Figure 3, the computer 150 includes, in addition to the DVD drive 182 and USB memory port 186, a CPU (Central Processing Unit) 170, a GPU (Graphics Processing Unit) 172, a bus 190 connected to the CPU 170, GPU 172, and DVD drive 182, a ROM (Read-Only Memory) 176 connected to the bus 190 for storing the computer 150's boot-up program, etc., a RAM (Random Access Memory) 178 connected to the bus 190 for storing program instructions, system programs, and work data, etc., and an SSD (Solid State Drive) 180, which is a non-volatile memory connected to the bus 190. The SSD 180 is for storing programs executed by the CPU 170 and GPU 172, as well as data used by those programs. The computer 150 further includes a network interface 188 that provides connection to a network 166 enabling communication with other terminals. A USB memory stick 164 is detachable from the USB memory port 186, providing communication between the USB memory stick 164 and various parts of the computer 150.

In the above embodiment, the main components of the VAD102, LID78, and multilingual ASR processing unit 80 shown in Figure 1 consist of a trained model, a neural network, and a program.

The programs and parameters used by these programs, which cause the computer 150 to function as the buffer 70, control unit 74, utterance segment detection unit 76, LID 78, multilingual ASR processing unit 80, pre-processing unit 100, VAD 102, and correction unit 104 of the multilingual live speech recognition device 50 shown in Figure 1, are stored on a DVD 158 inserted into the DVD drive 182 and transferred from the DVD drive 182 to the SSD 180. Alternatively, these programs may be stored on a USB memory 164, the USB memory 164 is inserted into the USB memory port 186, and the programs are transferred to the SSD 180. Alternatively, these programs may be transmitted to the computer 150 via the network 166 and stored on the SSD 180.

The program is loaded into RAM 178 at runtime. Of course, the source program may be input using the keyboard 154, monitor 152, and mouse 156, and the compiled object program may be stored in SSD 180. In the case of a scripting language, the script entered using the keyboard 154, etc., may be stored in SSD 180. For programs running on a virtual machine, a program that functions as a virtual machine must be pre-installed on computer 150. However, since inference using multiple ASR threads involves a large amount of computation, it is preferable to implement each part of the embodiment of the present invention as an object program consisting of the computer's native code rather than a scripting language.

The CPU 170 reads the program from RAM 178 according to the address indicated by an internal register called the program counter (not shown) and interprets the instructions. The CPU 170 then reads the data necessary for executing the instructions from RAM 178, SSD 180, or other devices according to the address specified by the instructions and executes the processing specified by the instructions. The CPU 170 stores the execution result data at an address specified by the program, such as RAM 178, SSD 180, or a register within the CPU 170. At this time, the value of the program counter is also updated by the CPU 170 operating according to the program. The computer program may be loaded directly into RAM 178 from DVD 158, USB memory 164, or via a network. Note that some tasks within the program executed by the CPU 170 (primarily numerical calculations that can be executed in parallel) are executed by the GPU 172 according to the instructions included in the program or according to the analysis results when the CPU 170 executes the instructions.

The program that implements the functions of each part according to the above embodiment using computer 150 includes a plurality of instructions written and arranged to operate computer 150 to implement those functions. Some of the basic functions necessary to execute these instructions are provided by an operating system running on computer 150, a third-party program, or modules of various toolkits installed on computer 150, and are linked to the object program via dynamic linking at runtime. Therefore, this program does not necessarily have to include all the functions necessary to implement the system and method of this embodiment. This program only needs to include instructions that execute the operations of each of the above-described devices and their components by calling appropriate functions or functions of the "programming toolkit" in a controlled manner to obtain the desired results. The method of operating computer 150 for this purpose is well known and will not be repeated here.

Furthermore, the GPU 172 is capable of parallel processing, allowing it to execute large amounts of computations associated with neural network processing simultaneously, in parallel, or via pipeline. For example, parallel computation elements discovered in the program during compilation, or during program execution, are issued from the CPU 170 to the GPU 172 as needed, executed, and the results are returned to the CPU 170 directly or via a predetermined address in RAM 178, and assigned to predetermined variables in the program.

(3) Overview of the process A. Detection of speech segments Referring to Figure 4, the preprocessing, detection of speech segments, and correction processing of the detected speech segments performed by the preprocessing unit 100, VAD 102, and correction unit 104 will be described.

As shown in Figure 4(A), the input audio is assumed to have utterances 200 and 202. In Figure 4(A) and subsequent similar graphs, vertical lines indicate one-second intervals in the utterances. Specifically, Figure 4(A) shows five consecutive periods 210, 212, 214, 216, and 218, each with a length of one second. However, the period is not limited to one second; it can be longer or shorter. According to Figure 4(A), if we assume that utterance 200 begins at, for example, 0 seconds, then utterance 200 continues for approximately 1.3 seconds. Utterance 202 begins around 1.8 seconds and continues until approximately 4.3 seconds.

As shown in Figure 4(B), within the utterance 200, a 1-second audible portion 220 exists within the period 210. In this embodiment, when detecting the first utterance segment after the start of processing, as shown in Figure 4(C), a 0.1-second silent portion 222 is added to the beginning of the audio signal for that period to be used for utterance segment detection. That is, in the example shown in Figure 4(C), a silent portion 222 is added to the beginning of the audible portion 220 before the utterance segment detection process is performed. By adding a 0.1-second silent portion to the beginning in this way, the accuracy of detecting the audible segment is improved. However, the length of this added portion is not limited to 0.1 seconds; it may be longer or slightly shorter.

Next, as shown in Figure 4(D), in the processing of the input audio for period 212, the audio signal 230 for this period becomes the target of speech segment detection. The audio signal 230 includes a sounded portion 232 (the end of the utterance 200), a sounded portion 234 (the beginning of the utterance 202), and the silent portion in between. In this embodiment, for the second and subsequent periods of the speech segment detection process, i.e., from period 212 onward, the audio signal for the period targeted for speech segment detection, in the case of audio signal 230 in Figure 4(D), is preceded by the audio signal 236 for the latter half of the audio signal of the latter half of period 210 (in this example, audio signal 236 is part of the sounded portion)), and then the silent portion 238 is preceded. In other words, the second speech segment detection results in an audio signal that concatenates audio signal 230, audio signal 236, and the silent portion 238. However, the audio signal from the preceding section, which is added immediately before the target section, is not limited to a value equivalent to half the length of the target section. It can be shorter or longer. A shorter value will result in lower latency (discussed later), but the detection accuracy of the audible section is expected to decrease slightly. A longer value will increase the detection accuracy of the audible section, but the latency will increase.

The same processing is performed thereafter. Specifically, referring to Figure 4(E), in the processing for period 214, the audio signal 240 (the portion of the utterance 202 within period 214), which is the portion of the audio signal for period 214, the added section 242, which is the latter 0.5 seconds of the audio signal 230 for the preceding period 212, and the silent portion 244 added immediately before it are the targets of processing. Referring to Figure 4(F), in the processing for period 216, the portion of the audio signal for period 216 (the audio signal 250 of the utterance 202 within period 216), the added section 252, which is the latter 0.5 seconds of the preceding audio signal 240, and the silent portion 254 added immediately before it are the targets of processing. Referring to Figure 4(G), the processing for period 218 includes the portion of the audio signal corresponding to period 218 (the audio signal 260 consisting of the end of the utterance 202 and the subsequent silent portion), an additional section 262 which is the latter half of the audio signal 250 in the preceding period 216 (0.5 seconds), and a silent portion 264 that is added immediately before that.

In this embodiment, except for the first processing step, before VAD, the audio signal is processed by adding not only the period targeted for speech segment detection, but also the latter half of the audio signal from the immediately preceding period and a silent portion of a constant sound. This audio signal is then fed to VAD. As will be described later, this allows for highly accurate detection of speech segments with low latency. Since the immediately preceding target segment does not exist in the first processing step, only the silent portion is added to the target segment for processing.

In this embodiment, the latter 0.5 seconds of the previously processed audio signal are added to the beginning of the audio signal to be processed. However, this does not limit the invention. The length of the added audio signal does not need to be constant. Furthermore, the length of the silent portion added to the beginning does not need to be constant. However, in light of ease of implementation, it is desirable to make these lengths constant as in this embodiment. Also, the length of the added audio signal is not limited to 0.5 seconds. Here, if the added audio signal is longer than 0 seconds, it can be expected that the accuracy will improve compared to not adding anything, but it is thought that there is a limit to the improvement in accuracy even if it is longer than 0.5 seconds. Furthermore, it is thought that this length will also change depending on the audio being processed. Therefore, it is desirable that the length of the added audio signal be greater than 0 seconds and 0.5 seconds or less, preferably greater than 0.1 seconds and 0.45 seconds or less, and even more preferably greater than 0.2 seconds and 0.4 seconds or less. Also, the length of the silent portion added to the beginning is not limited to 0.1 seconds, and may be greater or less than that. However, the added silent portion must be greater than 0 seconds. If the length of the silent portion is too long, the latency will increase, so it is undesirable for it to be longer than 0.2 seconds. Therefore, it is desirable for the silent portion added at the beginning to be longer than 0 seconds but 0.2 seconds or less, and preferably longer than 0.05 seconds but 0.15 seconds or less. Furthermore, it is desirable that the added silent portion be shorter than the length of the portion added at the beginning of the target section in the preceding audio signal. This pre-processing will be described later.

Thus, after preprocessing the audio signals for periods 210 and 212 and each subsequent period, the detection of audible sections is performed. Any existing method may be used for this audible section detection process.

When detecting audible segments in an audio signal that has undergone the preprocessing described above, it is possible that longer audible segments than those included in the original target segment may be detected. Therefore, before performing LID (Limited Identification), it is necessary to remove the extra detected audible segments. This method will be explained with reference to Figure 5.

Figure 5(A) shows the same audio signal as shown in Figure 4(A), specifically the period 212 to 216. The following describes the speech section correction process for periods 214 and 216. Referring to Figures 5(B) and 5(C), the target for detecting audible sections in period 214 is the portion of the audio signal corresponding to period 214 (audio signal 240). As previously mentioned, as a preprocessing step for detecting audible sections in audio signal 240, the additional section 242 of audio signal 230 in period 212 is added to the beginning of audio signal 240. Furthermore, a silent section 244 is added immediately before this to detect audible sections. Referring further to Figure 5(D), for audio signal 250 in period 216, the additional section 252, which is the latter half of audio signal 240, is added, and a silent section 254 is added before this to detect audible sections. Therefore, in the case of period 214, as shown in Figure 5(F), the detected audible portion 300 may include not only the audible portion 302 corresponding to the audio signal 240, but also the audible portion 304 corresponding to the audible portion 234 at the end of period 212 (Figure 5(B)). During correction, if the audible portion 234 was detected in the processing of the audio signal one second prior and has already been transmitted to the control unit 74 and the speech segment detection unit 76, then the audible portion 304 is not transmitted, and only the audible portion 302 is transmitted to the control unit 74 and the speech segment detection unit 76, as shown in Figure 5(G). If the audible portion 304 was not detected in the processing of the audio signal one second prior and has not been transmitted to the control unit 74 and the speech segment detection unit 76, then both the audible portion 304 and the audible portion 302 are transmitted to the control unit 74 and the speech segment detection unit 76.

Similarly, in the case of period 216, as shown in Figure 5(H), the detected audible portion 310 includes not only the audible portion 312 included in the audio signal 250, but also the audible portion 314 at the end of period 214. Therefore, the same processing as for the audible portion 304 is performed on this audible portion 314. That is, if the audible portion 314 was detected in the processing one second prior and transmitted to the control unit 74 and the speech segment detection unit 76, the correction removes the audible portion 314 from the audible portion 310, leaving only the audible portion 312, as shown in Figure 5(I). If the audible portion 314 was not transmitted to the control unit 74 and the speech segment detection unit 76 in the processing one second prior, the audible portion 314 is transmitted to the control unit 74 and the speech segment detection unit 76 together with the audible portion 312. In this example, as shown in Figure 5(G), the portion corresponding to the audible portion 314 has been detected and transmitted as an audible portion. Therefore, the audible portion 314 is deleted, and only the audible portion 312 is transmitted.

The program configuration for performing this correction will be described later.

I. LID Processing Referring to Figure 6, the language identification processing (LID) in this embodiment will be described. Here again, as shown in Figure 6(A), periods 210, 212, 214, and 216 will be used as examples.

As a result of the processing shown in Figure 6(A), as shown in Figure 6(B), audible portions 350 and 352 corresponding to utterances 200 and 202 are detected. In this embodiment, when an audible portion 350 is detected, either a predetermined length from the beginning of the corresponding utterance 200 (for example, 1.5 seconds) or the end of the utterance signal is detected, and the utterance signal from the beginning up to that point is fed into the LID 78 shown in Figure 1. In the example shown in Figure 6(A), the utterance 200 is shorter than 1.5 seconds. Therefore, when the end of the utterance signal 360 (Figure 6(C)) corresponding to utterance 200 is detected, the entire signal is fed into the LID 78. Language discrimination itself does not require much time. Therefore, after the utterance signal 360 is fed into the LID 78, language discrimination is performed with a slight time delay, and as shown in Figure 6(G), the LID discrimination result 362 is obtained approximately 1.5 seconds after the start of utterance 200.

On the other hand, after the silent interval following utterance 200, when detection begins for the audible portion 352 corresponding to utterance 202, as shown in Figure 6(D), the audio signal 370 corresponding to utterance 202, from the beginning until 1.5 seconds or the end of the audio signal, whichever comes first, is fed into the LID 78. In this example, the first 1.5 seconds of the audio signal corresponding to utterance 202 are fed into the LID 78. As a result, the LID discrimination result 372 is obtained approximately 3.5 seconds after the start of utterance 200.

Furthermore, referring to Figure 6(E), if utterance 202 has not reached its end by 0.75 seconds from the beginning of utterance 202 (referred to as "0.75 seconds elapsed" for simplicity of explanation; the same applies hereafter), then when the earlier of 1.5 seconds from the 0.75-second mark to the end of the speech signal is detected, the entire speech signal from 0.75 seconds elapsed up to that point is fed into LID 78. In this example, the end of utterance 202 has not been reached at the 1.5-second mark from the 0.75-second mark. Therefore, the speech signal 380 from 0.75 seconds elapsed to 2.25 seconds elapsed is fed into LID 78. As a result, the LID discrimination result 382 is obtained as shown in Figure 6(G) approximately 4.2 seconds after the start time of utterance 200. Finally, referring to Figure 6(F), when the earlier of 1.5 seconds elapsed or the end of utterance 202 is detected, the audio signal 390 from 1.5 seconds elapsed to that point (the end of utterance 202 in this example) is fed into the LID 78. As a result, the LID discrimination result 392 is obtained as shown in Figure 6(G) when 4.3 seconds have elapsed from the start of utterance 200.

In this embodiment, the speech signal is input to the LID 78 starting from a fixed time interval (0.75 seconds in this example) from the beginning of the utterance section, and continuing until a predetermined time (1.5 seconds in this example) or the end of the utterance section, whichever comes first, is detected. The result is then obtained. As a result, when the utterance is approximately 1.5 seconds or shorter, the language discrimination result is obtained almost simultaneously with the end of the utterance. When the utterance is longer than 1.5 seconds, the first language discrimination result is obtained approximately 1.7 seconds after the beginning of the utterance, and thereafter, discrimination results are obtained at intervals of approximately 0.75 seconds. However, discrimination results are often obtained at shorter intervals at the end of the utterance.

As multiple language classification results are obtained for a single utterance, these results may contradict each other. Even in such cases, it is necessary to classify the language. To achieve this, the final result may be determined by, for example, a majority vote of the classification results, or by selecting the language with the highest score (or probability) obtained along with the classification result as the output of the neural network, or, if there are multiple identical results, the language with the highest average score. In this embodiment, the language with the highest average score obtained for each language for each utterance is selected as the final result. A specific example will be described later with reference to Figure 7.

C. Switching of Speech Recognition Devices As mentioned above, the multilingual ASR processing unit 80 shown in Figure 1 is capable of executing speech recognition processing for multiple languages in parallel. However, not only that, the multilingual ASR processing unit 80 is also capable of executing multiple speech recognition processes for each language, and switches between them as appropriate. This is because speech recognition processing requires a large amount of computation and takes time, so it may be necessary to execute speech recognition for subsequent utterances in parallel while speech recognition of one utterance is being performed. In particular, in this embodiment, for a given utterance, multiple speech recognition processing units each treat it as a different language and execute speech recognition processing. Speech recognition processing requires considerable computation even for speech recognition in the correct language, and the amount of computation increases even further for speech recognition in the wrong language. Therefore, if there are only a limited number of speech recognition processing units for each language, it is possible that one of the necessary speech recognition processing units will become busy when needed, making it impossible to execute speech recognition processing. Therefore, in this embodiment, three speech recognition processing units are provided for each language, and one of these that is idle is selected to execute speech recognition processing.

For example, suppose that the audible portions 350 and 352 shown in Figure 7(B) are detected for utterances 200 and 202 shown in Figure 7(A). As explained using Figure 6, the LID discrimination result 362 is obtained after the end of utterance 200. For utterance 202, the LID discrimination results 372, 382, and 392 are obtained in this order. It is not generally guaranteed that utterances 200 and utterance 202 are utterances of the same language. In the example shown here, as shown in Figure 7(C), the LID discrimination result 362 for utterance 200 is Japanese (JA), and the LID discrimination results 372, 382, and 392 for utterance 202 are Japanese, English (EN), and English, respectively. The language of utterance 202 is ultimately determined to be English based on the average score of the already processed portion of the utterance section, but in any case, the relationship between the language discrimination result for utterance 200 and the language discrimination result for utterance 202 is uncertain. Therefore, it is necessary to appropriately switch the language used for speech recognition of utterance 200 and speech recognition of utterance 202. In this embodiment, as shown in Figure 7(D), after the end of the previous utterance (for example, utterance 200) is detected, the point at which a 0.5-second silent interval 422 is detected is considered the utterance switching point, or utterance segment 420, and the speech recognition processing unit used before and after this point is switched for each language. Therefore, even if the languages of utterances 200 and 202 are the same language, the speech recognition processing unit will be switched if there is a silent interval of 0.5 seconds or more between them. This has the effect that, for example, even if the speech recognition processing for utterance 200 takes time, the speech recognition processing for the subsequent utterance 202 in each language can be started at the same time as the audible portion 352 is detected. Note that if the next utterance starts after a silent interval of less than 0.5 seconds following the utterance, these two utterances are considered as one utterance, and the same speech recognition processing unit performs speech recognition processing for each language. The speech recognition processing used here outputs the results of speech recognition sequentially.

In the example shown in Figure 7(C), three LID discrimination results are obtained as a result of language discrimination of the utterance 202: LID discrimination results 372, 382, and 392. When LID discrimination result 372 is obtained, LID discrimination results 382 and 392 have not yet been obtained. Therefore, for example, the speech recognition result corresponding to the language represented by LID discrimination result 372 is output. If, when LID discrimination result 382 is obtained, the result is another language and its score is higher than the score of LID discrimination result 372, the language of the utterance 202 will change. In such cases, the language of the output (the text displayed on the screen) will change midway through the speech recognition process, from the speech recognition result selected using LID discrimination result 372 to the speech recognition result selected using LID discrimination result 382. The same applies when LID discrimination result 392 is obtained.

More specifically, language discrimination is performed as follows. For example, let's assume there are 10 languages to be discriminated against. Referring to Figure 7(C), when the LID discrimination result 372 for utterance 202 is obtained, one score is obtained for each of the 10 languages. Since there is only one score per language, the language with the highest score (e.g., Japanese (JA)) is selected. As a result, if an intermediate speech recognition result for Japanese has been obtained at the time of the LID discrimination result 372, the Japanese speech recognition result is output at the time of the LID discrimination result 372. If a Japanese speech recognition result has not been obtained at the time of the LID discrimination result 372, the speech recognition result is output as soon as it is obtained.

On the other hand, at the point when the LID discrimination result 382 is obtained, there are two scores for each language: the score obtained in the LID discrimination result 372 and the score obtained in the LID discrimination result 382. In this embodiment, the average of these two scores obtained for each language is taken, and the language with the highest average score is selected. For example, even if the score for Japanese was highest at the time of the LID discrimination result 372, if the average score calculated at the time of the LID discrimination result 382 shows that English (EN) is the highest, the language identification result for the utterance 202 will switch from Japanese to English. Therefore, when the LID discrimination result 382 is obtained, the speech recognition result switches from Japanese to English.

The same applies at the point of obtaining the final LID discrimination result 392 for utterance 202. At the point where the LID discrimination result 392 is obtained, three scores are acquired for each language. For each language, the average of these three scores is calculated. The language with the highest average score is then selected. As shown in Figure 7(C), if the judgment result based on the average scores obtained for each language by the LID discrimination results 372, 382, and 392 is also English, then the English speech recognition result is output continuously, and the output is updated each time an intermediate speech recognition result is obtained.

(4) Program Structure A. Overall Control Structure The control structure of the computer program for operating the computer in order to function as described above is described below. Figure 8 shows the overall structure. The following shows how the entire multilingual live speech recognition device 50 described above is realized by computer hardware and computer programs, but it is also possible to realize some or all of these with dedicated hardware. Furthermore, by programming a general-purpose computer to operate according to the control structure described below, the general-purpose computer can function as a dedicated multilingual live speech recognition device.

Referring to Figure 8, this program includes step 450, which starts each of the VAD thread, the LID thread, and the multiple speech recognition threads provided for each of the multiple languages; step 452, which establishes a connection with each of these threads; and step 454, which starts inputting the speech signal to be recognized. Here, a thread refers to multiple programs operating under a single process, sharing an address space.

This program further includes: step 456, which stores the input audio signal in a buffer and, each time a predetermined amount (1 second in this embodiment) of audio signal is stored in the buffer, performs preprocessing on the audio signal as described above with reference to Figure 4 and outputs it to the VAD thread; step 458, following step 456, which performs audible section correction processing as described above with reference to Figure 5 from the VAD output to obtain an output for determining the audible section; step 460, which, based on the output for determining the audible section obtained in step 458, detects any silent section of 0.5 seconds or more after the end of the audible section as a speech segment; step 462, which branches the control flow according to the result of the processing in step 460; and step 466, which, if the determination in step 462 is affirmative, switches the idle thread within the ASR for speech recognition of the next audible section's audio signal for each language.

This program further includes step 468, which is executed when the determination in step 462 is negative, and when the determination in step 462 is positive and the processing in step 466 is completed, and which provides audio data to each of the ASR threads switched within the multilingual ASR processing unit 80 and to the LID 78; step 470, which executes the respective processes in each ASR thread and LID 78 after step 468; step 472, which waits until the language identification result from LID 78 is obtained; and step 474, which, after the language identification result is obtained, displays the audio recognition result from the ASR thread of the language identified by LID 78 from the output of each ASR thread for each language, changing the previous display, and returns control to step 456.

I. VAD
(a) Preprocessing Figure 9 is a flowchart showing the control structure of a program that implements preprocessing in VAD. Referring to Figure 9, this program includes a step 500 which assigns zero to the variable i, a step 502 which reads the input data and stores it in buffer [i], and a step 504 which repeats step 502 until the audio data stored in the buffer amounts to one second. In this embodiment, at least two buffers, buffer [0] and buffer [1], are prepared, and when the value of i is even (i % 2 = 0), the audio data is stored in buffer [0], and when the value of i is odd (i % 2 = 1), the audio data is stored in buffer [1]. Of course, audio data may be stored in a different way than this. For example, audio data may be stored in a ring buffer, and one second of audio data may be read from its starting position. When the audio data exceeds the capacity of the ring buffer, the audio data stored in the ring buffer may be overwritten. The symbol "%" represents the modulo operator.

This program further includes steps 506, which are executed in response to the determination in step 504 being positive, i.e., when one second of audio data has been accumulated in the buffer, and which add a 0.1 second silent interval to the beginning of the audio data; step 508, which provides the audio data processed in step 506 to the VAD; step 510, which adds 1 to the value of variable i; step 512, which reads the input audio data and accumulates it in buffer [i%2]; and step 514, which repeats step 512 until one second of audio data has been accumulated in buffer [i%2]. Steps 512 and 514 are substantially the same as the processing in steps 502 and 504. However, in step 502, data is accumulated in buffer [0], whereas in step 512, buffer [1] and buffer [0] are switched and used depending on whether the value of variable i is odd or even.

This program further includes, when it is determined in step 514 that one second's worth of audio data has been accumulated in buffer [i%2], step 516 appends the latter 0.5 seconds of audio data from buffer [(i-1)%2] to the beginning of the audio data in buffer [i%2], step 518 appends a 0.1 second silent interval to the beginning of that, similar to step 506, and step 520 provides the audio data from buffer [i%2] processed in steps 516 and 518 to the VAD and returns control to step 510. In response to the provision of this audio data, the VAD detects the audio interval from the beginning of the audio data. The audio interval is output from the VAD as a pair of [start time, end time].

(i) Correction The output of the VAD is for audio data after preprocessing as shown in Figure 9. This audio data includes not only the audio data of the original target section, but also the last 0.5 seconds of audio data from the previous target section, except for the first second of audio data. Furthermore, a 0.1 second silent section is added to the beginning of this audio data. Therefore, there is a possibility that audio sections other than the audio sections of the original target section are included in the output of the VAD. If such audio sections have already been sent to the processing buffer, they are removed from the output of the VAD in the correction process described below; otherwise, they are sent as is to the control unit 74 and the speech segment detection unit 76.

Referring to Figure 10, this program includes a step 550 to receive VAD output, a step 552 to delete any audible intervals indicated by the VAD output received in step 550 that are prior to the target 1 second (a 0.1-second silent interval), and a step 554 to output a pair of start and end times for the audible interval determined by VAD after the processing in step 552 is completed. Specifically, in step 552, if there is a hypothetical audible interval within the added 0.1-second silent interval, it is deleted. If there is an audible interval that continues from the added silent interval to the 1-second target interval, the start time of the audible interval is corrected to the beginning of the target 1 second. Then, if there is a silent interval of less than 0.5 seconds within the 1-second target interval, it is considered an audible interval.

This program further includes, after step 554, step 556, which receives the VAD output in the same manner as in step 550; step 558, which corrects the VAD output received in step 556; and step 560, which outputs the audible section detection data corrected by the VAD in step 558 and returns control to step 556. In step 558, among the audible sections detected in the preceding 0.5 seconds of audio data added before the target 1-second section, those already transmitted to the control unit 74 and the speech segment detection unit 76 in the previous processing are deleted. Furthermore, any audible sections detected during the 0.1-second silent section added immediately before that are also deleted.

By performing this correction, it is possible to detect audible segments within each one-second interval of the audio data.

C. ASR Switching In this embodiment, multiple ASR threads are launched for each language targeted for speech recognition. By switching these ASR threads for each utterance to perform speech recognition, even if the speech recognition process for one utterance is lengthy, the speech recognition of the next utterance can be executed in parallel. In particular, when recognizing speech in a language that does not correspond to the language of the audio data, it is expected that the processing will be lengthy. Therefore, it is desirable to prepare multiple ASR threads for each of the multiple languages, as in this embodiment, and switch them at the end of the utterance. In this embodiment, when a silent section of 0.5 seconds or more is detected by the VAD after a sounded section, the ASR thread is switched at that point, and the next audio data is provided to the switched ASR thread.

Referring to Figure 11, the program for switching ASRs includes step 600, which performs step 602 described below for each of the languages that may be processed.

Step 602 includes step 610, which selects the next ASR thread after the previously selected ASR thread for that language; step 612, which determines whether the selected ASR is idling and branches the control flow accordingly; step 616, which attempts to select the next ASR thread if the determination in step 612 is negative; step 618, which branches the control flow according to whether there is an available thread among the threads to be selected; step 620, which generates a new thread if the determination in step 618 is negative, i.e., if there are no more available threads; and step 621, which switches the input destination of the audio data to this newly generated ASR thread. If the determination in step 618 is positive, that thread is selected and control returns to step 612. After step 621, control proceeds to step 622, which will be described later.

This program further includes step 614, which switches the input destination of the audio data to the selected ASR thread if the determination in step 612 is affirmative; step 622, which is executed after step 614 and after step 620, and stores the thread ID of the switched ASR thread in relation to the utterance of the audio data being input to that ASR thread; and step 624, which changes the state of the ASR thread switched in step 622 to busy.

For a given language, it is desirable to have at least two ASR threads running, preferably three or more. With at least two ASR threads, they can be operated alternately. In this case, when speech recognition for the next utterance needs to begin while the two preceding utterances are being processed by the two ASR threads, a new ASR thread is created and assigned to speech recognition for the new audio data. If three or more ASR threads become busy, additional threads can be launched.

Furthermore, the maximum number of ASR threads launched for each language, and the maximum number of target languages, depend on the performance of the computer on which this program runs, and therefore cannot be generalized.

E. Control of LID In this embodiment, LID is performed according to the following control. As already mentioned, the LID processing itself is performed by a language recognition model consisting of a trained neural network. Here, we will explain the control structure of the program that realizes when and what kind of audio data is input to this neural network to obtain the LID result.

Referring to Figure 12, this program includes the following steps: waiting until a sounded section to be processed is detected; reading the data of the sounded section from memory when a sounded section is detected (step 700); setting the start position of the audio data reading to the beginning of the audio data read in step 700 (step 702); reading audio data for a period of 1.5 seconds from the reading start position or the shorter of the time to the end of the sounded section (step 704); feeding the read audio data into the language identification model (step 706); advancing the reading start position by 0.7 seconds (step 708); determining whether the reading start position set in step 708 is before the end time of the sounded section (step 710); and, if the determination is positive, returning control to step 704 (step 712); and, if the determination in step 710 is negative, i.e., when processing has finished up to the end of the sounded section being processed, advancing the target to the next sounded section and returning the control flow to step 700 (step 712).

O. Display Update The speech recognition result is displayed by a program having the following control structure. Referring to Figure 13, this program includes a step 750 that branches the control flow according to whether or not there is an output of a language identification result by LID; a step 752 that, when the determination in step 750 is affirmative, stores the language ID of the language indicated by the output of LID in memory as information representing the current language; and a step 754 that, after step 752 and when the determination in step 750 is negative, displays the recognition result output from the ASR thread corresponding to the current language ID among the multiple ASR threads operating in response to the current utterance and returns control to step 750.

2. Operation (1) Startup Referring to Figure 8, in step 450, the VAD thread, the LID thread, and each of the multiple speech recognition threads provided for each of the multiple languages are started. These correspond to the speech interval detection unit 72, LID 78, and multilingual ASR processing unit 80 shown in Figure 1, respectively. In step 452, the main routine (corresponding to the control unit 74 in Figure 1) establishes connections with each of these threads.

In step 454, when the audio data input 52 is provided, the buffer 70 begins to store the audio data. In step 456, the process shown in Figure 9 is executed. That is, the audio data is stored in buffer [0] (step 502).

(2) Preprocessing for the initial input Once the first second of audio data has been accumulated (the determination in step 504 is positive), a 0.1 second silent interval is added to the beginning of the audio data (step 506), and the audio data is fed into the VAD (step 508). In response to this input, the VAD 102 begins outputting information representing the audio interval (a pair of start time and end time).

In step 510, the storage location for the audio data is changed to buffer [1]. Processing for the second and subsequent inputs is then performed.

(3) Preprocessing for the second and subsequent inputs For the second and subsequent inputs, the audio data input in buffer [0] or buffer [1] is stored in step 512. When 1 second of audio data has been stored (the determination in step 514 is positive), in step 516, the latter 0.5 seconds of audio data stored in a different buffer (buffer [(i-1)%2]) from the buffer currently being used for storage (buffer [i%2]) is added to the beginning of the audio data stored in buffer [i%2]. In step 518, a 0.1 second silent interval is added to the beginning of that. In step 520, this audio data is fed into the VAD.

This process will be repeated as long as there is audio data input.

(4) Correction of VAD output The VAD 102 shown in Figure 1 receives audio data that has been preprocessed by the preprocessing unit 100 and outputs a pair of [start time, end time] indicating the audible section within the audio data. This output may include silent sections added by preprocessing and audible sections detected in the latter half of the audio data for the immediately preceding period. The correction unit 104 executes a program whose control structure is shown in Figure 10 to delete any audible sections that have already been transmitted to the control unit 74 and the speech segment detection unit 76.

(5) Control of LID and switching of ASR Referring to Figure 1, the control unit 74 provides the audio data of the audible section detected by the VAD 102 to both the LID 78 and the multilingual ASR processing unit 80. The LID 78 is controlled by a program having the control structure shown in Figure 12, and for each input audible section, it reads out audio data with a length of 1.5 seconds and a shift length of 0.7 seconds and feeds it into the language identification model. This process is carried out for each audible section until the end of the audible section. The maximum length of the audio fed into the language identification model here is 1.5 seconds, and if the utterance ends before that, it will be the length to the end of the utterance.

The language recognition model performs language recognition processing on the 1.5-second audio data described above, and outputs the result when it is obtained. If there is further input of a sounded section, the language recognition model begins processing that audio data. Because this processing is repeated, as shown in Figure 6, the language recognition result is output at least once for each sounded section, and two or more times if the sounded section is long. The multilingual ASR processing unit 80 receives this language recognition result and displays the speech recognition result for the language corresponding to the language recognition result.

The multilingual ASR processing unit 80 operates as follows. In this embodiment, three ASR threads are generated for each language. The control unit 74 inputs the audio data to the ASR processing unit (corresponding to an ASR thread) for each language within the multilingual ASR processing unit 80. At the start of the program, one of these ASR threads is selected as the input destination for the audio data. In other words, the first audible section is processed by the ASR thread selected as the input destination for each language. At this point, the language represented by the audio is unknown. The ASR thread that receives the audio data performs speech recognition, assuming that the audio data belongs to the language it is responsible for. Once the language identification result is obtained, the multilingual ASR processing unit 80 selects the output of the ASR thread for the language identified by the language identification result from among the outputs of the ASR threads for multiple languages. The result display control unit 82, in response to the output of the LID discrimination result from the LID 78, displays the speech recognition result output from the multilingual ASR processing unit 80 at that time as display 54 in Figure 1 on the display device, or transmits it as text data to a remote computer.

The language identification result may be output several times, as shown in Figures 6(C) to (G), if the voiced section is long. Since the ASR thread outputs the speech recognition results sequentially, the output is updated each time. If the results of multiple language identification attempts differ, as in the language discrimination result for the voiced section 352 in Figure 7(C), the speech recognition result for the language selected by majority vote or the most confident discrimination result is output. For example, when determining the language by majority vote, in the example shown in Figure 7(C), the speech recognition result is initially displayed in Japanese, but then switches to English midway through.

The speech segment detection unit 76 shown in Figure 1, upon detecting a 0.5-second silent interval following a spoken interval, provides a speech segment detection signal to the multilingual ASR processing unit 80. Upon receiving this speech segment detection signal, the multilingual ASR processing unit 80 switches the input destination of the audio data for each language from the previously active ASR thread to another idle ASR thread. At this point, it is highly likely that two ASR threads are running. However, the leading ASR thread transitions to idle mode once speech recognition is complete. In this manner, the ASR thread receiving the audio signal is switched in multiple ASR threads for each language, triggered by speech segments.

Section 2 Evaluation (1) Latency Calculation Method As in the above embodiment, in order to convert utterances into text and display them immediately after they are spoken or to input them into automatic translation, it is necessary to minimize the time from when an utterance is spoken until that utterance is recognized as speech and converted into text, i.e., latency. However, conventional speech recognition systems have been evaluated based on latency, accuracy (word error rate: WER), and speed (real-time factor: RTF) on an utterance basis. While these evaluation metrics are useful as metrics for evaluating the performance of speech recognition systems, they are insufficient as evaluation metrics for so-called live speech recognition processing, which involves continuously recognizing multiple consecutive utterances in parallel with those utterances.

For example, when evaluating latency on an utterance basis, longer utterances result in longer latency, so it's necessary to eliminate the influence of utterance length. Furthermore, when pre-aligning utterance audio data and its transcription, and then comparing the speech recognition results with the pre-alignment as an evaluation metric, there's a problem with large calculation errors due to factors like misrecognition.

In live speech recognition, low latency is a critical goal. The time required for segmentation, language identification, speaker identification, and speech recognition all affect latency, and these effects are complex. To reflect the impact of system evaluation and improvements to each technology, this embodiment uses the temporary output of speech results from the speech recognition engine during speech recognition processing to calculate word-based latency. Here, the temporary output of speech results refers to the temporary speech recognition result that the speech recognition engine outputs externally when it performs speech recognition on a speech signal of a certain duration.

For example, consider the case where a voice signal from client 800 is performed live speech recognition on server 802, referring to Figure 14. First, at the beginning of the live speech recognition process, client 800 sends a pause signal 810 to server 802, followed by data 812, 814, 816, and 820 sent to server 802 at regular intervals. Server 802 performs speech recognition on this data, and each time it performs speech recognition on a voice signal of a certain length, it sends temporary recognition results 818, 822, 826, and 834 to client 800. In the example shown in Figure 14, server 802 further sends the final recognition result 836 to client 800 when recognition is complete. Conversely, client 800 sends the voice data of the utterance following data 820 to server 802 as data 828, 830, and 832. The latency calculation proposed here uses temporary recognition results such as temporary recognition result 818.

Furthermore, server 802 may send not only temporary recognition results such as temporary recognition result 818, but also something called a partial recognition result 824 to client 800. In this context, the partial recognition result 824 is a type of temporary recognition result, similar to temporary recognition result 818, but it is a speech recognition result that is output only when the speech recognition engine satisfies internal conditions that are uncontrollable from the outside.

In this embodiment, the latency of live speech recognition is measured using the recognition results output from the speech recognition engine during the speech recognition process. The following method assumes that the ASR (Audio-Speech Recognition) function outputs the speech duration for each word, and the latency is calculated using this information as follows.

(a) From the words output in the temporary speech recognition results (temporary recognition results) and partial speech recognition results, exclude those recognized in the previous temporary recognition result.

(i) Sum the time from the utterance of those words to the output of the recognition result, and divide by the number of words.

(c) The latency is calculated by subtracting the value obtained in (b) above from the speech recognition result output time.

(2) Experimental Evaluation Referring to Figure 15, the method for calculating latency will be explained using speech recognition of the utterance "please wait here. I'm going to look for a rope or something." as an example. The first ASR output is for "please wait here." Including the silent intervals at the beginning and end, the utterance times of each word are 0.00 seconds, 0.60 seconds, 0.81 seconds, 1.17 seconds, and 1.35 seconds, respectively, and the speech recognition result output time is 3.50 seconds. To calculate the latency, the utterance lengths of the three words indicated by the double underline (0.60 seconds, 0.81 seconds, and 1.17 seconds) are added together, and the value is divided by the number of words (3). The obtained value is subtracted from the speech recognition result output time of 3.50 seconds. As a result, a value of 2.64 is obtained. This is the word-based latency of the speech recognition processing for the first utterance.

The second ASR output here is the speech recognition result for the entire utterance mentioned earlier. The output time is 4.32 seconds. From this output, excluding the underlined word from the first output, the utterance lengths of the remaining nine words (indicated by double underlines) are added together. This value is then divided by the number of words (9), and this result (approximately 2.24) is subtracted from the output time of 4.32 seconds. The result is 2.08. This is the word-based latency for the second sentence of the utterance.

Note that here, latency is calculated using only the temporary recognition result. However, it goes without saying that latency can also be calculated using the final recognition result.

Figure 17 shows the live speech recognition processing according to the above embodiment, evaluated using the method calculated by this method, in comparison with other metrics, and Figure 16 shows an example of the utterance used for measurement.

In the table shown in Figure 17, the upper row shows the evaluation of the ASR unit alone, and the lower row shows the evaluation of the live speech recognition processing. The LID value indicates accuracy. The evaluation values for English and Japanese (four locations) are all WER. The second row of the column labeled "Latency" shows the measurements taken in the above embodiment.

As can be seen from Figure 17, live speech recognition shows higher accuracy in LID than ASR alone. Furthermore, in both English and Japanese, live speech recognition processing showed slightly lower performance than ASR alone, which is unaffected by language discrimination. Latency could not be measured in the evaluation of ASR alone and could only be measured during live processing.

Third Modification In the above embodiment, the same number of ASR threads were used for each language. However, this invention is not limited to such embodiments. Different numbers of ASR threads may be generated for each language. Also, each thread is generated at the beginning of the program. However, this is not limiting, and each thread may be generated when needed. For example, if all the threads generated for a certain language become busy and it becomes necessary to perform speech recognition for the next utterance, instead of interrupting the processing of any of the busy threads, a new thread may be generated.

Furthermore, in the above embodiment, the results of the live speech recognition processing are displayed on the screen with a slight delay after the utterance. However, the invention is not limited to this embodiment. For example, the results of the speech recognition processing may be transmitted to a remote computer via communication, or they may be input to a live automatic translation device and the translation results output in some form. For example, multiple speech sequences in multiple languages obtained from multiple venues can be processed separately by multiple processes running on a single server, and the results can be translated into multiple languages and output. Similar to the language identification part, by introducing a multi-speaker identification model, speaker information, not just language, can be added to the speech recognition results.

In the above embodiment, a speech segment is detected when there is a silent portion of 0.5 seconds or more. However, this invention is not limited to such embodiments. For example, a speech segment may be detected when a change in the language of an utterance is detected based on the result of language identification, or when a change in the speaker is detected using the result of speaker identification.

In the above embodiment, multiple threads are operated using a single CPU in a single computer system. However, this invention is not limited to such an embodiment. If a CPU has multiple cores, the above processing can be achieved by generating threads separately on each core. Furthermore, by utilizing multiple independent CPUs, the multilingual live speech recognition system according to the above embodiment can also be implemented as a distributed system.

[Note]
Other aspects of the above embodiment are described below.

(1) The information output by the language identification model includes multiple scores indicating the probability that the speech signal in a sub-interval is in one of several languages. The language determination means may include a condition verification means that, in response to the language identification model outputting a score for each sub-interval, outputs an identifier for the language among the multiple languages whose score satisfies predetermined conditions.

(2) The condition verification means may output the language identifier for the language that has produced the maximum score most frequently for each subinterval.

(3) The condition verification means may output the language identifier with the highest average score output by the language identification model for each subinterval.

(4) The signal addition means may include a tail portion storage means for storing a second predetermined length at the end of at least the immediately preceding target section; a leading section processing means for adding a first predetermined length of silent section to the beginning of the first target section and outputting it; and a subsequent section processing means for adding the second predetermined length at the end of the immediately preceding target section stored in the tail portion storage means to the beginning of each target section after the first target section, and further adding a first predetermined length of silent section immediately before that and outputting it. The second predetermined length may be longer than the first predetermined length and shorter than the length of the target section.

The embodiments disclosed herein are merely illustrative, and the present invention is not limited to those embodiments described above. The scope of the present invention is defined by the claims, with reference to the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wording contained herein.

50 Multilingual live speech recognition device 70 Buffer 72 Speech segment detection unit 74 Control unit 76 Speech segment detection unit 78 LID
80 Multilingual ASR processing unit 82 Result display control unit 100 Pre-processing unit 102 VAD
104 Correction Units 200, 202 Speech 210, 212, 214, 216, 218 Period 220, 232, 234, 300, 302, 304, 310, 312, 314, 350, 352 Audible Parts 222, 238, 244, 254, 264 Silent Parts 230, 236, 240, 250, 260, 360, 370, 380, 390 Speech Signals 242, 252, 262 Additional Sections 362, 372, 382, 392 LID Discrimination Result 420 Speech Separation 422 Silent Sections 812, 814, 816, 820, 828, 830, 832 Data 818, 822, 826, 834 Temporary Recognition Result 824 Partial recognition result 836 Final recognition result

Claims (5)

A speech segment detection means for detecting the start and end of a speech segment in an audio signal,
A speech segment detection means outputs a speech segmentation signal indicating a speech segment in response to the end of a speech segment detected by the speech segment detection means and the presence of a silent period of a predetermined time or longer.
Language identification means for identifying the language of the utterance in the immediately preceding utterance segment and outputting a language identifier in response to the aforementioned utterance segment signal,
Speech recognition means including a plurality of speech recognition processing means for a plurality of different languages, for performing speech recognition on the speech signal of the speech segment in response to the detection of the start of a speech segment by the speech segment detection means ,
The system includes a selection means for selecting and outputting the output of the speech recognition processing means for the language indicated by the identifier, from among the plurality of speech recognition processing means, in response to the identifier.
The aforementioned speech interval detection means is
A division means for dividing the aforementioned audio signal into target sections of a predetermined length,
A signal adding means for adding an additional signal to each of the aforementioned target sections, which includes at least a first predetermined length of silent section immediately preceding it,
A sound section detection means for detecting a sound section included in the target section to which the added signal has been added by the signal adding means, distinguishing it from a silent section;
A speech recognition device comprising correction means for correcting a sounded section by deleting the sounded section corresponding to the additional signal from the sounded section detected by the sounded section detection means. The language identification means is
In response to the detection of the end of a speech segment by the speech segment detection means, a sub-segment signal generation means generates a sub-segment audio signal of a predetermined length and shift amount from the audio signal of the speech segment,
A language identification model that receives the audio signal from each of the aforementioned sub-sections and outputs information indicating which of the plurality of languages the sub-section corresponds to,
The speech recognition device according to claim 1 , further comprising: language determination means that, in response to the information output by the language identification model, determines the language of the speech signal in the speech segment and outputs an identifier for that language. The speech recognition device according to claim 1 or 2, further comprising speaker identification means for identifying the speaker of the utterance in the immediately preceding speech segment in response to the utterance segmentation signal and providing information about the speaker to the output of the speech recognition means. Computers,
A speech segment detection means for detecting the speech segment of an audio signal,
A speech segment detection means outputs a speech segmentation signal indicating a speech segment in response to the end of a speech segment detected by the speech segment detection means and the presence of a silent period of a predetermined time or longer.
Language identification means for identifying the language of the utterance in the immediately preceding utterance segment and outputting a language identifier in response to the aforementioned utterance segment signal,
Speech recognition means including speech recognition processing means for multiple different languages, for performing speech recognition on the audio signals of the speech segments in response to the detection of a speech segment by the speech segment detection means ,
A computer program that, in response to the identifier, causes one of the plurality of speech recognition processing means to function as a selection means for selecting and outputting the output of the speech recognition processing means for the language indicated by the identifier,
The aforementioned speech interval detection means is
A division means for dividing the aforementioned audio signal into target sections of a predetermined length,
A signal adding means for adding an additional signal to each of the aforementioned target sections, which includes at least a first predetermined length of silent section immediately preceding it,
A sound section detection means for detecting a sound section included in the target section to which the added signal has been added by the signal adding means, distinguishing it from a silent section;
A computer program comprising correction means for correcting a sounded section by deleting the sounded section corresponding to the additional signal from the sounded section detected by the sounded section detection means. The computer detects the start and end of the speech interval of the audio signal,
The computer, in response to the end of the detected speech segment, outputting a speech segmentation signal indicating a speech segment,
A language identification step in which a computer, in response to the speech segmentation signal, identifies the language of the speech in the immediately preceding speech segment and outputs a language identifier,
The computer, in response to the detection of the start of the utterance segment, initiates speech recognition of the speech signal of the utterance segment using speech recognition processing means for each of several different languages ,
A speech recognition method comprising a selection step in which a computer, in response to an identifier, selects and outputs the output of a speech recognition processing means for the language indicated by the identifier from among a plurality of speech recognition processing means ,
The step of outputting the aforementioned speech segmentation signal is:
The steps include dividing the audio signal into target sections of a predetermined length,
A signal addition step is to add an additional signal to each of the aforementioned target sections, which includes at least a first predetermined length of silent section immediately preceding it.
The signal addition step includes a sounded section detection step for detecting a sounded section included in the target section to which the added signal has been added, distinguishing it from a silent section.
A speech recognition method comprising a correction step for correcting a sounded section by deleting the sounded section corresponding to the additional signal from the sounded section detected in the sounded section detection step.