JP7771015B2 - Speech recognition device, speech recognition method, and speech recognition program - Google Patents

Description

Embodiments of the present invention relate to a speech recognition device, a speech recognition method, and a speech recognition program.

Subtitle generation technology that utilizes speech recognition technology is known. Subtitle generation technology is used, for example, in the field of subtitling for television programs and the like. However, speech recognition can sometimes result in misrecognition due to background sounds, slip-ups, or speech containing technical terms. For this reason, users are encouraged to correct the speech recognition results and output the corrected results (see, for example, Non-Patent Document 1). Various techniques are being considered for improving speech recognition performance. For example, a speech recognition device has been disclosed that infers a topic from speech recognition results and uses the inferred topic to adjust the language score used in speech recognition (see, for example, Patent Document 1).

Patent Document 1 discloses a technology that extracts text corresponding to speech recognition results from topic information such as television program scripts and progress schedules, and obtains text related to the extracted text from external data on a website. Patent Document 1 also discloses a technology that uses text obtained from external data to adjust the language score used for speech recognition.

However, conventional technology adjusts language scores by obtaining text related to speech recognition results from external data on websites, making it difficult to apply to real-time subtitling for programs such as news programs, and in some cases preventing improvements in speech recognition performance.

Patent No. 5723711

NHK Science & Technology Research Laboratories, "NHK Science & Technology Research Laboratories R&D 2020 Summer Issue Commentary 02: Latest Trends in Automatic Subtitling for Live Broadcast Programs," [online], Special Issue on Image and Audio Processing Technology for Program Production Support, Summer 2020 Issue No. 182, [Retrieved April 28, 2022], Internet: <https://www.nhk.or.jp/strl/publica/rd/182/3.html>

The problem that this invention aims to solve is to provide a voice recognition device, a voice recognition method, and a voice recognition program that can improve voice recognition performance.

A speech recognition device according to an embodiment includes a recognition unit and an output control unit. The recognition unit generates a speech recognition result from speech data using topic data in which a topic appearance probability for each elapsed time for each of a plurality of topics that changes over time is registered in advance, a recognition model associated with each of the plurality of topics and including an adaptive language model for outputting a speech recognition result from speech data, and the elapsed time. The output control unit outputs the speech recognition result. The output control unit outputs the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit, and outputs the topic whose topic appearance probability is equal to or greater than a threshold to the display unit in a different display format from the topic whose topic appearance probability is less than the threshold.

FIG. 1 is a functional block diagram of a voice recognition device. Schematic diagram of the data structure of topic data. Schematic diagram of the data structure of topic data. FIG. Schematic diagram of a correction converter. FIG. 10 is a schematic diagram of the data structure of revision history data. FIG. 10 is a schematic diagram of the data structure of revision history data. Schematic diagram of a display screen. FIG. 10 is a flowchart showing the flow of information processing. FIG. 2 is a diagram showing an example of a hardware configuration.

The voice recognition device, voice recognition method, and voice recognition program are described in detail below with reference to the accompanying drawings.

Figure 1 is a functional block diagram of an example of a speech recognition device 10 according to this embodiment.

The speech recognition device 10 is an information processing device that performs speech recognition processing on input speech data and outputs the speech recognition results.

The speech recognition device 10 includes a control unit 20, a memory unit 30, a UI (user interface) unit 40, and a communication unit 50. The control unit 20, memory unit 30, UI unit 40, and communication unit 50 are connected to each other so that they can communicate with each other via a bus or the like.

The storage unit 30 stores various types of information. In this embodiment, the storage unit 30 stores topic data 31, recognition models 32, correction models 33, and correction history data 34. Details of these data and models will be described later.

The UI unit 40 includes a display unit 40A and an input unit 40B. The display unit 40A displays various types of information. The display unit 40A is, for example, a display. The input unit 40B accepts operational instructions from the user. The input unit 40B is, for example, a pointing device such as a mouse or touchpad, a keyboard, etc. The UI unit 40 may be a touch panel that integrates the display unit 40A and the input unit 40B. The UI unit 40 may also be configured to further include output functions such as a speaker, a projection device, etc.

The communication unit 50 is a communication interface for communicating with information processing devices external to the speech recognition device 10. For example, the communication unit 50 communicates with external information processing devices or electronic devices via a wired network such as Ethernet (registered trademark), or a wireless network such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark).

The control unit 20 executes information processing in the speech recognition device 10. The control unit 20 includes an acquisition unit 20A, a recognition unit 20B, a correction unit 20C, an output control unit 20D, a memory control unit 20E, an update unit 20F, and a topic data update unit 20G.

The acquisition unit 20A, recognition unit 20B, correction unit 20C, output control unit 20D, storage control unit 20E, update unit 20F, and topic data update unit 20G are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may also be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may also be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

In addition, the information stored in the memory unit 30 and at least some of the above-mentioned components included in the control unit 20 may be installed in an external information processing device communicatively connected to the voice recognition device 10.

The acquisition unit 20A acquires voice data. Voice data is data of voice. Voice may be, for example, voice spoken by a user, or voice output from various devices via speakers, etc. In this embodiment, we will assume that the voice data is data of voice spoken by a user.

For example, the acquisition unit 20A acquires audio data of continuous audio in a time series by sequentially digitizing real-time audio collected by a microphone (not shown) connected to the control unit 20. The acquisition unit 20A may also acquire audio data by reading audio data from an audio file that has been recorded in advance and stored in the storage unit 30, etc.

The recognition unit 20B generates speech recognition results from the speech data acquired by the acquisition unit 20A using the topic data 31, the recognition model 32, and the elapsed time.

Topic data 31 is pre-registered data that contains the topic appearance probability for each elapsed time for each of multiple topics that change over time.

A topic is information that indicates the subject of the content represented by the audio data.

Elapsed time is information that indicates the amount of time that has elapsed since a specific timing. For example, the elapsed time may be the time that elapsed from the timing at which speech starts in the voice data acquired by the acquisition unit 20A, the time that elapsed from the timing at which voice recognition by the control unit 20 starts, the time that elapsed from the first topic in the topic data 31, etc.

In this embodiment, we will explain an example in which the elapsed time is the time elapsed from the start of voice recognition of the voice data acquired by the acquisition unit 20A. Note that if the start points of the elapsed time used for the data to be processed differ, the control unit 20 adjusts the elapsed time using information related to the timestamp included in each data so that the start points match, and then executes the processes described below. In this embodiment, we will continue the explanation assuming that the start points of the elapsed time used for each data are the same.

Figure 2A is a schematic diagram showing an example of the data structure of topic data 31A. Topic data 31A is an example of topic data 31 when the audio data is Japanese audio data. As shown in Figure 2A, topic data 31A is a database that associates, for each elapsed time, topics that appear in the audio data at that elapsed time with the topic appearance probability of that topic at that elapsed time. Figure 2A shows an example in which topics and topic appearance probabilities are linked for each elapsed time of a specified period indicated in minutes.

Figure 2B is a schematic diagram showing an example of the data structure of topic data 31B. Topic data 31B is an example of topic data 31 when the audio data is Japanese audio data. In Figure 2B, the horizontal axis represents elapsed time, and the vertical axis represents the topic appearance probability. As shown in Figure 2B, topic data 31A may be data that graphically represents, for each elapsed time, the topics that appear in the audio data at that elapsed time and the topic appearance probability of that topic at that elapsed time. Figure 2B shows an example in which topics and topic appearance probabilities are linked for each elapsed time of a predetermined period indicated in minutes.

Figures 2A and 2B show examples of topics such as "General," "Sports," and "Weather." Figures 2A and 2B also show an example in which the topics with the highest topic appearance probability change over time in the order "General," "Sports," and "Weather."

The data format of the topic data 31 is not limited to a database or graph format. For example, the topic data 31 may be data that represents the topic appearance probability of a topic corresponding to elapsed time according to a commonly used probability distribution such as a normal distribution.

Note that Figures 2A and 2B show an example in which the elapsed time for each topic shown in topic data 31 is measured in minutes. However, the unit of elapsed time is not limited to minutes. For example, the unit of elapsed time for each topic shown in topic data 31 may be a period longer than the broadcast time of a typical television program. Specifically, the unit of elapsed time may be weekly, monthly, seasonal, event period, etc.

For example, during the Olympic Games, there is a high possibility that the topic "Olympics" will be featured in news programs and the like. For this reason, the control unit 20, for example, sets the duration of the Olympics, which is an example of an event period, as the elapsed time, and stores in advance in the storage unit 30 topic data 31 that specifies the topic "Olympics" as the topic with the highest topic appearance probability corresponding to that elapsed time, and the topic "sports" as the topic with the next highest topic appearance probability. The control unit 20 may also generate a general-purpose language model 32B and an adapted language model 32C for the Olympics in advance and register them as recognition models 32. Details of the general-purpose language model 32B and the adapted language model 32C will be described later.

Topic data 31 may be information generated in advance at least prior to the recognition process by recognition unit 20B, and may be generated either before or after the user speaks.

When topic data 31 is information generated in advance before a user speaks, topic data 31 may be generated in advance using, for example, information from a program progress schedule sheet or information from lecture notes. A program progress schedule sheet is information used when producing a broadcast program that describes the planned progress of the program being produced. Lecture notes are notes that describe the speech content to be used during a lecture.

When topic data 31 is information generated after a user speaks, topic data 31 may be, for example, information on lecture records compiled after a lecture or information on meeting minutes.

Topic data 31 may be generated in advance by the control unit 20, or may be generated in advance by external information processing.

Let's return to Figure 1 and continue the explanation.

The recognition model 32 is a pre-trained learning model used during speech recognition. The recognition model 32 includes an acoustic model 32A, a general-purpose language model 32B, and an adapted language model 32C.

Acoustic model 32A is a machine learning model that takes speech data as input and outputs the phoneme occurrence probability for each phoneme included in the speech data for each elapsed time. Acoustic model 32A may be a model that has been trained in advance using a known method.

The generic language model 32B is a machine learning model that takes as input the phonemes and phoneme occurrence probabilities for each elapsed time contained in the speech data and outputs a generic candidate word string for each elapsed time. The generic language model 32B may be a model that has been trained in advance using a known method.

The adapted language model 32C is a machine learning model for outputting speech recognition results from speech data. The adapted language model 32C is linked to each of multiple topics and pre-stored in the storage unit 30. In other words, the recognition model 32 includes multiple adapted language models 32C, and each of the multiple adapted language models 32C is linked to a different topic.

In detail, the adapted language model 32C is a machine learning model that takes a phoneme string represented by a generic candidate word string as input and outputs a candidate word string. The adapted language model 32C may be a model that has been trained in advance using a known method.

As described above, the recognition unit 20B uses the topic data 31, the recognition model 32, and the elapsed time to generate a speech recognition result from the speech data acquired by the acquisition unit 20A.

In detail, for each elapsed time of the speech data acquired by the acquisition unit 20A, the recognition unit 20B generates a speech recognition result using an adapted language model 32C linked to the topic with the highest probability of topic appearance corresponding to the elapsed time.

The recognition unit 20B inputs the speech data acquired by the acquisition unit 20A into the acoustic model 32A. Through this processing, the recognition unit 20B derives the phoneme occurrence probability for each phoneme included in the speech data for each elapsed time as output from the acoustic model 32A.

The recognition unit 20B then arranges the derived phoneme and its phoneme occurrence probability pairs in chronological order along with the speech data, and inputs them into the general-purpose language model 32B. Through these processes, the recognition unit 20B derives a general-purpose candidate word string for each elapsed time as output from the general-purpose language model 32B.

Furthermore, the recognition unit 20B derives multiple candidate word sequences by inputting the phoneme sequence represented by the generic candidate word sequence corresponding to the elapsed time into each of the adapted language models 32C corresponding to each of the multiple topics.

Then, for each of the multiple candidate word strings derived for each elapsed time and topic, the recognition unit 20B calculates a candidate word string score by adding the product of the language score obtained by the adapted language model 32C corresponding to each topic and the corresponding topic occurrence probability, the acoustic score obtained by the acoustic model 32A, and the general-purpose language score obtained by the general-purpose language model 32B.

That is, the recognition unit 20B calculates the candidate word string score for each of multiple candidate word strings for each elapsed time and topic using the following formulas (1) and (2).

In formula (1), n represents the topic ID (identification information). t represents the elapsed time. _{Score n,t} represents the candidate word string score of the candidate word string for the elapsed time t and topic n. Acoustic _t represents the acoustic score for the elapsed time t. Language _t represents the general-purpose language score for the elapsed time t. λ represents a constant. _{Topic n,t} represents the topic occurrence probability for the elapsed time t and topic n. _{Adapt n,t} represents the adaptive language score for the elapsed time t and topic n.

The recognition unit 20B calculates a score Score _n,t for each topic in the topic data 31 using the above formula (1). Alternatively, to reduce the amount of calculation, the recognition unit 20B may be configured to calculate scores only for multiple topics whose topic appearance probability corresponding to the elapsed time t in the topic data 31 is equal to or greater than a threshold. For example, this threshold is 0.5, but is not limited to this value.

Specifically, assume a situation in which "sports" and "Olympics" are identified as topics with topic appearance probabilities equal to or greater than a threshold corresponding to a certain elapsed time t in the topic data 31. Also assume a situation in which the topic appearance probability of the topic "sports" is the highest, and the topic appearance probability of the topic "Olympics" is the next highest. In this case, the recognition unit 20B determines the topic appearance probabilities of the topic "sports" and the topic appearance probabilities of the topic "Olympics" as the topic appearance probabilities TOPIC _n,t of the topics corresponding to the elapsed time t.

Of the multiple candidate word strings derived for each elapsed time and topic, the recognition unit 20B adopts the candidate word string with the highest candidate word string score for each elapsed time as the speech recognition result corresponding to each elapsed time.

Through these processes, the recognition unit 20B generates, for each elapsed time in the speech data, a speech recognition result using the adapted language model 32C associated with the topic with the highest probability of occurrence corresponding to that elapsed time, as the speech recognition result corresponding to that elapsed time in the speech data.

Next, we will explain the correction unit 20C.

The correction unit 20C uses a correction model 33 for correcting the speech recognition result to generate a corrected speech recognition result by correcting the speech recognition result. The corrected speech recognition result is one in which at least some of the words that make up the speech recognition result have been corrected to a correct recognition result.

The correction model 33 is a machine learning model used to correct speech recognition results. The correction model 33 includes a correction base model 33A and a correction converter 33B.

The correction base model 33A is a machine learning model for acquiring the first feature of the speech recognition result from the speech recognition result. The correction base model 33A is trained in advance using pairs of speech recognition results and correct text. The correct text represents the correct speech recognition result. The first feature is a tensor output from the intermediate layer of the correction base model 33A. The first feature is sometimes referred to as an intermediate representation or a latent representation.

The correction converter 33B is a machine learning model for outputting corrected speech recognition results from the first feature.

Figure 3A is an explanatory diagram of an example of correction processing by the correction unit 20C.

The correction unit 20C acquires a first feature of the speech recognition result using the speech recognition result received from the recognition unit 20B and the correction base model 33A. The correction unit 20C inputs the speech recognition result to the correction base model 33A, thereby acquiring the first feature as an output from the intermediate layer of the correction base model 33A.

The correction unit 20C generates a corrected speech recognition result by inputting the acquired first feature amount to the correction converter 33B.

Figure 3B is a schematic diagram showing an example of the configuration of the correction converter 33B. For example, the correction converter 33B is composed of nodes representing multiple words and arcs, which are paths representing the connections between the multiple words. In Figure 3B, the words that are nodes are represented by circles, and the paths that are arcs are represented by arrows.

The correction unit 20C searches for a path representing the connection relationship between the multiple words defined by the correction converter 33B, using the similarity between the input first feature and the second feature, which is a latent expression of each of the multiple words defined by the correction converter 33B. Cosine similarity may be used as this similarity. During this search, the correction unit 20C excludes words with a similarity below a threshold from the path search candidates. The correction unit 20C then searches for the shortest path and outputs a word string consisting of the words that make up the shortest path as the corrected speech recognition result.

Returning to Figure 1, we will continue the explanation. It is preferable that the correction unit 20C use the elapsed time to calculate the similarity score, which is the similarity used in the search.

In other words, the correction unit 20C generates a corrected speech recognition result in which the similarity to the first feature value increases the closer the second feature value of a word matches a corrected string included in a corrected speech recognition result linked to a correction time in the correction history data 34 that is closer to the elapsed time of the word for which similarity is to be calculated. A corrected string refers to a string of characters that makes up the corrected speech recognition result and that differs from the corresponding speech recognition result. In other words, a corrected string is a string that is different between the corrected speech recognition result and the speech recognition result. Note that the string may consist of one character or multiple characters.

Correction history data 34 is data for managing the history of user corrections made to speech recognition results. The correction history data 34 is updated by the storage control unit 20E, which will be described later.

The revision history data 34 is data that links together the speech recognition result, the revised speech recognition result, the topic used in the speech recognition of the speech recognition result, and the revision time, which is the elapsed time.

The corrected speech recognition result represents a speech recognition result corrected by the user with respect to the speech recognition result output from the recognition unit 20B or the corrected speech recognition result corrected by the correction unit 20C for the speech recognition result. In this embodiment, the corrected speech recognition result represents a speech recognition result corrected by the user with respect to the corrected speech recognition result, as an example. The correction time represents the elapsed time corresponding to the corrected speech recognition result corrected by the user.

Figure 4A is a schematic diagram showing an example of the data structure of correction history data 34A. Correction history data 34A is an example of correction history data 34 when the speech data is Japanese speech data. As shown in Figure 4, correction history data 34A registers the correction time, topic, speech recognition result, and corrected speech recognition result in association with each other.

Figure 4B is a schematic diagram showing an example of the data structure of correction history data 34B. Correction history data 34B is an example of correction history data 34 when the audio data is English audio data. As shown in Figure 4B, correction history data 34B registers the time of correction, the topic, the speech recognition result, and the corrected speech recognition result in association with each other.

Returning to Figure 1, we will continue the explanation. When using elapsed time to calculate the similarity score, the correction unit 20C calculates the similarity score using the following equation (3).

In formula (3), n represents the ID (identification information) of the topic. t represents the elapsed time. FixScore _n,t represents the similarity score between the first feature of the speech recognition result of the elapsed time t and topic n and the second feature, which is a latent expression of each of the multiple words defined in the correction converter 33B. W _fix represents the corrected character string included in the corrected speech recognition result. W _asr represents the speech recognition result or the corrected speech recognition result. In this embodiment, a case where W _asr represents the corrected speech recognition result will be described as an example.

logP(W _fix |W _asr ) represents the similarity score before correction. _{f n} (t, W _fix ) represents a weighting value using the correction time. _{f n} (t, W _fix ) represents a word that constitutes the corrected character string included in the corrected speech recognition result, among the multiple words defined in the correction converter 33B, and the closer the correction time (elapsed time) of the word is to the elapsed time t of the comparison word included in the speech recognition result, the higher the value represents.

In this way, the correction unit 20C may use elapsed time to calculate the similarity score, which is the similarity used in the search, and calculate a higher similarity score the closer it is to the time of the last modification.

By calculating a higher similarity score the closer the correction unit 20C is to the time of the previous correction, the correction unit 20C can generate a corrected speech recognition result that corrects misrecognitions caused by speech recognition of currently spoken speech data in more real time and with higher accuracy.

The correction unit 20C may also search for a path for the correction converter 33B using a modified similarity obtained by adding a weighting value corresponding to the topic used to derive the speech recognition result to the similarity score, which is the similarity used for the search.

For example, the revision history data 34 may be configured to further be linked with a weight value corresponding to each topic. The weight value corresponding to a topic may be registered, for example, in an operation instruction by a user. The correction unit 20C may then use, as fn(t, W _fix ) in the above formula (3), the weight value linked with the topic corresponding to the revision time, which is the elapsed time t. The correction unit 20C may also use, as fn(t, W _fix ) in the above formula (3), the multiplication or addition value of the weight value using the revision time and the weight value linked with the topic.

In this way, by using a correction similarity to which a weight value corresponding to the topic used by the correction unit 20C to derive the speech recognition result is added, it is possible to generate corrected speech recognition results that are corrected in real time and with high accuracy, for speech recognition results that include corrected words or speech recognition results that include words that were corrected more recently.

For example, news that differs from the script may be urgently inserted and spoken. In such cases, specific words contained in the urgently inserted news content are often spoken multiple times. For example, if the urgently inserted news is an earthquake alert, it is expected that the location of the earthquake will be spoken repeatedly. Therefore, by having the correction unit 20C use a modified similarity to which a weight value corresponding to the topic used by the correction unit 20C to derive the speech recognition result is added, it is possible to generate a corrected speech recognition result that is corrected in more real time and with higher accuracy.

The output control unit 20D outputs the speech recognition result obtained by the recognition unit 20B. If the control unit 20 is configured to include a correction unit 20C, the output control unit 20D outputs the corrected speech recognition result obtained by the correction unit 20C as the speech recognition result.

The output control unit 20D outputs the voice recognition results to the display unit 40A. The output control unit 20D may also output the voice recognition results to an external information processing device via the communication unit 50. The output control unit 20D may also output the voice recognition results to various output devices, such as speakers and projection devices, that are communicatively connected to the control unit 20.

In this embodiment, the output control unit 20D outputs the speech recognition result to the display unit 40A. The output control unit 20D outputs the speech recognition result and the topic used for the speech recognition of the speech recognition result to the display unit 40A. The output control unit 20D may output topics with a topic appearance probability equal to or greater than a threshold to the display unit 40A in a display format different from that of topics with a topic appearance probability below the threshold. The threshold may be set in advance. Furthermore, the threshold may be changeable as appropriate in response to user instructions on the input unit 40B. A different display format means different character color, character size, character format, etc.

Figure 5 is a schematic diagram of an example of a display screen 60A. Display screen 60A is an example of a display screen 60 when the voice data is in Japanese. Display screen 60 is an example of a display screen for the voice recognition results by output control unit 20D.

For example, display screen 60A includes a recognition result display field 62A and a topic display field 62B. Recognition result display field 62A is a display field that shows, for each elapsed time, the topic with the highest topic appearance probability corresponding to that elapsed time in topic data 31, in association with the speech recognition result. Topic display field 62B is a display field that shows, for each elapsed time, the topic with the highest topic appearance probability corresponding to that elapsed time.

Let's return to Figure 1 and continue the explanation.

The storage control unit 20E acquires the user's corrections to the speech recognition results and stores them in the correction history data 34.

The user operates the input unit 40B while viewing the display screen 60 displayed on the display unit 40A, thereby inputting a correction string for the displayed speech recognition result, thereby inputting a corrected speech recognition result. The storage control unit 20E associates the corrected speech recognition result input by the user's operation instruction of the input unit 40B with the speech recognition result from which the corrected speech recognition result was made, the topic used in the speech recognition of the speech recognition result, and the correction time, which is the elapsed time, and stores them in the correction history data 34.

For this reason, the storage unit 30 stores, for example, correction history data 34A shown in FIG. 4A. In the case of correction history data 34A expressed in Japanese shown in FIG. 4A, a corrected speech recognition result in which the character string "eh" included in the speech recognition result has been corrected to the corrected character string "A" is registered. Also, in the case of correction history data 34A, a corrected speech recognition result in which the character string "eh-oki" included in the speech recognition result has been corrected to the corrected character "oki" is registered.

In addition, the storage unit 30 stores, for example, correction history data 34B shown in Figure 4B. In the case of correction history data 34B expressed in English shown in Figure 4B, a corrected speech recognition result in which the character string "blockade" included in the speech recognition result has been corrected to the corrected character string "Block A" is registered. In the case of correction history data 34B, a corrected speech recognition result in which the character string "loke" included in the speech recognition result has been corrected to the corrected character string "Oki" is registered.

Let's return to Figure 1 and continue the explanation.

The update unit 20F updates the correction model 33 using the correction history data 34.

Figure 6 is an explanatory diagram of an example of update processing by the update unit 20F.

The update unit 20F obtains a first feature by inputting the speech recognition result output by the output control unit 20D into the correction base model 33A. The update unit 20F then updates the correction converter 33B of the correction model 33 by training the correction converter 33B of the correction model 33 using pairs of the obtained first feature and the corrected speech recognition result corresponding to the obtained speech recognition result included in the correction history data 34 as training data.

As described above, the correction base model 33A is a machine learning model that has been trained in advance using pairs of speech recognition results and correct text. For example, the correction base model 33A can be created by using the well-known technology BART (Bidirectional Auto-Regressive Transformer) to fine-tune a model that has been pre-trained using a large corpus using pairs of speech recognition results and correct text.

The update unit 20F then obtains the first feature, which is a latent expression, by inputting the speech recognition result output by the output control unit 20D into the corrected base model 33A created in advance.

The update unit 20F then extracts the obtained first feature amount and a corrected character string, which is the difference between the corrected speech recognition result and the speech recognition result included in the correction history data 34. The update unit 20F then updates the correction converter 33B using the first feature amount, the corrected character string, and the correction location, which is the position of the corrected character string in the corrected speech recognition result.

First, the update unit 20F initializes the correction converter 33B. The initialized correction converter 33B exhibits a loop structure as shown in Figure 3B. In addition, an arc that matches any input word and outputs the input word as is is added to the top node of the initialized correction converter 33B.

The update unit 20F obtains a first feature, which is a latent expression, by inputting the speech recognition result output by the output control unit 20D into a previously created correction base model 33A. The update unit 20F also obtains a pair of the speech recognition result and the corrected speech recognition result corresponding to the speech recognition result. The update unit 20F then extracts a corrected character string, which is the difference between the speech recognition result and the corrected speech recognition result.

Then, the update unit 20F adds arcs and nodes to the correction converter 33B so that the first feature, which is a latent representation of the speech recognition result, is input and the corrected speech recognition result is output. In detail, the update unit 20F adds nodes and arcs between the start node and end node for each word, which is a plurality of nodes included in the correction converter 33B.

Through these processes, the update unit 20F updates the correction converter 33B from the first feature so that the latest corrected speech recognition result for the speech recognition result of the first feature is output as the corrected speech recognition result. Through these update processes, the update unit 20F updates the correction model 33.

For this reason, the correction model 33 is updated to output corrected speech recognition results that are more accurate and correspond to real-time speech.

Let's return to Figure 1 and continue the explanation.

The topic data update unit 20G updates the topic data 31 using the revision history data 34.

For example, the topic data update unit 20G updates the topic data 31 based on the revision history data 34 so as to reduce the topic appearance probability of topics whose number of revisions per unit time is equal to or exceeds a threshold. The threshold number of revisions and the unit time may be determined in advance. The unit time is, for example, 10 minutes, but is not limited to this time. The threshold number of revisions and the unit time may be changeable as appropriate by the user's operation instructions on the input unit 40B, etc.

In detail, the topic data update unit 20G counts the number of revisions per unit time for each topic based on the revision history data 34. Then, the topic data update unit 20G identifies from the revision history data 34 topics whose number of revisions per unit time is equal to or greater than a threshold value, or whose revision rate per unit time is equal to or greater than a threshold value.

The topic data update unit 20G then changes the topic appearance probability in the topic data 31 linked to the topic identified based on the revision history data 34 so that the topic appearance probability is lowered. At this time, the topic data update unit 20G changes the topic appearance probability in the topic data 31 linked to the topic identified from the revision history data 34 and the elapsed time, which is the revision time corresponding to that topic. The topic data update unit 20G then allocates the probabilities evenly so that the topic appearance probabilities of other topics are increased by the amount of the reduced probability.

Through these processes, the topic data update unit 20G updates the topic data 31.

The topic data update unit 20G may also update the topic data 31 by the following process.

For example, the topic data update unit 20G counts the number of revisions per unit time for each topic based on the revision history data 34. Then, the topic data update unit 20G identifies topics from the revision history data 34 for which the number of revisions per unit time is equal to or greater than a threshold, or the rate of revisions per unit time is equal to or greater than a threshold. Furthermore, the topic data update unit 20G identifies other topics associated with speech recognition results in the revision history data 34 that match or are similar to the revised speech recognition results corresponding to the identified topic as similar topics.

If a similar topic can be identified from the revision history data 34, the topic data update unit 20G identifies the elapsed time, which is the revision time corresponding to the similar topic in the revision history data 34. The topic data update unit 20G then reduces the topic appearance probability associated with the topic corresponding to the identified elapsed time among the topics associated with elapsed times within a period including the identified elapsed time in the topic data 31, and increases the topic appearance probability associated with topics that match the similar topic.

Through these processes, the topic data update unit 20G updates the topic data 31 so as to reduce the topic appearance probability of topics whose number of revisions per unit time is equal to or exceeds a threshold value.

For this reason, the topic data update unit 20G can update the topic data 31 in a voice recognition device 10 that outputs voice recognition results for voices spoken in real time, so that voice recognition performance is improved even when the topic changes sequentially.

Next, we will explain an example of the flow of information processing performed by the speech recognition device 10 of this embodiment.

Figure 7 is a flowchart showing an example of the flow of information processing performed by the speech recognition device 10 of this embodiment.

The acquisition unit 20A acquires audio data (step S100).

For each elapsed time of the speech data acquired in step S100, the recognition unit 20B identifies the topic appearance probability corresponding to the elapsed time from the topic data 31 (step S102). Then, for each elapsed time, the recognition unit 20B identifies an adapted language model 32C associated with the topic with the highest topic appearance probability identified in step S102 (step S104). Then, for each elapsed time, the recognition unit 20B uses the identified adapted language model 32C to generate a speech recognition result for the speech data acquired in step S100 (step S106).

The correction unit 20C uses the correction model 33 to generate a corrected speech recognition result by correcting the speech recognition result generated in step S106 (step S108).

The output control unit 20D displays the corrected speech recognition result generated in step S108 as the speech recognition result on the display unit 40A (step S110). For example, the output control unit 20D displays the display screen 60A shown in FIG. 5 on the display unit 40A.

The storage control unit 20E acquires the user's revised speech recognition result for the speech recognition result displayed in step S110 (step S112). The storage control unit 20E then stores the revised speech recognition result acquired in step S112 in the correction history data 34 (step S114).

The update unit 20F updates the correction model 33 using the correction history data 34 stored in step S114 (step S116).

The topic data update unit 20G updates the topic data 31 using the revision history data 34 stored in step S114 (step S118). This routine then ends.

As described above, the speech recognition device 10 of this embodiment includes a recognition unit 20B and an output control unit 20D. The recognition unit 20B generates speech recognition results from speech data using topic data 31 in which the topic appearance probability for each elapsed time for each of a plurality of topics that changes over time is registered in advance, a recognition model 32 linked to each of the plurality of topics and including an adapted language model 32C for outputting speech recognition results from speech data, and the elapsed time. The output control unit 20D outputs the speech recognition results.

Here, conventional technology discloses a technique for extracting text corresponding to speech recognition results from topic information such as television program scripts and progress schedules, and then obtaining text related to the extracted text from external data on a website. Furthermore, conventional technology discloses a technique for adjusting the language score used for speech recognition using text obtained from external data. However, because conventional technology adjusts the language score by obtaining text related to speech recognition results from external data on a website, it is difficult to apply this to real-time subtitling of programs such as news programs, and there are cases where it is not possible to improve speech recognition performance.

On the other hand, in the speech recognition device 10 of this embodiment, the recognition unit 20B generates speech recognition results from speech data using the topic appearance probability for each elapsed time for each of multiple topics that changes over time, a recognition model 32 including an adapted language model 32C linked to each of the multiple topics, and the elapsed time.

In this way, the speech recognition device 10 of this embodiment generates speech recognition results from speech data using the topic appearance probability of the topic according to the elapsed time of the speech data and the adapted language model 32C of the topic according to the elapsed time.

For this reason, the speech recognition device 10 of this embodiment can generate speech recognition results that follow topic changes, even when the topic changes depending on the elapsed time of the speech data.

Therefore, the voice recognition device 10 of this embodiment can improve voice recognition performance.

Furthermore, the speech recognition device 10 of this embodiment can generate speech recognition results that follow topic changes, even when the topic changes depending on the elapsed time of the speech data, making it suitable for use in systems that generate subtitles in real time.

For example, the speech recognition device 10 of this embodiment can be suitably applied to a system that provides speech recognition results such as subtitles in real time when live broadcasts such as television news programs are aired in real time.

In addition, in the speech recognition device 10 of this embodiment, the correction unit 20C generates a corrected speech recognition result by correcting the speech recognition result using a correction model 33 for correcting the speech recognition result.

Therefore, in addition to the above-mentioned effects, the speech recognition device 10 of this embodiment can immediately output corrected speech recognition results if the speech recognition results contain errors.

In addition, in the speech recognition device 10 of this embodiment, the update unit 20F updates the correction model 33 using correction history data 34 that links the speech recognition result, the corrected speech recognition result corrected by the user, and the correction time.

As a result, the speech recognition device 10 of this embodiment can output speech recognition results that instantly reflect user corrections to the speech recognition results. Therefore, in addition to the above-mentioned effects, the speech recognition device 10 of this embodiment can also reduce the development costs, correction costs, and user correction burden required for speech recognition such as subtitling.

(Hardware configuration)
Next, the hardware configuration of the speech recognition device 10 of the above embodiment will be described.

Figure 8 is a diagram showing an example of the hardware configuration of the speech recognition device 10 of the above embodiment.

The speech recognition device 10 of the above embodiment has a hardware configuration that utilizes a conventional computer, with a CPU 80, ROM (Read Only Memory) 82, RAM (Random Access Memory) 84, HDD 86, and I/F unit 88, etc., interconnected via a bus 90.

The CPU 80 is a computing device that controls the information processing executed by the speech recognition device 10 of the above embodiment. The RAM 84 stores data necessary for various processes performed by the CPU 80. The ROM 82 stores programs and the like that realize various processes performed by the CPU 80. The HDD 86 stores data. The I/F unit 88 is an interface for sending and receiving data to and from other devices.

The programs for executing the various processes described above executed by the speech recognition device 10 of the above embodiment are provided pre-installed in the ROM 82, etc.

The programs executed by the speech recognition device 10 of the above embodiment may be provided by being recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk) in a format that can be installed or executed by these devices.

The program executed by the speech recognition device 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program for executing each of the above processes in the speech recognition device 10 of the above embodiment may also be provided or distributed via a network such as the Internet.

The programs for executing the various processes executed by the speech recognition device 10 of the above embodiment are generated on the main memory device by the above-mentioned components.

The various information stored in the HDD 86 may also be stored in an external device. In this case, the external device and the CPU 80 may be connected via a network or the like.

Although the above describes embodiments of the present disclosure, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments and their variations are within the scope and spirit of the invention, and are also included in the inventions and their equivalents set forth in the claims.

10 Speech recognition device 20B Recognition unit 20C Correction unit 20D Output control unit 20E Storage control unit 20F Update unit 20G Topic data update unit 31 Topic data 32 Recognition model 32C Adapted language model 33 Correction model 33A Correction base model 33B Correction converter 34 Correction history data 40A Display unit

Claims (13)

a recognition unit that generates a speech recognition result from the speech data using topic data in which a topic appearance probability for each elapsed time of a plurality of topics that changes with the elapsed time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adaptive language model for outputting a speech recognition result from the speech data, and the elapsed time;
an output control unit that outputs the speech recognition result;
Equipped with
The output control unit
outputting the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit;
outputting the topic having the topic appearance probability equal to or greater than a threshold value to the display unit in a display format different from that of the topic having the topic appearance probability less than the threshold value;
Voice recognition device. The recognition unit
generating the speech recognition result using the adapted language model associated with the topic having the highest topic appearance probability corresponding to the elapsed time for each elapsed time of the speech data;
2. The speech recognition device according to claim 1. a correction unit that generates a corrected speech recognition result by correcting the speech recognition result using a correction model for correcting the speech recognition result;
Equipped with
The output control unit
outputting the corrected speech recognition result as the speech recognition result;
2. The speech recognition device according to claim 1. The correction model is
a correction base model for acquiring a first feature of the speech recognition result from the speech recognition result;
a correction converter that outputs the corrected speech recognition result from the first feature;
Including,
the first feature is a latent representation output from an intermediate layer of the correction base model;
The correction unit
using the speech recognition result and the correction base model to acquire the first feature of the speech recognition result, and inputting the acquired first feature to the correction converter to generate the corrected speech recognition result;
4. The speech recognition device according to claim 3 . The correction unit
a path representing a concatenation relationship between the plurality of words defined by the correction converter is searched for using a similarity between the input first feature and a second feature which is a latent expression of each of the plurality of words defined by the correction converter, and an output for the path with the greatest similarity is output as the corrected speech recognition result;
5. The speech recognition device according to claim 4 . The correction unit
generating the corrected speech recognition result such that the similarity between the second feature of the word and the first feature is higher as the second feature of the word matches the corrected character string included in the corrected speech recognition result associated with the closer modification time;
6. The speech recognition device according to claim 5 . The correction unit
searching for the path using a modified similarity obtained by adding a weight value corresponding to the topic used in deriving the speech recognition result to the similarity;
6. The speech recognition device according to claim 5 . a storage control unit that acquires a speech recognition result corrected by a user for the speech recognition result, and stores the speech recognition result, the corrected speech recognition result, and the correction time, which is the elapsed time, in correction history data that are linked together;
an update unit that updates the correction model using the correction history data;
The speech recognition device according to claim 3 . The update unit
a first feature obtained by inputting the speech recognition result into a correction base model that has been trained in advance using a pair of the speech recognition result and a correct text;
the corrected speech recognition result corresponding to the speech recognition result included in the correction history data; and
The speech recognition device according to claim 8 , wherein the correction model is updated by training the correction model using pairs of the following as training data: a topic data updating unit that updates the topic data using the revision history data;
The speech recognition device according to claim 8, comprising: The topic data update unit
updating the topic data based on the revision history data so as to reduce the topic appearance probability of the topic whose number of revisions per unit time is equal to or exceeds a threshold value;
The speech recognition device according to claim 10. a recognition step of generating a speech recognition result from the speech data using topic data in which a topic appearance probability for each elapsed time of a plurality of topics that changes with the elapsed time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adaptive language model for outputting a speech recognition result from the speech data, and the elapsed time;
an output control step of outputting the speech recognition result;
Including,
The output control step includes:
outputting the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit;
outputting the topic having the topic appearance probability equal to or greater than a threshold value to the display unit in a display format different from that of the topic having the topic appearance probability less than the threshold value;
Speech recognition methods. A speech recognition program to be executed by a computer,
a recognition step of generating a speech recognition result from the speech data using topic data in which a topic appearance probability for each elapsed time of a plurality of topics that changes with the elapsed time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adaptive language model for outputting a speech recognition result from the speech data, and the elapsed time;
an output control step of outputting the speech recognition result;
Including,
The output control step includes:
outputting the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit;
outputting the topic having the topic appearance probability equal to or greater than a threshold value to the display unit in a display format different from that of the topic having the topic appearance probability less than the threshold value;
Speech recognition program.