JP7627753B2 - Voice Recognition Device - Google Patents

Description

The present invention relates to a voice recognition device that performs voice recognition.

Patent document 1 describes a conference support system that distinguishes between speech segments and non-speech segments contained in audio data during a conference, etc., and recognizes text data indicating speech from the speech segments.

JP 2018-45208 A

However, in a meeting, etc., it is not possible to recognize laughter, etc., and it is not possible to provide that atmosphere. Therefore, it is difficult to analyze the quality of the meeting atmosphere, etc.

The present invention aims to provide a voice recognition device that can grasp the atmosphere of a conversation in a meeting, etc.

The voice recognition device of the present invention comprises a sound information acquisition unit that acquires sound information, and a sound information processing unit that performs voice recognition processing on first sound information among the sound information to obtain a recognition result, determines that second sound information related to the first sound information is the occurrence of a non-verbal event, and outputs information indicating the event in association with the recognition result.

According to the present invention, it is possible to obtain voice recognition results that make it easy to grasp the atmosphere of a conversation in a meeting, etc.

1 is a block diagram showing a functional configuration of a speech recognition device 100 according to the present disclosure. 10 is an explanatory diagram showing the reliability of a speech recognition result and the reliability of each recognition event; FIG. FIG. 13 is a diagram showing an example of a processing result for an utterance. FIG. 13 is a diagram showing an outline of a determination process when a determination section is changed. 4 is a flowchart showing an operation of the voice recognition device 100. FIG. 13 is a diagram illustrating a process when an utterance is in English. FIG. 13 is a diagram showing an example of a result output in a process of adding event tag information. 13 is a diagram showing an example of processing determination for user A and user B. FIG. 13 is a diagram showing an example of a result output in another example. FIG. 13 is a diagram showing an example of a result output in another example. An example in which event tag information is associated with recognition result text will be described. FIG. 13 is a block diagram showing a functional configuration of a speech recognition device 100a according to a modification of the present disclosure. FIG. 13 is a diagram showing a specific example of correction. FIG. 11 is a block diagram showing a functional configuration of a voice recognition device 100b according to another modified example. 13 is a graph showing the frequency of event types by topic. FIG. 1 is a diagram illustrating an example of a hardware configuration of a voice recognition device 100 according to an embodiment of the present disclosure.

An embodiment of the present disclosure will be described with reference to the accompanying drawings. Where possible, identical parts are designated by the same reference numerals and duplicated explanations will be omitted.

Figure 1 is a block diagram showing the functional configuration of a speech recognition device 100 in the present disclosure. As shown in the figure, the speech recognition device 100 is configured to include a speech acquisition unit 101, a speech recognition unit 102, a non-language speech recognition unit 103 (non-language sound recognition unit), a score determination unit 104, and a result output unit 105. Each component will be described below.

The voice acquisition unit 101 is a part that acquires voice during a meeting, lecture, etc. For example, the voice acquisition unit 101 is a microphone. However, this is not limited to this, and the voice acquisition unit 101 may be a part that acquires a voice signal transmitted by wire or wirelessly. This voice acquisition unit 101 detects a voice section from the voice waveform signal, and outputs each section to the voice recognition unit 102 and the non-language voice recognition unit 103.

The speech recognition unit 102 performs speech recognition processing on speech sounds or non-speech sounds in the speech section output from the speech acquisition unit 101 using a publicly known language model and acoustic model, acquires recognition result text for each recognition unit, and derives the reading and reliability of the recognition result text. The speech recognition unit 102 outputs the recognition result text, its reading, and its reliability for each recognition unit, such as an utterance unit, a sentence unit, a phrase unit, a word unit, a kana or phoneme unit, or a time unit.

This reliability is information indicating how much one can trust the recognition results when speech recognition processing is performed, and is generally indicated between 0 and 1, but is not limited to this and may be expressed as an integer or between 0 and 100. It may also be a normalized numerical value. In this disclosure, the reliability of the speech recognition result is calculated based on the reliability stored in the language model and the acoustic model, but is not limited to this and other known reliability derivation methods, such as end-to-end speech recognition, may be used.

The non-language speech recognition unit 103 is a part that performs recognition processing on non-language sounds using a publicly known recognition model for non-language sounds for speech sounds or non-language sounds in the speech section output from the speech acquisition unit 101. This non-language speech recognition unit 103 generates a reliability for each event according to the type of non-language sound for each recognition time according to the recognition unit recognized by the speech recognition unit 102. For example, when the speech recognition unit 102 recognizes on a word-by-word basis, the speaking time for the recognized word becomes the recognition unit of the non-language speech recognition unit 103.

An event indicates a type of non-verbal sound, and can be a positive event or a negative event. For example, in this disclosure, non-verbal sounds refer to non-verbal sounds based on speech, such as laughter, interjections, nodding sounds (which can be distinguished as positive or negative), sighs, sneezing, and coughing, as well as keyboard sounds and musical sounds such as background music. For example, a positive event indicates that it is laughter, and a negative event indicates that it is not laughter. The non-verbal speech recognition unit 103 recognizes these non-verbal sounds and generates a confidence level for the positive event and the negative event.

The reliability of an event is information indicating how much one can trust the recognition results when non-speech speech recognition processing is performed. This reliability is generally indicated between 0 and 1, but is not limited to this and may be expressed as an integer or between 0 and 100. It may also be a normalized numerical value. This reliability is calculated based on the reliability stored in the recognition model for recognizing non-speech sounds, but is not limited to this and other known reliability derivation methods may be used.

Figure 2 is an explanatory diagram showing the reliability of the speech recognition result text and the reliability for each event. As shown in the figure, assume that there is an utterance "Ahahaha" (spoken as "ahahaha"). This indicates laughter.

The speech recognition unit 102 attempts to recognize speech as speech sounds, and therefore recognizes the text data as "あぼ派" (converted into Japanese kanji characters) and the reading "あははハハ" (ahahaha). The speech recognition unit 102 derives the reliability of the recognized recognition result text using the acoustic model and the language model.

On the other hand, the non-language speech recognition unit 103 derives the confidence for each event. In FIG. 2, the confidence is derived according to the presence or absence of laughter. That is, the confidence that "ahahaha" (spoken as "ahahaha") is not laughter and the confidence that it is laughter are calculated. In FIG. 2, the confidence that there is no laughter is calculated as 0.3, and the confidence that there is laughter is calculated as 0.7.

In addition, the reliability of the text data as laughter may be calculated. Also, the reliability of other types of non-verbal sounds, such as the presence or absence of coughing or the presence or absence of keyboard sounds, may be calculated.

The non-verbal speech recognition unit 103 has a recognition model for recognizing laughter, coughing, or keyboard sounds, and can calculate the reliability of each based on this recognition model. Of course, it may also have recognition models for recognizing other sounds such as backchannel sounds, nodding sounds, and sneezing. It may also have a recognition model that outputs the reliability of each of multiple events.

The score determination unit 104 is a part that determines whether the recognition result text or the event is valid based on the reliability of the recognition result text for the speech sounds recognized by the speech recognition unit 102 and the reliability of the event recognized by the non-speech speech recognition unit 103.

The detailed processing will be explained with reference to FIG. 2. In FIG. 2, it is assumed that the voice "ahahaha" is input and processed by the voice recognition unit 102 and the non-language voice recognition unit 103. The score determination unit 104 compares the reliability of the recognition result text "あぼ派" (Japanese in which "ahahaha" is recognized (here, an example of a misconversion)) of the voice recognition unit 102 with the reliability of the positive event and negative event (with/without laughter) of the non-language voice recognition unit 103. In FIG. 2, the voice recognition unit 102 outputs the reliability of the recognition result text: 0.3. Furthermore, the non-language voice recognition unit 103 outputs the reliability of the negative event (without laughter): 0.3 and the reliability of the positive event (with laughter): 0.7.

The score determination unit 104 determines the recognition result text or event with the highest reliability. In FIG. 2, the reliability of the positive event (laughing) is 0.7, which is the highest reliability, so "ahahaha" is determined as an event indicating laughter.

As described above, the score determination unit 104 may determine the recognition result text or event based on the highest reliability, or may use values adjusted by weighting for each reliability. For example, the score determination unit 104 may determine which event the utterance content is based on a value obtained by multiplying the reliability of a positive event or a negative event by a predetermined coefficient. More specifically, the score determination unit 104 may determine a non-speech sound by multiplying the reliability of a positive event by 2 and comparing the result with the reliability of a negative event.

The score determination unit 104 may also determine whether a positive event in a non-speech sound is appropriate by multiplying the reliability of the positive event by 0.7, subtracting 0.1 from it, and comparing it with a threshold value. These coefficients and threshold values may be fixed values stored in advance in a memory or the like, or an input unit may be provided so that they can be input from the outside. Furthermore, the given coefficients and threshold values may be varied using a predetermined formula. The score determination unit 104 is not limited to comparing the reliability of the positive event and the negative event with two values, but may include other values, for example, it may compare with three or more values. For example, laughter and coughing may be considered as positive events, no laughter and no coughing as negative events, and keyboard sounds may be considered as noise events, and the three reliability values of the three events may be compared.

Such weighting adjustments are determined, for example, according to the attributes or type of user using the speech recognition device 100 of the present disclosure, or the content of the conference. For example, laughter is easy to recognize in conference content that is likely to cause laughter, but there are also conference contents in which this is not the case. In such cases, laughter is difficult to recognize as laughter. For such conferences or users, adjustments can be made as described above to enable accurate recognition.

The result output unit 105 selects the recognition result text with a high reliability based on the judgment result by the score judgment unit 104, or adds event tag information corresponding to the event and outputs it. When outputting, the result output unit 105 inputs information of the event judged by the score judgment unit 104 and acquires event tag information corresponding to the event from a storage unit or the like. This storage unit or the like stores this event tag information in advance. The event tag information is, for example, information that graphically represents a non-verbal sound, and in the case of laughter, it is a mark that graphically represents laughter. The event tag information may be, for example, pre-defined text information. The output may be to an external terminal, or may be displayed on a display screen.

Figure 3 shows an example of the processing results for a certain utterance. As an example of the processing results, Figure 3 shows the recognition result text, recognition result morphemes, recognition result text confidence, event confidence (laughing), event confidence (coughing), event confidence (keyboard sound), judgment result, result output, and supplementary output. It also shows the confidence on a word-by-word basis, result output, etc.

The following speech content is acquired and the recognition result is obtained by the voice recognition unit 102.

Speech: I, um, hahaha, kind of, (cough: Gogh), (keyboard sounds: cackle cackle), it's fine.
The above utterance is in Japanese and is pronounced as watashiwa ano- hahaha nanka (gohho) kakakaka ii. In Japanese, the "wa" that comes after the subject is pronounced as wa.

Recognition result text: I am, um, mother, something, Gogh, kakakaka, good. The above is the recognition result text converted into Japanese. There is a cough and keyboard sounds mixed in here. This recognition result text, in romanized notation, shows that it has been segmented into watashi, wa, ano-, hahaha, nanka, gohho, kakakaka, and ii and converted into Japanese.
Here, for the utterance "watashiwa," the recognition results "watashi" and "wa" are obtained. The reliability of the recognition result text is derived as 0.95 and 0.91, respectively. Meanwhile, the reliability of each event for "watashi" by the non-language speech recognition unit 103 is as follows:

Confidence of laughter positive event: 0.23
Reliability of laughter negation event: 0.90
Confidence of positive cough event: 0.21
Confidence of negative cough event: 0.70
Confidence of positive keyboard event: 0.15
Confidence of negative keyboard sound event: 0.75
The reliability of each flag for "wa" by the non-language speech recognition unit 103 is as follows:

Confidence of laughter positive event: 0.35
Confidence of laughter negation event: 0.85
Confidence of positive cough event: 0.05
Confidence of negative cough event: 0.81
Confidence of positive keyboard event: 0.12
Confidence of negative keyboard sound event: 0.85
The score determination unit 104 selects the most reliable recognition result texts "watashi" and "wa" based on these reliability values, and the result output unit 105 outputs "watashiwa" (watashiwa).

On the other hand, for the utterance: "ano-" and the recognition result text: "ano-", the speech recognition unit 102 sets the speech recognition confidence level to 0.9 and recognizes it as a filler.

For the utterance: "hahaha" (hahaha) and the recognition result text: "mother's voice" (text in which "hahaha" is recognized) (confidence level: 0.1), the non-language speech recognition unit 103 calculates a confidence level of 0.7 for the positive event (laughing occurs). As the confidence level of the positive event (laughing occurs) is high compared to the confidence level of the recognition result text and other positive events (coughing, keyboard sounds), the score determination unit 104 determines that the recognition result: "mother's voice" is laughter. The result output unit 105 outputs event tag information indicating that laughter has occurred as supplementary output. Note that the result output unit 105 does not have to output the event tag information.

In the present disclosure, the score determination unit 104 determines that the recognition result text determined to be a filler based on the recognition result morpheme is a filler, even if the recognition result text has a high reliability. Then, the result output unit 105 does not output the recognition result text as a result output. Note that, if necessary, event tag information indicating that it is a filler may be output as a supplementary output.

In the example of FIG. 3, the reliability is calculated for each word, and the judgment is made based on the reliability for each word, but the present invention is not limited to this. The score judgment unit 104 may change the recognition unit of the recognition result text output by the speech recognition unit 102 and the non-language speech recognition unit 103 to a judgment unit for score judgment and make a judgment. For example, in FIG. 3, the speech recognition unit 102 and the non-language speech recognition unit 103 calculate the reliability for each word, but the score judgment unit 104 may integrate the reliability for each phrase or sentence, and make a judgment based on the reliability integrated for each phrase or sentence. By changing the score judgment range in this way, for example, the position where the event tag information indicating non-language sounds is added can be changed, making the text easier to read. For example, when the score is judged for each sentence, the event tag information is added to the end of the sentence.

Figure 4 shows an overview of the processing contents when the score determination unit is changed to sentence unit. For ease of explanation, the description of event reliability, etc. is simplified compared to Figure 3. Assume that the following speech content is input and the recognition result text is obtained.

Speech: I, um, (laughing: hahaha), something, (cough: gohho), (keyboard sound: kakakaka), good. The above utterance is in Japanese and is spoken as watashiwa ano- hahaha nanka (gohho) kakakaka ii. In Japanese, the "wa" that comes after the subject is pronounced as wa.
Recognition result text: I, um, mother, something, Gogh, kakakaka, good. The above is the recognition result text converted to Japanese. Coughing and keyboard sounds are mixed in. This recognition result text, in romanized notation, shows that it has been segmented into watashi, wa, ano-, hahaha, nanka, gohho, kakakaka, ii and converted to Japanese.
Here, the score determination unit 104 performs score determination on a sentence-by-sentence basis. That is, the score determination unit 104 sums up the speech recognition reliability and the non-language speech recognition reliability in the recognition result text of one sentence. For example, in the example of FIG. 4, the score determination unit 104 calculates the total value of the reliability of the recognition result text in one sentence and the total value of the reliability of the event (the reliability of each of the positive event and the negative event). Based on this total value, the score determination unit 104 determines whether the recognition result text is appropriate and whether an event exists. In the example of FIG. 4, since the total value of the reliability of the recognition result text and the positive event is equal to or greater than a predetermined value, the score determination unit 104 determines that the recognition result text includes speech, laughter, coughing, or keyboard sounds.

Then, the score determination unit 104 outputs each recognition result text whose reliability is equal to or greater than a predetermined value ("I," "I," "um," "something," "good") to the result output unit 105, as well as outputting event information (laughing, coughing, keyboard sounds). The result output unit 105 acquires event tag information from the event information and outputs it together with the recognition result text whose reliability is equal to or greater than a predetermined value.

This makes it possible to output event tag information on a sentence-by-sentence basis and add event tag information to the end of the recognition result text, resulting in easier-to-read text.

Next, the operation of the speech recognition device 100 of the present disclosure will be described. FIG. 5 is a flowchart showing the operation of the speech recognition device 100. The speech acquisition unit 101 acquires a speech waveform signal (S101), performs speech section detection from the speech waveform signal, and outputs the speech (or other sound) of the speech section as a speech signal to the speech recognition unit 102 and the non-language speech recognition unit 103 (S102).

The speech recognition unit 102 processes the speech signal for speech recognition and outputs the recognition result text, reading, and reliability (S103). The non-language speech recognition unit 103 processes the speech signal for non-language speech recognition and outputs the reliability of each event for each recognition target time (S104).

The score determination unit 104 determines the validity of the recognition result text or event for each recognition target based on the reliability of the recognition result text from the speech recognition process and the reliability of each event recognized by the non-language speech recognition (S105).

The result output unit 105 selects appropriate recognition result text from the recognition result text based on the judgment result, or obtains and outputs event tag information (S106).

This type of processing makes it possible to recognize speech and non-speech sounds during meetings, etc.

Next, application examples for other languages will be explained. The above speech examples are for Japanese, but similar processing is naturally possible for other languages. Figure 6 is a diagram showing processing when the speech is in English. Note that the description has been simplified for ease of explanation. In Figure 6, the following speech is made:

Utterance: I go (laughing: hahaha) to (cough: off coff cough) (keyboard sounds: clatter) school.
Recognition result text: I go ah the her head to Costa Rica.
Here, the speech recognition reliability of the recognition result text "I", "go", "to", and "school" recognized by the speech recognition unit 102 is high.

On the other hand, the recognition result texts "the her head," "Costoco," and "caca gratta" recognized by the speech recognition unit 102 have low recognition result text reliability but high event reliability. For example, the recognition result text "the her head" has a high laughter event reliability. This is because the laughter "hahaha" was attempted to be recognized as a speech sound.

The same is true for the recognition result texts "costoco" and "caca grata", which are derived with low speech confidence as a result of attempting to recognize a cough or keyboard sounds, respectively, as speech sounds. On the other hand, since these utterances are coughs and keyboard sounds, it is necessary not to output the recognition result text for them.

In the example of Figure 6, the following results are output based on the recognition result text confidence and event confidence:

Result output: I go to school.
In this way, it is possible to prevent non-speech sounds from being output even for English speech recognition.

Next, we will explain the process of adding event tag information (a mark such as an image, symbol, or text) indicating what event the non-speech sound is related to when the speech recognition device 100 of the present disclosure recognizes a non-speech sound. In the following, when there are expressions and processes unique to Japanese, they will be explained using Japanese notation.

Figure 7 shows a specific example. Figure 7(a) shows the actual speech content. Figure 7(b) shows the result output based on the recognition result.

7 shows a conversation between users A and B. To simplify the explanation, the content of the conversation is omitted and indicated as "---". In FIG. 7(a), user A is laughing while talking, and user B is making interjections in response to the conversation. The speech recognition unit 102 and the non-language speech recognition unit 103 can recognize who is speaking based on the speech channel or sound source separation, etc.
For example, written in English, it would be as follows:
A: I am Japanese. (Laughs) I live in Tokyo.
B: yeah (answer) yeah (answer)

In FIG. 7, the speech recognition unit 102 and the non-language speech recognition unit 103 distinguish between the speech (sound sources) of user A and user B, and perform speech recognition processing and non-language speech recognition processing, respectively. The speech recognition unit 102 and the non-language speech recognition unit 103 recognize the elapsed time of the speech, and perform speech recognition processing and non-language speech recognition processing on the speech of user A while recognizing the elapsed time of the speech. Similarly, the speech recognition unit 102 and the non-language speech recognition unit 103 perform speech recognition processing and non-language speech recognition processing on the speech of user B while recognizing the elapsed time of the speech. Therefore, the result output unit 105 can use these elapsed times to indicate that user B is making a backchannel response to the speech of user A.

Figure 8 is a diagram showing an example of processing determination for user A and user B. As shown in the figure, the speech recognition unit 102 and the non-language speech recognition unit 103 distinguish between user A and user B, respectively, based on a sound source separation technique, and perform speech recognition processing and non-language speech recognition processing.

As shown in FIG. 8, the speech recognition unit 102 and the non-language speech recognition unit 103 that recognize the speech of user A recognize the speech of user A, "It's." (pronounced "desu" and a verb in Japanese). This "It's." indicates the ending of a word in Japanese. In Japanese, endings become verbs, but in other languages, they do not necessarily become verbs. In addition, the speech recognition unit 102 and the non-language speech recognition unit 103 that recognize the speech of user B recognize the backchannel event "un" by user B. In addition, the speech recognition unit 102 and the non-language speech recognition unit 103 each keep track of the passage of time. Although omitted in FIG. 8, the elapsed time is associated and managed for each recognition result text. When recognizing the ending of a word in an English conversation, it is determined from the entire conversation.

The result output unit 105 outputs "desu" (desu) of user A and the event tag information (backchannel mark) of user B in accordance with the determination process of the score determination unit 104. At that time, the result output unit 105 outputs the recognized recognition result text "desu" (desu) and the event tag information: backchannel mark at positions according to the time lapse of the utterance.

Figure 7(b) is a diagram showing a specific example of the result output. As shown in the figure, the result output unit 105 outputs an interjection mark in accordance with the elapsed time of user B's speech. As a result, this interjection mark corresponds to the position of the recognition result text of user A's "Watashi wa ------ desu." (meaning "I am ------.").

In this way, by adding event tag information according to the content and timing of speech of multiple users, the atmosphere of the conversation content can be accurately expressed.

Figure 9 is a diagram showing an example of result output in another example. As shown in the figure, Figure 9 (a) shows the content of the utterance. The speech recognition unit 102 and the non-language speech recognition unit 103 recognize the content of the utterance of user A and user B, respectively. The result output unit 105 outputs the recognition result text and event tag information according to the judgment result of the score judgment unit 104.

In FIG. 9B, the result output unit 105 adds event tag information to the recognition result text on a sentence-by-sentence basis. That is, event tag information is added to each user's utterance. In FIG. 9B, the result output unit 105 adds user B's event tag information: backchannel mark x1) to user A's recognition result text "I am ---" (meaning I am ---). Similarly, the result output unit 105 adds user A and B's event tag information: A's comedy mark x1, B's backchannel mark x1 to the recognition result text of user A's next utterance "What is ---" (meaning I think ---. Note that this is not limited to think, and varies depending on the conversation). As described above, the non-verbal speech recognition unit 103 recognizes the time when user B's backchannel occurs, and can insert it into user A's recognition result text according to that time.

In FIG. 9(c), the result output unit 105 outputs the recognition result text and event tag information "It's ----. What's ---- (meaning I am ----. I think ----.) Mr. A: 1 laughing mark, Mr. B: 2 interjection marks" for each paragraph (each user who spoke). The result output unit 105 then operates to output the event tag information to the end of the recognition result text of user A's speech.

That is, the result output unit 105 adds event tag information of a mark indicating that user A laughed and a mark indicating that user B gave a backchannel to the recognition result text and outputs the result. At that time, the non-verbal speech recognition unit 103 recognizes that user A laughed once and that user B gave a backchannel twice, and the result output unit 105 adds a numerical value indicating the number of times to the mark. Note that, here, utterances and responses thereto in a conversation between multiple users are considered to be one paragraph unit. A paragraph unit means a user unit that spoke the recognition result text, but may also be defined based on the interval between utterances.

In the example of Figure 9 (d), similar processing is performed, but the result output unit 105 adds a laughing mark to the recognition result text of user A, and adds event tag information (answer mark), which is the recognition result of user B, to the end of the recognition result text of user A.

That is, the result output unit 105 outputs the recognition result text and event tag information of user A based on the utterance of a speaker including the recognition result text, i.e., user A, and, if user B's utterance is only an event, outputs the event tag information of user B to the end of the recognition result text (or event tag information) of user A. Note that, if the recognition result of user B's utterance includes the recognition result text, the result output unit 105 may output the event tag information together with the recognition result text of user B.

Next, a result output example in another example is shown. FIG. 10 is a diagram showing a result output example in another example. According to this figure, the result output unit 105 can recognize a break in a topic based on the speech content (or the recognition result text) of user A and user B. For example, when the result output unit 105 detects a character string indicating an intention to change the topic, such as "by the way," it performs a process of adding a mark to the end of the topic up to that point. In FIG. 10, the result output unit 105 determines that the point where user B makes a backchannel "hmmm" (hmmm) is the end of one topic, and adds a laughing mark for user A and a backchannel mark for user B to the end of that topic. In addition, a topic estimation engine or a topic segmentation engine, which are publicly known technologies, can be used to estimate a topic or segment a scene.

This allows event tag information to be added to each topic. Topics can be used to distinguish between explanation periods and question-and-answer periods in a meeting, for example.

In FIG. 10(b), a mark corresponding to the number of times or a number indicating the number of times is added, but if an event has occurred a predetermined number of times or more, an image mark emphasizing the frequency may be added. For example, if there are 10 or more laughs, a big smiley face mark may be added.

7 and 9 show an example in which laughter occurs, but the voice recognition device 100 may prepare event tag information (image, mark, text) for each event and according to the frequency of occurrence in a storage unit such as a memory (not shown), and change the event tag information according to the frequency. In other words, a reference value for the frequency of occurrence may be prepared, and the event tag information may be changed according to the reference value.

For example, if laughter occurs two or more times, a mark indicating laughter is added, and if coughing occurs five or more times, a mark indicating coughing is added. Event tag information may also be based on frequency, for example, if coughing occurs less than five times, a mark indicating coughing is added, and if coughing occurs five or more times, a mark indicating a worried face is added. In addition, in a meeting or the like, in sections with many laughter events (above a specified frequency), an approval mark can be used instead of a laughing mark.

In addition, for keyboard sounds, if a keyboard sound occurs a predetermined number of times or less, a mark may be added to display a warning to that effect. If the volume is above a certain level, a mark indicating a stronger indication (for example, that the keystrokes are loud) may be added. Furthermore, if speech recognition is performed by the speech recognition unit 102 and the recognition result text is output above a certain level (if the reliability of the recognition result text is above a predetermined value), a mark indicating an even stronger warning message may be added.

These reference values or predetermined number of times may be preset values, or may be determined based on the frequency of occurrence of an event in the entire conversation. For example, the frequency of occurrence of a certain event in all utterances may be averaged over time (or over paragraphs or sentences, etc.) to set the reference value (or predetermined number of times), and if the frequency of occurrence of an event is greater than this, event tag information may be added.

Also, instead of a mark, the event tag information may be an emoticon such as a smiley face or a video. Also, instead of adding event tag information, the color of the characters in the recognition result text of the immediately preceding utterance may be changed. For example, for a smiley face, the recognition result text may be a bright color (such as yellow) or the background may be a bright color (the color of blue sky). For example, for a sigh, the recognition result text may be a dark color (such as gray) or the background may be a dark color (gray).

In the above, the contents of the event tag information are changed depending on the type of event, the score determined by the score determination unit 104, and the frequency of the event, but when multiple events occur, the display format of the event tag information may be changed depending on the proportion of occurrence of each event relative to the total number of events.

In addition, if a dark event such as a sigh occurs, a warning or advice may be displayed and a message may be output to speak in a cheerful voice.

In the present disclosure, the speech recognition device 100 may perform speech recognition processing and non-verbal speech recognition processing on one or more user conversations in real time, or may perform speech recognition processing and non-verbal speech recognition processing on recorded conversation data.

When performing real-time speech recognition processing and non-language speech recognition processing in a conference or the like, the speech recognition device 100 may output (display) only to a specific person (terminal), or output (display) so that everyone can see. Also, in a web conference or the like, if there is a display column associated with each speaker (terminal), it may be displayed in the column corresponding to that speaker. It may also be displayed only to user A, or it may be displayed on the screens of other participants such as users C or D.

Next, we will explain a case where event tag information and recognition result text are associated with each other and can be switched between display. Figure 11 is a diagram showing an example. Figure 11(a) shows the actual speech content, and Figure 11(b) shows an example of the result output. As shown in Figure 11(b), the result output unit 105 outputs the recognition result text recognized by the speech recognition unit 102 and the event tag information recognized by the non-verbal speech recognition unit 103. At that time, the result output unit 105 links the event tag information (laughing mark L1 or backchannel marks L2, L3) with the recognition result text and manages the recognition result output.

When the result output unit 105 displays the recognition result text and event tag information on the display as the result output, if a user viewing the display selects the event tag information L (L1 to L3) (e.g., by clicking with a mouse), the result output unit 105 accepts the selection and outputs the recognition result text of that event tag information to the display, which then displays it (Figure 11 (c)).

This process may be performed when the result of the determination by the score determination unit 104 satisfies a predetermined condition. That is, when a certain recognition result text and its corresponding event reliability do not reach a predetermined numerical value, it may be difficult for the score determination unit 104 to accurately determine which is valid. In that case, the result output unit 105 may obtain the recognition result text, the event tag information, and their respective reliability from the score determination unit 104, and enable switching between the display by user operation.

When the result output unit 105 outputs to another external terminal, the recognition result is output in association with the recognition result text. The external terminal displays the recognition result text and the event tag information, and when the event tag information L (L1 to L3) is selected by a user operation of the external terminal (e.g., by clicking with a mouse), the external terminal displays the recognition result text associated with the event tag information.

Next, the process of correcting the speech recognition result will be described. FIG. 12 is a block diagram showing the functional configuration of a speech recognition device 100a in a variant of the present disclosure. As shown in the figure, the speech recognition device 100a is configured to include a correction unit 106 in addition to the functional configuration of the speech recognition device 100 in FIG. 1.

The correction unit 106 is a part that corrects the recognition result text or event tag information of the result output that is output from the result output unit 105 and displayed on the display. This display displays the recognition result text and event tag information, and when the correction unit 106 receives a correction portion (a part of the recognition result text or event tag information) indicated by a pointer or the like in accordance with a user's operation, it displays one or more correction candidates corresponding to the correction portion on the display to the user. For example, the correction candidates are displayed in a pull-down menu.

The result output unit 105 associates the recognition result text and event tag information for the same utterance and manages (stores) them as correction candidates. The recognition result text includes text converted to kanji, text in hiragana, text in katakana, and other converted symbols or text.

Then, the correction unit 106 switches to one of the correction candidates selected by the user, and the result output unit 105 outputs and displays the correction candidate on the display. The correction candidates are the recognition result text and events recognized by the speech recognition unit 102 and the non-language speech recognition unit 103. As described above, the correction candidates include text converted to kanji, text remaining in hiragana, text remaining in katakana, or other converted symbols or text.

Figure 13 is a diagram showing a specific example of the correction. Figure 13(a) shows the spoken content, and Figure 13(b) is a diagram showing an example of a correction screen by the user. As shown in the figure, the correction unit 106 moves the pointer P according to the user's operation. When the correction point indicated by the pointer P is selected, the correction unit 106 displays a correction candidate B. When the user selects an arbitrary candidate from the correction candidates B, the result output unit 105 outputs the candidate to the display as a result output. In Figure 13, the correction candidates B include "mother" and it is also possible to select text in which "hahaha" is recognized as it is. Note that in this disclosure, "mother" is erroneously recognized text.

In the figure, the event tag information is described as the subject of correction, but of course the recognition result text can also be the subject of correction.

Next, the aggregation of the frequency of occurrence of events will be described. The voice recognition device 100a in a modified example of the present disclosure may aggregate the frequency of events when performing voice recognition. By aggregating the frequency of occurrence of events, it is possible to determine the importance of the conversation, the quality of the conversation, etc. FIG. 14 is a block diagram showing the functional configuration of a voice recognition device 100b in another modified example. In addition to the functional configuration of the voice recognition device 100, this voice recognition device 100b is equipped with an aggregation unit 107.

The tallying unit 107 is a part that tallies the occurrence frequency of events recognized by the non-language speech recognition unit 103 as determined by the score determination unit 104. For example, the tallying unit 107 tallies the types of events, such as laughter, nodding, etc., determined by the score determination unit 104, and their occurrence frequency. In this case, the tallying unit 107 may aggregate by time period, paragraph, topic, or speaker. It may also aggregate by the presence of long pauses, changes in paragraphs, etc. This allows a user analyzing the speech recognition results to determine which utterances are important and their importance, based on the frequency of laughter or nodding for each classification, such as time period or topic.

In addition, when determining the division of a topic, the aggregation unit 107 determines a predetermined character string (e.g., "by the way") obtained by the speech recognition unit 102 through speech recognition as the division of a topic, but of course the topic may be determined by other methods. Similarly, the speech recognition unit 102 and the non-language speech recognition unit 103 can distinguish speakers based on speakers distinguished by sound source separation or speech channel.

Figure 15 is a graph showing the frequency of event types for each topic. In order to provide this graph, the speech recognition device 100b can provide information on the frequency of occurrence of each event for each topic. Note that a table may be used instead of a graph. Also, instead of a topic, the frequency of occurrence of each event for each speaker may be provided. This makes it possible to obtain an analysis result of whether each topic or speaker is a positive or negative topic or speaker.

Similarly, the time at which an event occurred may be referenced and classified by time period. A time period in which there was a lot of laughter or nodding may be considered a positive time period, whereas a time period in which there was little laughter or nodding or a lot of sighing may be considered a negative time period, and these can be used as the analysis results.

Next, the effects of the speech recognition device 100 disclosed herein will be explained.

The speech recognition device 100 of the present disclosure includes a speech acquisition unit 101 that functions as a sound information acquisition unit that acquires sound information, a speech recognition unit 102 that acquires recognition result text of user A's speech (first sound information) from the sound information, a non-verbal speech recognition unit 103 that determines that user B's speech (second sound information) (related to) user A's speech (first sound information) is the occurrence of a non-verbal event (e.g., a nod), and a result output unit 105 that associates event tag information indicating the event with the recognition result text and outputs it.

With this configuration, the occurrence of an event can be associated with the recognition result and presented to the user, etc. For example, event tag information (additional information) such as a mark indicating laughter, which is one of the non-speech sound recognition results, can be added to the recognition result text, or the recognition result text can be processed by coloring it, etc. Therefore, when the occurrence of an event is visually associated, a user who receives the event tag information can intuitively and intuitively recognize the event. In the present disclosure, adding event tag information to the recognition result text enables intuitive or intuitive recognition of the event, whether by the user or a different user.

These marks and other additional information may be represented by symbols or by color.

In addition, the result output unit 105 in the speech recognition device 100 associates the event tag information with the recognition result text and outputs it when the frequency of occurrence of the event satisfies a predetermined condition.

For example, if the frequency of laughter is greater than a certain number of times, a mark indicating laughter is added to the recognition result text. This allows the user to intuitively grasp the atmosphere of the conversation. In other words, the atmosphere of a conversation is different if laughter occurs once or twice compared to if laughter occurs more than once.

Furthermore, the result output unit 105 associates user B's event tag information (information indicating an event) with the recognition result text for a group of sound information (user A's speech and user B's speech) that includes user A's speech and satisfies a predetermined condition. For example, the predetermined condition is that the group of sound information is divided into sentences, paragraphs, or topics. By adding event tag information to the end of a sentence, paragraph, or topic, it is possible to easily grasp the event in a coherent conversation.

In addition, when an event such as that of user B occurs, the result output unit 105 adds event tag information to the end of the recognition result text of user A. In other words, event tag information is added to the end of the recognition result text for utterances by different speakers. This allows the atmosphere of the conversation to be intuitively grasped.

The speech recognition device 100a further includes a correction unit 106 that corrects the recognition result text or information indicating an event. The speech recognition unit 102 performs speech recognition processing on the speeches (first sound information and the second sound information) of user A and user B to obtain the respective recognition result texts. The non-language speech recognition unit 103 performs non-language speech recognition processing on the speeches (first sound information and the second sound information) of user A and user B to determine the occurrence of each event. The correction unit 106 corrects the respective recognition result texts or event tag information of these pieces of sound information.

This allows you to correct the recognition result text and event tag information to obtain correct recognition results.

In addition, in the voice recognition device 100, the voice recognition unit 102 and the non-language voice recognition unit 103 perform voice recognition processing and event occurrence judgment for the sound information (first sound information, second sound information) of both users A and B. Then, the result output unit 105 acquires the recognition result text and event tag information according to the judgment result of the score judgment unit 104. Then, the result output unit 105 outputs, for example, the event tag information of user B's laughter together with the recognition result text of that laughter to a display. The operating user looking at this display can switch between the event tag information of user B and the recognition result text for display. In other words, when the operating user performs a switching operation (e.g., selection by the pointer), the result output unit 105 performs output control for switching between the display of the event tag information and the recognition result.

When the output destination is an external terminal rather than a display, the result output unit 105 outputs user B's event tag information and the recognition result text to the external terminal. This output corresponds to output control that enables switching between the event tag information and the recognition result text. On the external terminal side, the event tag information is displayed, and switching between the display of the event tag information and the recognition result text is enabled in accordance with user operations of the external terminal.

In the above embodiment, the reliability was determined and the result was output according to the determination within the speech recognition device 100, but these processes may be outsourced to an external terminal. That is, as a process for not outputting the result based on the speech recognition process, the speech recognition device 100 outputs the recognition result text from the speech recognition process and its reliability, as well as the recognition result (events, etc.) from the non-language speech recognition process and its reliability to the external terminal. The external terminal can obtain the recognition result text, etc. based on this information.

In addition, in the speech recognition device 100 of the present disclosure, the sound information processing unit further includes a score determination unit 104 that determines the recognition results of the speech recognition unit 102 and the non-language speech recognition unit 103, and a result output unit 105 that processes and outputs the recognition result text by the speech recognition unit 102 in accordance with the determination. That is, based on the determination result by the score determination unit 104, the result output unit 105 does not output the non-language sound part of the recognition result text, and outputs only the language part.

This allows for easy-to-read recognition result text to be obtained by not outputting recognition result text that contains non-speech sounds.

In addition, in the speech recognition device 100 of the present disclosure, the speech recognition unit 102 derives speech sound confidence that the sound information is language, i.e., confidence in the recognition result text, and the non-language speech recognition unit 103 derives non-language sound confidence that the sound information is non-language, i.e., confidence in the event. Then, the score determination unit 104 determines the recognition results (recognition result text and each event) by the speech recognition unit 102 and the non-language speech recognition unit 103 based on these confidences (speech sound confidence and non-language sound confidence).

Here, the reliability of an event indicating non-speech sound reliability indicates the reliability that the speech waveform signal (sound information) is a non-speech sound, and the reliability that the speech waveform signal (sound information) is not a non-speech sound. In other words, it indicates a positive event and a negative event, respectively.

Then, the score determination unit 104 performs a weighting process on at least one of the reliability of each of the positive events and the negative events, and performs a determination process.

By performing such weighting processing, it is possible to make judgments based on the user's attributes or type, and the content of the meeting.

In the above, non-verbal sounds include at least one of the following sounds: laughter, coughing, nodding, sneezing, and keyboard sounds, but are not limited to these.

In addition, in the present disclosure, the speech recognition unit 102 performs speech recognition in a predetermined speech recognition unit (sentence unit, phrase unit, word unit, etc.), and the non-language speech recognition unit 103 performs non-language speech recognition in time units corresponding to the predetermined speech recognition unit.

This makes it possible to recognize non-speech sounds in accordance with the units of speech recognition.

The score determination unit 104 may determine the recognition result in a determination unit different from the speech recognition unit. For example, speech recognition and non-language speech recognition may be performed on a word-by-word basis, and the score may be determined on a sentence-by-sentence basis.

The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and directly or indirectly connected (e.g., using wires, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

Functions include, but are not limited to, judgement, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, election, establishment, comparison, assumption, expectation, regard, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs the transmission function is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on the method of realization for either of these.

For example, the speech recognition devices 100, 100a, and 100b in one embodiment of the present disclosure may function as a computer that performs processing of the speech recognition method of the present disclosure. Below, the speech recognition device 100 will be described, but the same applies to the speech recognition devices 100a and 100b. Figure 16 is a diagram showing an example of the hardware configuration of the speech recognition device 100 according to one embodiment of the present disclosure. The above-mentioned speech recognition device 100 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

In the following description, the term "apparatus" may be interpreted as a circuit, device, unit, etc. The hardware configuration of the speech recognition apparatus 100 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Each function in the speech recognition device 100 is realized by loading a specific software (program) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned speech recognition unit 102 and non-language speech recognition unit 103, etc. may be realized by the processor 1001.

The processor 1001 also reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes the computer to execute at least a part of the operations described in the above-mentioned embodiment is used. For example, the voice recognition unit 102 may be realized by a control program stored in the memory 1002 and running on the processor 1001, and other functional blocks may be similarly realized. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a speech recognition method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmission/reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize at least one of, for example, Frequency Division Duplex (FDD) and Time Division Duplex (TDD). For example, the above-mentioned voice acquisition unit 101, etc. may be realized by the communication device 1004. The voice acquisition unit 101 may be implemented as a transmission unit and a reception unit that are physically or logically separated.

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

In addition, the speech recognition device 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

The notification of information is not limited to the aspects/embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), higher layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block)), other signals, or combinations thereof. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

The processing steps, sequences, flow charts, etc. of each aspect/embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or appended. The output information may be deleted. The input information may be transmitted to another device.

The determination may be based on a value represented by a single bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave, etc.), then these wired and/or wireless technologies are included within the definition of a transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

In addition, terms explained in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meaning.

As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

When used in this disclosure, the terms "include," "including," and variations thereof are intended to be inclusive, as is the term "comprising." Additionally, the term "or," as used in this disclosure, is not intended to be an exclusive or.

In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." In addition, the term may mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

100... speech recognition device, 101... speech acquisition unit, 102... speech recognition unit, 103... non-language speech recognition unit, 104... score determination unit, 105... result output unit, 106... correction unit, 107... aggregation unit.

Claims (9)

a sound information acquisition unit that acquires sound information;
a sound information processing unit that performs speech recognition processing on first sound information by one speaker among the sound information to obtain a recognition result, and determines that second sound information by a different speaker related to the first sound information is the occurrence of a non-verbal event, and outputs information indicating the event in association with the recognition result;
a correction unit that corrects the recognition result or the information indicating the event;
With
The sound information processing unit includes:
performing speech recognition processing on the first sound information and the second sound information to obtain respective recognition results, and performing non-language speech recognition processing on the first sound information and the second sound information to determine occurrence of each event;
the correction unit corrects the first sound information and the second sound information by using the recognition results or the information indicating the event,
Speech recognition device. a sound information acquisition unit that acquires sound information;
a sound information processing unit that performs speech recognition processing on first sound information by one speaker among the sound information to obtain a recognition result, and determines that second sound information by a different speaker related to the first sound information is the occurrence of a non-verbal event, and outputs information indicating the event in association with the recognition result;
Equipped with
The sound information processing unit includes:
performing a speech recognition process on the second sound information to obtain a recognition result;
outputting a recognition result of the second sound information together with information indicating the event;
performing output control to enable switching between presentation of the information indicating the event and presentation of the recognition result of the second sound information;
Speech recognition device. The sound information processing unit includes:
adding additional information indicating the event to the recognition result;
3. The speech recognition device according to claim 1 . The sound information processing unit includes:
outputting the recognition result and information indicating the event for each speaker;
When adding additional information indicating an event of the second sound information by the one speaker to the recognition result of the first sound information by the one speaker, the additional information is output in association with the recognition result.
4. The speech recognition device according to claim 3 . The sound information processing unit includes:
processing the recognition result according to the event;
The speech recognition device according to any one of claims 1 to 4 . When an occurrence frequency of the event satisfies a predetermined condition, the sound information processing unit outputs information indicating the event in association with the recognition result.
The speech recognition device according to any one of claims 1 to 5 . The sound information processing unit includes:
associating information indicating the event with a recognition result for a group of sound information that includes the first sound information and satisfies a predetermined condition;
The speech recognition device according to any one of claims 1 to 6 . The predetermined condition is that the sound information group is divided into sentence units, paragraph units, or topic units.
8. The speech recognition device according to claim 7 . the sound information processing unit, when an event caused by a speaker different from the one speaker of the first sound information occurs, adds information indicating the event to the end of the recognition result for the sound information group.
9. A speech recognition device according to claim 7 or 8 .

Images