JP7790111B2 - Information processing method, program, information processing device, and information processing system - Google Patents

Description

The present invention relates to an information processing method, a program, an information processing device, and an information processing system.

Conventionally, in speech recognition for recognizing audio recorded along with video, a technology has been known in which a language model is selected based on the results of image recognition of the video, and the selected language model is used to perform speech recognition. Specifically, for example, a language model is selected using attributes such as the gender and age of a person speaking in the video as the image recognition results, and speech recognition is performed using the selected language model.

However, even for the same person, a person's speech style can vary depending on the situation when the speech is made, such as the person they are speaking with and the setting in which the conversation takes place. For this reason, when selecting a language model based on a person's attributes, as in the conventional technology described above, the speech style according to the situation may not be reflected in the language model, which could reduce the accuracy of speech recognition.

The disclosed technology was developed in consideration of the above circumstances and aims to improve the accuracy of voice recognition.

The disclosed technology is an information processing method by a computer, in which the computer receives input of video data including a video of a speaker, and when the video data is input, outputs estimated information indicating the probability that each speech style corresponds to the speech style of the speaker for each of a plurality of speech styles corresponding to a plurality of language models stored in a storage unit, and when a setting is made to select a language model according to the speech style of the speaker, the computer selects a language model corresponding to the speech style with the highest probability from the plurality of language models stored in the storage unit as the language model according to the speech style of the speaker. and when a setting is made to generate a new language model according to the speaker's speech style based on the estimated information, generate a new language model according to the speaker's speech style based on the estimated information, provide the selected language model according to the speaker's speech style or the new language model to a speech recognition unit that performs speech recognition, and display text data resulting from speech recognition of the speech data included in the video data using the selected language model according to the speaker's speech style or the new language model on a display device.

This can improve the accuracy of voice recognition.

FIG. 1 illustrates an example of an information processing system according to a first embodiment. FIG. 2 illustrates an example of a hardware configuration of an information processing device. FIG. 2 illustrates an example of a hardware configuration of a terminal device. FIG. 2 is a diagram illustrating functions of the information processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating a functional configuration of a terminal device according to the first embodiment. 10 is a flowchart illustrating processing of the information processing apparatus of the first embodiment. FIG. 2 is a first diagram illustrating processing by the information processing apparatus according to the first embodiment. FIG. 10 is a second diagram illustrating the processing of the information processing apparatus of the first embodiment. 10 is a flowchart illustrating processing by an information processing apparatus according to a second embodiment. 10 is a flowchart illustrating processing of an information processing apparatus according to a third embodiment. FIG. 10 illustrates an example of a system configuration according to a fourth embodiment. FIG. 13 illustrates an example of a system configuration according to a fifth embodiment.

(First embodiment)
The first embodiment will be described below with reference to the drawings. Fig. 1 is a diagram illustrating an example of an information processing system according to the first embodiment.

The information processing system 100 of this embodiment includes an information processing device 200 and a terminal device 300, and the information processing device 200 and the terminal device 300 are connected via a network or the like.

In the information processing system 100 of this embodiment, the information processing device 200 has a voice recognition processing unit 220. In other words, the information processing system 100 of this embodiment is an example of a voice recognition system.

When the information processing device 200 receives input of video data including audio data from the terminal device 300, the speech recognition processing unit 220 estimates, from the video data, the speaker's speaking style corresponding to the situation in which the speech was made.

Then, the information processing device 200 performs speech recognition on the audio data included in the video data using a language model corresponding to the estimated speaking style, and outputs text data that is the result of the speech recognition. The text data may be output to the terminal device 300 and displayed on the terminal device 300, for example.

A speaker's manner of speaking, in other words, is the manner of speech, including the words and manner of speaking used by the speaker. In the following explanation, the manner of speech may be referred to as the speaker's speech style.

In this embodiment, the situations in which speech occurs specifically include, for example, a situation in which a speaker is reading aloud a sentence printed on paper or a sentence displayed on a display, a situation in which multiple speakers who are close friends are enjoying a conversation, a situation in which multiple speakers are in a superior-subordinate relationship and the subordinate is reporting to the superior, and a situation in which each speaker is meeting the other for the first time.

Furthermore, a speaker's speaking style may include, for example, written speech, spoken speech, and informal speech.

Written language is the language used in writing. Spoken language is the language used in conversation, and is more polite than informal speech. Informal speech is familiar language or everyday conversational language.

For example, if the speech situation is one in which a sentence is being read aloud, the speaker's speech style is likely to be the written language used when writing a sentence. Also, if the speech situation is one in which the speakers are close friends, the speaker's speech style is likely to be informal colloquial language. Also, if the speech situation is one in which the speakers are meeting for the first time, the speaker's speech style is likely to be colloquial language.

In this embodiment, the information processing device 200 performs speech recognition using a language model corresponding to written language when, for example, it is estimated from the speech situation shown in the video data that the speaker's speech style is likely to be written language.

The terminal device 300 of this embodiment transmits, for example, video data including audio data to the information processing device 200, and may be a smartphone, tablet terminal, or the like. Furthermore, the terminal device 300 may be, for example, an imaging device that acquires video data, or may be the imaging device itself.

Furthermore, the terminal device 300 of this embodiment may be a spherical imaging device and may be installed, for example, at the center of a conversation. In this way, by using a spherical imaging device as the terminal device 300, it is possible to capture video data of all speakers participating in a conversation.

In this way, the information processing device 200 of this embodiment estimates the speaker's speaking style from the speech situation indicated by the video data, and performs speech recognition using a language model corresponding to the speaking style. Furthermore, the information processing device 200 of this embodiment performs speech recognition using video data, which is time-series information that takes into account the situation before and after the utterance that is the target of speech recognition.

For this reason, in this embodiment, for example, even if the same person speaks in different situations, speech recognition can be performed using a language model appropriate to the situation in which the utterance was made, thereby improving the accuracy of speech recognition.

Note that in the example of Figure 1, the information processing system 100 includes one terminal device 300, but this is not limited to this. The information processing system 100 may include multiple terminal devices 300, and the terminal device 300 that transmits voice data to the information processing device 200 and the terminal device 300 that receives text data output from the information processing device 200 may each be separate terminal devices.

In addition, in this embodiment, voice data may be received from the terminal device 300, and text data may be output to a display device of the information processing device 200. In addition, in this embodiment, voice data may be input directly to the information processing device 200, and text data may be output to the terminal device 300. In this embodiment, voice data may be input directly to the information processing device 200, and text data may be output to a display device of the information processing device 200.

Furthermore, in the example of FIG. 1, the information processing device 200 has the voice recognition processing unit 220, but this is not limited to this. The voice recognition processing unit 220 may be realized by multiple information processing devices 200.

Next, the hardware configuration of the information processing device 200 and the terminal device 300 will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of the hardware configuration of an information processing device.

The information processing device 200 is constructed by a computer, and as shown in FIG. 2, includes a CPU 201, ROM 202, RAM 203, HD 204, HDD (Hard Disk Drive) controller 205, display 206, external device connection I/F (Interface) 208, network I/F 209, bus line B1, keyboard 211, pointing device 212, DVD-RW (Digital Versatile Disk Rewritable) drive 214, and media I/F 216.

Of these, the CPU 201 controls the overall operation of the information processing device 200. The ROM 202 stores programs used to drive the CPU 201, such as IPL. The RAM 203 is used as a work area for the CPU 201. The HD 204 stores various data such as programs. The HDD controller 205 controls the reading and writing of various data from and to the HD 204 under the control of the CPU 201.

The display (display device) 206 displays various types of information such as a cursor, menu, window, text, or image. The external device connection I/F 208 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I/F 209 is an interface for data communication using a communication network. The bus line B1 is an address bus, data bus, etc. for electrically connecting each component such as the CPU 201 shown in Figure 2.

The keyboard 211 is a type of input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 212 is a type of input device for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The DVD-RW drive 214 controls the reading and writing of various data from a DVD-RW 213, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R or the like. The media I/F 216 controls the reading and writing (storing) of data from a recording medium 215, such as a flash memory.

Figure 3 shows an example of the hardware configuration of a terminal device. The terminal device 300 of this embodiment includes a CPU 301, ROM 302, RAM 303, EEPROM 304, a CMOS sensor 305, an image sensor I/F 306, an acceleration and orientation sensor 307, a media I/F 309, and a GPS receiver 311.

Of these, CPU 301 is an arithmetic processing unit that controls the overall operation of terminal device 300. ROM 302 stores CPU 301 and programs used to drive CPU 301, such as IPL. RAM 303 is used as a work area for CPU 301. EEPROM 304 reads and writes various data, such as smartphone programs, under the control of CPU 301. ROM 302, RAM 303, and EEPROM 304 are examples of storage devices for terminal device 300.

The CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a type of built-in imaging device that captures images of a subject (mainly a self-portrait) under the control of the CPU 301 to obtain video data. Note that instead of a CMOS sensor, an imaging device such as a CCD (Charge Coupled Device) sensor may also be used.

The imaging element I/F 306 is a circuit that controls the operation of the CMOS sensor 305. The acceleration/azimuth sensor 307 is a variety of sensors, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I/F 309 controls the reading and writing (storage) of data from the recording media 308, such as flash memory. The GPS receiver 311 receives GPS signals from GPS satellites.

The terminal device 300 also includes a long-distance communication circuit 312, an antenna 312a for the long-distance communication circuit 312, a CMOS sensor 313, an image sensor I/F 314, a microphone (sound collection device) 315, a speaker 316, a sound input/output I/F 317, a display (display device) 318, an external device connection I/F (Interface) 319, a short-distance communication circuit 320, an antenna 320a for the short-distance communication circuit 320, and a touch panel 321.

Of these, the long-distance communication circuit 312 is a circuit that communicates with other devices via a communication network. The CMOS sensor 313 is a type of built-in imaging means that captures an image of a subject and obtains video data under the control of the CPU 301. The image sensor I/F 314 is a circuit that controls the driving of the CMOS sensor 313. The microphone 315 is a built-in circuit that converts sound into an electrical signal. The speaker 316 is a built-in circuit that converts an electrical signal into physical vibrations to generate sound such as music or voice. The sound input/output I/F 317 is a circuit that processes the input and output of sound signals between the microphone 315 and the speaker 316 under the control of the CPU 301.
Note that either the CMOS sensor 305 or the CMOS sensor 313 may be disposed near the display 318 and the other may be disposed on the back surface of the terminal device 300 .

The display 318 is a type of display means, such as a liquid crystal display or organic electroluminescence (EL) display, that displays images of subjects and various icons. The external device connection I/F 319 is an interface for connecting various external devices. The short-range communication circuit 320 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 321 is a type of input means that allows the user to operate the terminal device 300 by pressing the display 318. The display 318 is an example of a display unit that the terminal device 300 has.

In this embodiment, the terminal device 300 is a smartphone or tablet terminal, but is not limited to this. The terminal device 300 may also be a general computer having a hardware configuration similar to that of the information processing device 200 shown in FIG. 2.

Next, the functions of the information processing device 200 of this embodiment will be described with reference to Figure 4. Figure 4 is a diagram explaining the functions of the information processing device of the first embodiment.

The information processing device 200 of this embodiment has a speech recognition processing unit 220, an input receiving unit 230, a speech period detection unit 231, a speech acquisition unit 232, and a speech style estimation unit 233.

The speech recognition processing unit 220 also includes a speech recognition unit 221, a language model providing unit 222, a language model storage unit 223, and an output text generation unit 224.

The language model storage unit 223 stores multiple language models. Specifically, the language model storage unit 223 stores a first language model 241, a second language model 242, and a third language model 243.

The first language model 241 is a language model corresponding to written language, the second language model 242 is a language model corresponding to spoken language, and the third language model 243 is a language model corresponding to casual spoken language.

In this embodiment, the first language model 241, second language model 242, and third language model 243 each receive the speech recognition results output by the speech recognition unit 221 as input, and output to the language model providing unit 222 the probability of the next phoneme, character, or word appearing for each processing unit of the speech recognition unit 221.

The input receiving unit 230 receives input of video data transmitted from an external device. Note that in this embodiment, video data includes audio data and video data.

The speech section detection unit 231 detects sections in the video data where speech is taking place based on the audio data included in the input video data. The speech section detection unit 231 also outputs the audio data for each detected speech section to the audio acquisition unit 232, and outputs the video data to the speech style estimation unit 233.

The speech acquisition unit 232 acquires the speech data output from the speech period detection unit 231, extracts speech features from the speech data, and outputs them to the speech recognition unit 221 of the speech recognition processing unit 220. In other words, the speech acquisition unit 232 outputs the speech data acquired during a predetermined period designated as a speech period to the speech recognition unit 221.

MFCC is a well-known speech feature, but LPC (Linear Predictive Coding), FBANK (Log Mel-Filterbank Coefficients), etc. may also be used.

The speech style estimation unit 233 determines the circumstances under which the speaker spoke from the video data output from the speech section detection unit 231, and estimates the speaker's speech style based on the circumstances of the speech. The speech style estimation unit 233 then generates speech style estimation information indicating the estimation results, and outputs the speech style estimation information to the language model provision unit 222 of the speech recognition processing unit 220. In other words, the speech style estimation unit 233 generates speech style estimation information for each predetermined period of time that is considered to be a speech section, and outputs it to the language model provision unit 222. Details of the speech style estimation information will be described later.

The speech recognition unit 221 performs speech recognition based on the speech features input from the speech acquisition unit 232 and the language model selected by the language model provision unit 222, and outputs the recognition results. Specifically, the speech recognition unit 221 sequentially outputs text data, which is the result of speech recognition for each utterance section, to the output text generation unit 224 and each language model stored in the language model storage unit 223.

The processing unit of the speech recognition in the speech recognition unit 221 of this embodiment may be a phoneme, a character, a word, etc. The processing unit of the speech recognition in the speech recognition unit 221 is the same as the processing units of the first language model 241, the second language model 242, and the third language model 243 stored in the language model storage unit 223.
The language model providing unit 222 selects a language model corresponding to the speaker's speech style from among the multiple language models stored in the language model storage unit 223, based on the speech style estimation information input from the speech style estimation unit 233. Then, the language model providing unit 222 provides the selected language model to the speech recognition unit 221.

In this embodiment, the language model to be provided can be specified in advance by inputting a language model selection signal to the language model providing unit 222. Specifically, for example, if the speaking style of a speaker appearing in the video data has been specified in advance, the language model to be provided to the speech recognition unit 221 may be specified in advance by the language model selection signal. The language model selection signal may be input in advance by, for example, a user of the information processing system 100.

By doing this, speech recognition can be performed using the identified language model without being affected by the estimation results of the speech style estimation unit 233.

The output text generation unit 224 generates output text data by chronologically concatenating text data that is the result of speech recognition for processing units, such as phonemes, characters, and words, output from the speech recognition unit 221. The output text generation unit 224 then outputs the generated output text data to the terminal device 300.

Here, we will explain the speaking style estimation unit 233. The speaking style estimation unit 233 of this embodiment is machine-learned in advance using pairs of training data in which video data is input data and the speaking style of the speaker in the video (the language used in the speaker's speech) is used as correct answer data.

In other words, the speech style estimation unit 233 of this embodiment is a trained model that takes video data as input data and outputs speech style estimation information for multiple speech styles, including the probability that each speech style matches the speech style of the speaker in the video.

In this embodiment, for example, the input data included in the learning data is video data captured of a presenter giving a presentation to a large audience using a screen on which presentation materials are projected. In this case, the correct answer data for the video data (input data) during the period in which the presenter is speaking to the audience is a spoken speaking style. The correct answer data for the video data during the period in which the presenter is reading from the screen or materials in front of him/her is a written speaking style. The correct answer data for the video data during the period in which one audience member stands up and speaks is a spoken speaking style.

In addition, in this embodiment, for example, the input data included in the learning data is video data captured of a meeting with a small number of participants in a conference room or the like. In this case, the correct answer data for video data taken during a time period in which the speaker is reading a document will be a written speaking style, and the correct answer data for video data taken during a time period in which the atmosphere is formal will be a spoken speaking style. Furthermore, the correct answer data for video data taken during a time period in which the speaker is having a casual conversation with a colleague will be a casual spoken speaking style.

Furthermore, if a superior is included among the participants in a meeting, the correct answer data corresponding to the video data of the meeting taken during the time period in which the superior is speaking will be an informal speaking style. Furthermore, the correct answer data corresponding to the video data taken during the time period in which participants other than the superior are speaking to the superior will be an informal speaking style.

Furthermore, in this embodiment, if the speakers appearing in the video data that serves as input data are limited (for example, only members of a certain department of a certain company), individual members may be recognized and learning may be performed that also includes each person's speaking habits and interpersonal relationships. By training the speaking style estimation unit 233 in this way, the accuracy of estimating the speaker's speaking style can be improved.

Next, we will explain the speech style estimation information of this embodiment.

The speech style estimation information of this embodiment includes information indicating the probability of each of multiple speech styles. Specifically, for example, the speech style estimation information includes the probability that the input video data will be a speech style using written language (first speech style), a speech style using spoken language (second speech style), and a speech style using casual spoken language (third speech style).

Note that the probability for each speaking style is the probability that multiple speaking styles match the speaking style of the speaker in the video. The multiple speaking styles may be determined in advance.

An example of speech style estimation information is, for example, "the probability of the first speech style is 5%, the probability of the second speech style is 70%, and the probability of the third speech style is 25%." In this case, it can be seen that the speech style of the speaker in the video is most likely to be a spoken language style (the second speech style).

In the example of FIG. 4, the information processing device 200 is provided with the input acceptance unit 230, speech period detection unit 231, speech acquisition unit 232, and speech recognition processing unit 220, but this is not limited to this. Some or all of these units may be provided in a device other than the information processing device 200 that is capable of communicating with the information processing device 200. In other words, the information processing device 200 may be realized by multiple information processing devices.

Next, the functional configuration of the terminal device 300 will be described with reference to Figure 5. Figure 5 is a diagram illustrating the functional configuration of the terminal device of the first embodiment.

The terminal device 300 of this embodiment includes a video data acquisition unit 330, an output unit 340, and a communication unit 350.

The video data acquisition unit 330 acquires video data. Specifically, the video data acquisition unit 330 acquires video data captured by an imaging device or the like included in the terminal device 300.

The output unit 340 outputs various types of information from the terminal device 300. Specifically, the output unit 340 displays text data received from the information processing device 200 on the display 318.

The communication unit 350 controls communication between the terminal device 300 and the information processing device 200. Specifically, the communication unit 350 transmits video data from the terminal device 300 to the information processing device 200 and receives text data from the information processing device 200.

Next, the processing of the information processing device 200 of this embodiment will be described with reference to Figure 6. Figure 6 is a flowchart illustrating the processing of the information processing device of the first embodiment.

In this embodiment, the information processing device 200 receives input of video data from the terminal device 300 via the input receiving unit 230 (step S601). Next, the information processing device 200 detects speech segments from the video data via the speech segment detection unit 231 (step S602).

Specifically, for example, the speech period detection unit 231 detects a first period from a certain timing to another timing as a speech period of a first speaker, and detects a second period following the first period as a speech period of a second speaker.

Next, the information processing device 200 extracts audio data and video data from the video data using the speech section detection unit 231, outputs the audio data to the speech recognition unit 221, and outputs the video data to the speech style estimation unit 233 (step S603).

Following step S603, the information processing device 200 extracts speech features from the speech data using the speech recognition unit 221 (step S604), and proceeds to step S607, which will be described later.

Following step S603, the information processing device 200 inputs the video data to the speech style estimation unit 233 and acquires speech style estimation information (step S605).

Next, the information processing device 200 inputs the speech style estimation information to the language model providing unit 222, selects a language model to be provided to the speech recognition unit 221 from the language models stored in the language model storage unit 223, and provides it to the speech recognition unit 221 (step S606).

Specifically, the language model providing unit 222 refers to the probability of each language model included in the speech style estimation information, selects the language model with the highest probability from the language models stored in the language model storage unit 223, and provides it to the speech recognition unit 221.

Next, the information processing device 200 performs speech recognition for each processing unit in the speech recognition unit 221 based on the notified language model and speech features, and outputs the text data for each processing unit to the output text generation unit 224 (step S607).

Next, the information processing device 200 generates output text data that summarizes the text data for each processing unit using the output text generation unit 224, and outputs the output text data to the terminal device 300 (step S608).

In this embodiment, the speaking style according to the speaker's speech situation is estimated based on video data of the speaker, and a language model corresponding to the estimated speaking style is selected and used for speech recognition.

Accordingly, this embodiment makes it possible to estimate a speaker's speaking style based on time-series information (video data) that takes into account the circumstances before and after the utterance that is the subject of speech recognition. Therefore, a speaking style that corresponds to the circumstances in which the speaker made the utterance can be used for speech recognition, thereby improving the accuracy of speech recognition.

The processing of the information processing device 200 of this embodiment will be described in detail below with reference to Figures 7 and 8. Figure 7 is the first diagram illustrating the processing of the information processing device of the first embodiment.

In Figure 7, the video data G1 received by the input receiving unit 230 is a captured image of two people, P1 and P2, who are relatively close, enjoying a conversation while eating and drinking.

The speech section detection unit 231 detects speech sections of person P1 and person P2 from this video data G1, outputs the audio data of the detected sections to the audio acquisition unit 232, and outputs the video data to the speech style estimation unit 233.

When video data of an utterance section in which person P1 is the speaker is input, the speech style estimation unit 233 outputs speech style estimation information for the utterance section.

In the example of Figure 7, person P1 and person P2 have a relatively close relationship. Therefore, the speech style estimation unit 233 generates speech style estimation information that is most likely to be the third language model 243, next most likely to be the second language model 242, and least likely to be the first language model 241, and outputs this information to the language model providing unit 222.

When this speech style estimation information is input, the language model providing unit 222 provides the third language model 243 with the highest probability to the speech recognition unit 221.

The speech recognition unit 221 outputs text data resulting from speech recognition based on the speech features of person P1's speech data and the third language model 243.

Next, when video data of a speech section in which the speaker is person P2 is input, the speech style estimation unit 233 outputs speech style estimation information for the speech section. At this time, just as with person P1, the speech style estimation unit 233 generates speech style estimation information in which the third language model 243 is most likely to be the third language model, the second language model 242 is next most likely to be the second language model, and the first language model 241 is least likely to be the first language model, and outputs this to the language model providing unit 222.

When this speech style estimation information is input, the language model providing unit 222 provides the third language model 243 with the highest probability to the speech recognition unit 221.

The speech recognition unit 221 outputs text data resulting from speech recognition based on the speech features of person P2's speech data and the third language model 243.

The information processing device 200 then generates output text data that combines the two pieces of text data and displays it on the terminal device 300.

Screen 71 shown in Figure 7 is an example of a screen on which output text data is displayed on the terminal device 300.

Screen 71 includes display areas 72 and 73. Display area 72 displays text data T1, which is the result of speech recognition performed on person P1's speech data using the third language model 243. Display area 73 displays text data T2, which is the result of speech recognition performed on person P2's speech data using the third language model 243.

In other words, in this embodiment, the text data resulting from speech recognition is displayed on screen 71 for each speaker.

On screen 71, text data T1 is "This candy is delicious," which is casual speech. Also on screen 71, text data T2 is "It's delicious, after all," which is also casual speech.

Figure 8 is a second diagram illustrating the processing of the information processing device of the first embodiment. In Figure 8, the video data G2 received by the input receiving unit 230 is an image of person P3 giving a presentation while displaying materials on the display 85.

The speech section detection unit 231 detects speech sections of person P3 from this video data G2, outputs the audio data of the detected sections to the audio acquisition unit 232, and outputs the video data to the speech style estimation unit 233.

When video data of an utterance section in which person P3 is the speaker is input, the speech style estimation unit 233 outputs speech style estimation information for the utterance section.

In the example of Figure 8, person P3 is reading a sentence displayed on display 85. Therefore, the speech style estimation unit 233 generates speech style estimation information that is most likely to be the first language model 241, next most likely to be the second language model 242, and least likely to be the third language model 243, and outputs this information to the language model providing unit 222.

When this speech style estimation information is input, the language model providing unit 222 provides the most likely first language model 241 to the speech recognition unit 221.

The speech recognition unit 221 outputs text data resulting from speech recognition based on the speech features of person P3's speech data and the first language model 241.

Screen 81 shown in Figure 8 is an example of a screen on which output text data is displayed on the terminal device 300.

Screen 81 displays text data T3, which is the result of speech recognition performed on person P3's voice data using the first language model.

On screen 81, the text data T3 is "In this case, the result will be like this," and is written in written form.

In this way, in this embodiment, the speaker's speech style, which indicates the language and manner of speaking used in the speaker's speech, is estimated based on video data (video data) capturing the situation in which the speaker is speaking, and speech recognition is performed using a language model corresponding to the speech style.

Therefore, for example, even if person P1 in Figure 7 and person P3 in Figure 8 are the same person, speech recognition can be performed using a language model that corresponds to the speaking style that corresponds to the situation in which the utterance was made, thereby improving the accuracy of speech recognition.

Furthermore, in this embodiment, for example, when displaying text data resulting from speech recognition, the terminal device 300 may display the speech style corresponding to the language model used for speech recognition.

Specifically, for example, on screen 71 shown in FIG. 7, information such as "A language model corresponding to casual speech was used" may be displayed in association with text data T1 and T2.

By displaying this information, the estimated speaking style can be presented to the user viewing screen 71.

In addition, in this embodiment, the speech style estimation unit 233 outputs information including the probability of each of multiple speech styles as speech style estimation information, but this is not limited to this. The speech style estimation unit 233 may, for example, output only the speech style with the highest probability as speech style estimation information.

Second Embodiment
The second embodiment will be described below with reference to the drawings. The second embodiment differs from the first embodiment in that a language model to be provided to the speech recognition unit 221 is generated using speech style estimation information and a plurality of language models stored in the language model storage unit 223. In the following description of the second embodiment, differences from the first embodiment will be described, and components having similar functional configurations to those of the first embodiment will be assigned the same reference numerals as those used in the description of the first embodiment, and their description will be omitted.

Figure 9 is a flowchart illustrating the processing of an information processing device according to the second embodiment. The processing from steps S901 to S905 in Figure 9 is similar to the processing from steps S601 to S605 in Figure 6, and therefore will not be described further.

Following step S904 in FIG. 9, the information processing device 200 causes the language model providing unit 222 to generate a language model to be provided to the speech recognition unit 221 based on the speech style estimation information output from the speech style estimation unit 233 (step S906).

The processing performed by the language model providing unit 222 in this embodiment is described below.

The language model providing unit 222 of this embodiment accepts input of speech style estimation information, generates a new language model using the probability of each speech style included in the speech style estimation information and the language model corresponding to each speech style, and provides it to the speech recognition unit 221.

Specifically, for example, suppose that the language model providing unit 222 receives input of speech style estimation information indicating that the speech style of the speech section is 5% to be a speech style using written language (first speech style), 70% to be a speech style using spoken language (second speech style), and 25% to be a speech style using casual spoken language (third speech style).

In this case, for example, when text data resulting from speech recognition of words in a processing unit by the speech recognition unit 221 is input to the first language model 241, the probability that word A will appear in the next processing unit is 5%. Similarly, when this text data is input to the second language model 242, the probability that word A will appear in the next processing unit is 10%, and when this text data is input to the third language model 243, the probability that word A will appear in the next processing unit is 20%.

In this case, the language model providing unit 222 of this embodiment adds the results of multiplying the probability of each speech style included in the speech style estimation information by the probability of appearance of word A in the language model corresponding to each speech style. The language model providing unit 222 then generates a language model in which the probability of appearance of word A in the next processing unit is equal to the calculated value, and provides this to the speech recognition unit 221.

In this case, the language model providing unit 222 determines that the occurrence probability of word A in the next processing unit is
Probability of first speaking style: 5% × Probability of appearance in first language model 241: 5% +
Probability of the second speaking style being 70% × probability of appearance in the second language model 242 being 10% +
25% probability of being the third speaking style × 20% probability of appearance in the third language model 243
= 12.25%
The language model is generated so as to satisfy the following condition and is provided to the speech recognition unit 221.

In this embodiment, the information processing device 200 proceeds to step S907 after step S906. Steps S907 and S908 are similar to steps S607 and S608 in Figure 6, so their description will be omitted.

In this embodiment, a language model generated based on the probability of each speech style included in the speech style estimation information and the occurrence probability of words in each processing unit in each language model is used for speech recognition. Therefore, in this embodiment, even if there is an error in the estimation result by the speech style estimation unit 233, the impact of the error can be reduced.

Furthermore, in this embodiment, speech recognition is performed using, for example, a language model that reflects the probability of occurrence of words for each language model corresponding to each speaking style. Therefore, in this embodiment, the accuracy of speech recognition can be improved even in cases where the probabilities of multiple speaking styles included in the speaking style estimation information are close to each other and it is difficult to estimate the speaking style.

(Third embodiment)
A third embodiment will be described below with reference to the drawings. The third embodiment differs from the first embodiment in that a language model is selected or generated according to settings for the information processing device. In the following description of the third embodiment, differences from the first embodiment will be described, and components having similar functional configurations to those of the first embodiment will be assigned the same reference numerals as those used in the description of the first embodiment, and descriptions thereof will be omitted.

The information processing device 200 of this embodiment can set the language model providing unit 222 to select a language model stored in the language model storage unit 223 based on the estimation result of the speech style estimation information, or to generate a new language model.

When the language model providing unit 222 of this embodiment receives speech style estimation information from the speech style estimation unit 233, it selects or generates a language model depending on the settings. Furthermore, if neither language model selection nor language model generation is set, the language model providing unit 222 of this embodiment provides the speech recognition unit 221 with the language model specified by the language model selection signal.

Figure 10 is a flowchart illustrating the processing of an information processing device according to the third embodiment. The processing from steps S1001 to S1003 in Figure 10 is similar to the processing from steps S601 to S603 in Figure 6, and therefore will not be described further.

In the information processing device 200 of this embodiment, when speech style estimation information is input, the language model providing unit 222 determines whether a setting for selecting a language model has been made (step S1004).

If a setting to select a language model is made in step S1004, the information processing device 200 proceeds to steps S604 and S605 in Figure 6.

If, in step S1004, the setting to select a language model has not been made and the setting to generate a language model has been made, the information processing device 200 proceeds to steps S904 and S905 in Figure 9.

If the setting to generate a language model has not been made in step S1005, the speech recognition processing unit 220 selects a language model from multiple language models stored in the language model storage unit 223 in accordance with the language model selection signal (step S1006).

Next, the information processing device 200 causes the speech acquisition unit 232 to extract speech features from the speech data output from the speech period detection unit 231 (step S1007), and proceeds to step S1008.

The processing of steps S1008 and S1009 in Figure 10 is similar to the processing of steps S607 and S608 in Figure 6, so description thereof will be omitted.

In this way, in this embodiment, it is possible to set whether a language model is selected based on the probability of each speech style included in the speech style estimation information, or whether a language model is newly generated using the speech style estimation information.

Furthermore, in this embodiment, for example, a predetermined threshold may be set for the probability of each speech style included in the speech style estimation information. Then, in this embodiment, it may be determined whether or not a setting for selecting a language model has been made based on the relationship between the probability of each speech style included in the speech style estimation information and the predetermined threshold.

Specifically, in this embodiment, if the highest probability value of each speech style included in the speech style estimation information is equal to or greater than a predetermined threshold, it may be determined that a setting to select a language model has been made.

In other words, in this embodiment, if the highest probability of each speech style included in the speech style estimation information is less than a predetermined threshold, a new language model is generated. The predetermined threshold may be, for example, 50%.

In this embodiment, by doing this, if the probabilities of each speech style included in the speech style estimation information are close to each other, a language model that reflects the word appearance probability in the language model corresponding to each speech style can be used for speech recognition. Therefore, according to this embodiment, the accuracy of speech recognition can be improved.

(Fourth embodiment)
The fourth embodiment will be described below with reference to the drawings. In the fourth embodiment, an example of a specific usage scene of the information processing system according to the first to third embodiments is shown.

Figure 11 is a diagram showing an example of the system configuration of the fourth embodiment. Figure 11 shows a case where any one of the first to third embodiments is used in a remote conference system.

The remote conference system 100A of this embodiment includes an information processing device 200, a hemispherical imaging device 400, and an electronic whiteboard 500, all of which are connected via a network.

In this embodiment, the hemispherical imaging device 400 and the electronic whiteboard 500 may be installed in geographically separate locations. Specifically, for example, the hemispherical imaging device 400 may be installed in a conference room of a business office located in City A, Prefecture A, and the electronic whiteboard 500 may be installed in a conference room of a business office located in City B, Prefecture B.

The hemispherical imaging device 400 captures hemispherical image data within the conference room. The hemispherical imaging device 400 also has a sound collection device and acquires audio data of speech made within the conference room. The hemispherical imaging device 400 may also include a communication device that transmits video data including the hemispherical image data and audio data to the information processing device 200.

The electronic whiteboard 500, which has, for example, a large display with a touch panel, detects coordinates on the board indicated by the user and displays strokes by connecting the coordinates, is an example of a display device. Note that the electronic whiteboard 500 is also sometimes called an electronic information board or electronic whiteboard.

The information processing device 200 of this embodiment has a voice recognition processing unit 220, and converts the speech of each speaker participating in a conference into text data based on, for example, video data acquired in a conference room where a hemispherical imaging device 400 is installed, and displays the text data on the electronic whiteboard 500. Note that the electronic whiteboard 500 may also display the video data acquired by the hemispherical imaging device 400 together with the text data.

In this embodiment, by using the information processing device 200 in this manner, for example, in a conference where multiple people with different positions are speaking, speech recognition can be performed using a language model that corresponds to the speaking style that corresponds to the position of the person speaking.

According to this embodiment, even if only text data resulting from speech recognition is displayed on the electronic whiteboard 500, the speaking style of each speaker is reflected in the text data. Therefore, in this embodiment, for example, viewers of the electronic whiteboard 500 can understand the speaker's position and interpersonal relationships.

Note that while FIG. 11 illustrates the video data being captured by a hemispherical imaging device 400, this is not limiting. The video data may also be acquired by a general imaging device, a spherical imaging device, or the like.

(Fifth embodiment)
The fifth embodiment will be described below with reference to the drawings. The fifth embodiment shows an example of a specific usage scene of the information processing systems of the first to third embodiments.

Figure 12 is a diagram showing an example of the system configuration of the fifth embodiment. Figure 12 shows a case where any one of the first to third embodiments is used in a counseling support system.

The counseling support system 100B of this embodiment may be installed in a location where counseling is provided by a counselor such as a clinical psychologist. The counseling support system 100B of this embodiment includes an information processing device 200, an imaging device 700, and a terminal device 800, all of which are connected via a network.

The imaging device 700 may be set up, for example, in a counseling room where counseling is conducted, and acquires video data including the person receiving counseling. In other words, the imaging device 700 acquires video data including video and audio of the person receiving counseling. The imaging device 700 then transmits the acquired video data to the information processing device 200. In the following description, the person receiving counseling may be referred to as the client.

The video data acquired by the imaging device 700 may also include images of the counselor and audio data showing the counselor's speech.

The terminal device 800 of this embodiment may be, for example, a terminal device carried by a counselor. The terminal device 800 may be, for example, a tablet-type terminal device having a display.

When the information processing device 200 acquires video data from the imaging device 700, it estimates the client's speaking style from the image of the client during counseling and converts the client's voice data into text data. Then, when the information processing device 200 receives a request to play the video data from the terminal device 800, it overlays the text data on the video data and displays it on the terminal device 800.

A client's speaking style during counseling is likely to reflect their psychological state. For example, if the text data converted from the client's voice data is written, it can be seen that the client is in a tense state. Also, if the text data converted from the client's voice data is spoken, it can be seen that the client is in a relatively relaxed state.

Also, for example, if the text data converted from the client's voice data changes from spoken to written language, it can be determined that the topic of the counseling session has changed to one that is making the client nervous.

In this embodiment, the voice data of the client during counseling is converted into text data that matches the client's speaking style and presented to the counselor, thereby assisting the counselor in understanding the client's psychological state. Furthermore, in this embodiment, by allowing the counselor to understand the client's psychological state during counseling, the counselor can learn whether appropriate topics were selected, etc.

In this embodiment, the counseling support system 100B is applied to counseling by a clinical psychologist or the like, but is not limited to this. The counseling support system 100B may also be used, for example, for career counseling for students and job seekers, or for interviews within organizations such as companies.

The functions of the above-described embodiments can be realized by one or more processing circuits. In this specification, the term "processing circuit" includes processors programmed to perform functions by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to perform the above-described functions.

Furthermore, the devices described in the embodiments represent only one of several computing environments for implementing the embodiments disclosed herein.

In one embodiment, information processing apparatus 200 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other over any type of communication link, including a network, shared memory, etc., and perform the processing disclosed herein. Similarly, information processing apparatus 200 may include multiple computing devices configured to communicate with each other.

Furthermore, the information processing device 200 can be configured to share the disclosed processing steps in various combinations. For example, a process performed by the information processing device 200 can be performed by another information processing device. Similarly, the functions of the information processing device 200 can be performed by another information processing device. Furthermore, the elements of the information processing device and the other information processing device may be integrated into a single information processing device, or may be separated into multiple devices.

The present invention has been described above based on various embodiments, but the present invention is not limited to the requirements set forth in the above embodiments. These aspects can be modified without departing from the spirit of the present invention, and can be determined appropriately depending on the application form.

REFERENCE SIGNS LIST 100 Speech recognition system 200 Information processing device 220 Speech recognition processing unit 221 Speech recognition unit 222 Language model providing unit 223 Language model storage unit 224 Output text generating unit 230 Input receiving unit 231 Speech period detecting unit 232 Speech acquisition unit 233 Speech style estimating unit 300 Terminal device

Japanese Patent Application Laid-Open No. 2004-333738

Claims (11)

An information processing method by a computer, comprising:
Accepts input of video data including video of the speaker,
When the video data is input, for each of a plurality of speech styles corresponding to a plurality of language models stored in a storage unit, output estimation information indicating the probability that each speech style matches the speech style of the speaker;
When a setting is made to select a language model according to the speech style of the speaker, a language model corresponding to the speech style with the highest probability is selected as the language model according to the speech style of the speaker;
generating a new language model according to the speech style of the speaker based on the estimated information when a setting is made to generate a new language model according to the speech style of the speaker;
providing a language model corresponding to the speech style of the selected speaker or the new language model to a speech recognition unit that performs speech recognition;
An information processing method, comprising: displaying on a display device text data resulting from speech recognition of audio data contained in the video data using a language model corresponding to the speech style of the selected speaker or the new language model . An information processing method by a computer, comprising:
Accepts input of video data including video of the speaker,
When the video data is input, for each of a plurality of speech styles corresponding to a plurality of language models stored in a storage unit, output estimation information indicating the probability that each speech style matches the speech style of the speaker;
selecting a language model according to the manner of speech of the speaker based on the estimated information when a setting is made to select a language model according to the manner of speech of the speaker;
generating a new language model using the plurality of language models and a probability for each speech mode included in the estimated information when a setting is made to generate a new language model according to the speech mode of the speaker;
providing a language model corresponding to the speech style of the selected speaker or the new language model to a speech recognition unit that performs speech recognition;
An information processing method, comprising: displaying on a display device text data resulting from speech recognition of audio data contained in the video data using a language model corresponding to the speech style of the selected speaker or the new language model . The computer
When the highest probability value among the probabilities that each of the plurality of speech modes matches the speech mode of the speaker is less than a predetermined threshold value,
The information processing method according to claim 1 or 2 , further comprising determining that a setting is set to generate a new language model according to the speaker's speech style. The computer
The information processing method according to claim 1 , wherein the text data is displayed on the display device for each utterance of the speaker. The computer
5. The information processing method according to claim 4 , further comprising the step of displaying information indicating a speech style corresponding to the language model provided to the speech recognition unit on the display device together with the text data. Accepts input of video data including video of the speaker,
When the video data is input, for each of a plurality of speech styles corresponding to a plurality of language models stored in a storage unit, output estimation information indicating the probability that each speech style matches the speech style of the speaker;
When a setting is made to select a language model according to the speech style of the speaker, a language model corresponding to the speech style with the highest probability is selected from among a plurality of language models stored in a storage unit as the language model according to the speech style of the speaker;
generating a new language model according to the speech style of the speaker based on the estimated information when a setting is made to generate a new language model according to the speech style of the speaker;
providing a language model corresponding to the speech style of the selected speaker or the new language model to a speech recognition unit that performs speech recognition;
A program that causes a computer to execute a process of displaying on a display device text data resulting from speech recognition of the audio data contained in the video data using a language model that corresponds to the speech style of the selected speaker, or the new language model. Accepts input of video data including video of the speaker,
When the video data is input, for each of a plurality of speech styles corresponding to a plurality of language models stored in a storage unit, output estimation information indicating the probability that each speech style matches the speech style of the speaker;
selecting a language model according to the manner of speech of the speaker based on the estimated information when a setting is made to select a language model according to the manner of speech of the speaker;
When a setting is made to generate a new language model according to the manner of speech of the speaker,
generating a new language model using the probability for each mode of speech included in the estimation information and the plurality of language models;
providing a language model corresponding to the speech style of the selected speaker or the new language model to a speech recognition unit that performs speech recognition;
A program that causes a computer to execute a process of displaying on a display device text data resulting from speech recognition of the audio data contained in the video data using a language model that corresponds to the speech style of the selected speaker, or the new language model. an input receiving unit that receives input of video data including a video of a speaker;
an estimation unit that, when the video data is input, outputs estimation information indicating the probability that each speech style corresponds to a speech style of the speaker for each of a plurality of language models stored in a storage unit;
a model providing unit that, when a setting is made to select a language model according to the speaker's speech style, selects a language model corresponding to the speech style with the highest probability as a language model according to the speaker's speech style, and, when a setting is made to generate a new language model according to the speaker's speech style, generates a new language model according to the speaker's speech style based on the estimated information;
an output text generation unit that provides a language model corresponding to the speech style of the selected speaker or the new language model to a speech recognition unit that performs speech recognition, and displays text data resulting from speech recognition of the audio data included in the video data using the language model corresponding to the speech style of the selected speaker or the new language model on a display device. an input receiving unit that receives input of video data including a video of a speaker;
an estimation unit that, when the video data is input, outputs estimation information indicating the probability that each speech style corresponds to a speech style of the speaker for each of a plurality of language models stored in a storage unit;
a model providing unit that selects a language model according to the speaker's speech style based on the estimated information when a setting is made to select a language model according to the speaker's speech style, and that generates a new language model according to the speaker's speech style using the probability for each speech style included in the estimated information and the plurality of language models when a setting is made to generate a new language model according to the speaker's speech style;
an output text generation unit that provides a language model corresponding to the speech style of the selected speaker or the new language model to a speech recognition unit that performs speech recognition, and displays text data resulting from speech recognition of the audio data included in the video data using the language model corresponding to the speech style of the selected speaker or the new language model on a display device. 10. An information processing system comprising: the information processing device according to claim 8 ; and an imaging device that captures the video data. The information processing system according to claim 10 , wherein the imaging device is a spherical imaging device or a semi-spherical imaging device.