JP7575804B2 - Voice recognition program, voice recognition method, voice recognition device, and voice recognition system - Google Patents

Description

This invention relates to a voice recognition program, a voice recognition method, a voice recognition device, and a voice recognition system for performing voice recognition on recorded audio, etc.

A large number of audio files recorded using IC recorders or recording applications (apps) can be stored on IC recorders, etc. It is desirable to be able to efficiently find and play back the required audio files from among the many recorded audio files. It is also desirable to be able to specifically present to the user the speakers included in the audio data.

The number of speakers included in an audio file can be estimated using clustering techniques such as the k-means method. In clustering, the number of speakers is set in advance and the audio included in the audio file is divided into speakers based on the number of speakers. One technique for estimating the number of speakers is to set the number of speakers in advance before recording a meeting or other discussion, prepare microphones for each speaker, and estimate the number of speakers by processing to detect the direction of each speaker (see, for example, Patent Document 1 below).

JP 2009-301125 A

However, with conventional technology, estimating the number of speakers included in recorded data required cumbersome advance preparations, such as setting the number of speakers before recording the meeting, preparing microphones for each speaker, and processing to detect the direction of each speaker. In addition, although the estimated number of speakers had a certain degree of accuracy, it could differ from the number of speakers actually recorded, and in such cases the estimated number of speakers could not be easily corrected.

In addition, it would be desirable to be able to easily estimate who the specific speaker is in a recorded audio file.

In view of the above problems, the present invention aims to easily estimate the number and speakers of audio contained in an audio file without any prior setting.

To achieve the above object, the speech recognition program of the present invention is characterized in that it causes a computer to execute a process of estimating the number of speakers for each speaker included in a speech file, referencing a trained model for each speaker that has been prepared in advance, recognizing each speaker of the estimated number of speakers, and tagging the speakers included in the speech file.

The recognition process is characterized in that it presents to a user information on the estimated number of speakers and speaker candidates corresponding to the number of speakers, and recognizes each speaker of the number of speakers based on an operation by the user to identify the speaker from the speaker candidates, and the tagging process tags the speakers based on the operation by the user.

The estimation process is also characterized in that it presents the estimated number of speakers to the user, and re-estimates the number of speakers included in the audio file based on the user's operation to change the number of speakers.

Furthermore, the tagged speaker information is learned and accumulated, and the recognition process is characterized in that each speaker of the estimated number of speakers is recognized based on the learned model.

Another feature of the system is that the characters contained in the audio file are generated in real time when the audio is recorded or played back.

The speech recognition method of the present invention is characterized in that the computer executes a process of estimating the number of speakers for each speaker included in the speech file, referring to a trained model for each speaker that has been prepared in advance, recognizing each speaker of the estimated number of speakers, and tagging the speakers included in the speech file.

The voice recognition device of the present invention also includes a control unit that recognizes the number of speakers and the speakers included in the voice file, and the control unit estimates the number of speakers for each speaker included in the voice file, refers to a trained model for each speaker that has been prepared in advance, recognizes each speaker of the estimated number of speakers, and tags the speakers included in the voice file.

The speech recognition system of the present invention is a speech recognition system in which a terminal and a cloud are communicatively connected, the terminal has a voice recording unit and a communication unit that uploads recorded or played back voice files to the cloud, and the cloud estimates the number of speakers for each speaker included in the voice file, refers to a trained model for each speaker that has been prepared in advance, recognizes each speaker of the estimated number of speakers, and notifies the terminal of information tagged with the speakers included in the voice file.

The terminal is also characterized in that it has a display unit that presents to the user information on the number of speakers estimated by the cloud and speaker candidates corresponding to the number of speakers, transmits information on an operation by the user to identify the speaker from the speaker candidates to the cloud, and the cloud recognizes each speaker of the number of speakers based on the information on the operation to identify the speaker from the speaker candidates received from the terminal, and transmits information tagged with the speaker based on the user's operation to the terminal.

The control unit of the terminal also presents the number of speakers estimated by the cloud to the user via the display unit, and the cloud transmits to the terminal the result of re-estimating the number of speakers included in the audio file based on the user's operation to change the number of speakers received from the terminal.

The cloud also has a storage unit that stores the audio files uploaded from the terminal.

With the above configuration, after estimating the number of speakers included in an audio file, each speaker can be specifically recognized, making it easy to know the number of speakers included in the audio file and each speaker.

The present invention has the effect of easily estimating the number and speakers of the audio contained in an audio file without any prior setting.

FIG. 1 is a functional configuration diagram of a voice recognition system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the voice recognition device. FIG. 3 is a flowchart showing an example of a process related to voice recognition. FIG. 4 is a flowchart showing an example of a process for voice learning. FIG. 5 is a diagram showing an example of a table structure used for voice recognition. FIG. 6 is a transition diagram of processes for estimating the number of speakers and for speaker recognition. FIG. 7 shows a display screen on the terminal when recording for the first time. FIG. 8 shows the recording screen on the terminal during recording. FIG. 9 is a diagram showing a display screen on the terminal after the number of speakers has been estimated. FIG. 10 shows a display screen on the terminal of a speaker candidate. FIG. 11 is a diagram showing a display screen showing a list of speaker selections on a terminal. FIG. 12 is a diagram showing a display screen of the transcription on the terminal. FIG. 13 shows a display screen on the terminal when an audio file is being played back. FIG. 14 is a diagram showing an example of a waveform of an audio contained in an audio file. FIG. 15 is a diagram showing groupings of speakers included in an audio file. FIG. 16 is a diagram showing changes in the estimated number of speakers. FIG. 17 is a diagram showing changes in the estimated number of speakers. FIG. 18 shows a screen for adding speaker candidates on a terminal.

(Embodiment)
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a voice recognition program, a voice recognition method, a voice recognition device, and a voice recognition system according to the present invention will be described in detail below with reference to the accompanying drawings.

(System Overview)
1 is a functional configuration diagram of a voice recognition system according to an embodiment. The voice recognition system includes a terminal 100 that records voice, and a cloud 110. The terminal 100 is an IC recorder, a smartphone having a recording application, a tablet, a PC, or the like. In the following description, a configuration in which the terminal 100 such as an IC recorder or a smartphone records voice from a microphone will be described as an example, but the present invention is not limited to this, and the terminal 100 can also be configured to record a call with a counterpart party via a smartphone or the like.

The terminal 100 includes a microphone 101, a control unit 105, a keyboard 106, and a display 107. The control unit 105 includes a recording unit 102, a transcription unit 103, and a speaker tagging unit 104. The recording unit 102 stores the voices uttered by speakers (such as multiple participants in a conference) via the microphone 101 as audio files D. The control unit 105 transmits the audio files D to the cloud 110 when estimating the number of speakers and the speakers.

The transcription unit 103 performs speech recognition on the voice included in the audio file D to generate character data such as text. The speaker tagging unit 104 tags the audio file D with the number of speakers and speaker information of the voice included in the audio file D as tags. In the example system configuration of FIG. 1, the speaker tagging unit 104 obtains tagging information from the cloud 110 that estimates the number of speakers and the speakers, and tags the speakers included in the audio file D on the terminal 100 so that they can be identified.

The control unit 105 of the terminal 100 may be configured to call and execute each of the installed functions, for example, via an API (Application Programming Interface).

The cloud 110 has a plurality of PC groups, servers, and storage groups, and is communicatively connected to the terminal 100 via a network such as the Internet. The cloud 110 shown in the configuration example of FIG. 1 includes, for example, a storage server 120, a machine learning server 130, and a trained model database (DB) 140 that stores trained models.

The storage server 120 transmits and receives audio files D to and from the terminal 100. The storage server 120 temporarily stores the audio files D transmitted from the terminal 100 in the storage unit 121. The storage server 120 also temporarily stores the audio files D, which include information on the number of speakers and the speakers estimated by the machine learning server 130, in the storage unit 121, and transmits the audio files D to the terminal 100.

The machine learning server 130 has the functions of a speaker number estimation unit 131 and a speaker recognition unit 132. The trained model DB 140 holds trained models for estimating the number and speakers of audio file D. The trained models are the results of learning speaker recognition information for each voice, and are updatable and stored in the trained model DB 140 as the learning results of audio file D uploaded for each voice recognition request from the terminal 100.

The number of speakers estimation unit 131 of the machine learning server 130 performs speech recognition on the audio file D and estimates the number of speakers included in the audio file D. The speaker recognition unit 132 estimates the speakers included in the audio file D after estimating the number of speakers. The number of speakers estimation unit 131 and the speaker recognition unit 132 access the trained model in the trained model DB 140 and estimate the number of speakers and the speakers.

In the embodiment, the cloud 110 (machine learning server 130) estimates the number of speakers and the speakers contained in the audio file D, and transmits the estimation result to the terminal 100 once. The terminal 100 displays the number of speakers and speaker information of the audio file D estimated by the cloud 110 on the screen. Then, the user on the terminal 100 performs corrections and confirmations.

In this manner, in the embodiment, the number of speakers and speakers of audio file D estimated on the cloud 110 side are corrected or confirmed by an auxiliary operation performed by the user of terminal 100. This correction and confirmation operation information is transmitted from terminal 100 to cloud 110 (machine learning server 130). During these correction and confirmation processes, there is no need to transmit and receive audio file D itself between terminal 100 and cloud 110, and by transmitting only the correction and confirmation information regarding the number of speakers and the information regarding the speakers, the amount of transmitted data can be reduced.

When the number of speakers is corrected, the machine learning server 130 divides the speakers again for the audio file D according to the corrected number of speakers. When the speaker is corrected, the machine learning server 130 tags the corrected speaker for the audio file D.

In this way, in the embodiment, the number of speakers and the speakers are estimated based on the recorded audio file D without prior preparation. By allowing the estimated number of speakers and speakers to be modified by user operation, the estimation accuracy of the number of speakers and the speakers included in the audio file D can be improved, and the estimation process can be performed easily.

In the example shown in FIG. 1, a voice recognition system is configured in which the cloud 110 performs voice recognition on the voice file D recorded by the terminal 100. However, it is not limited to this. By arranging the speaker number estimation and speaker recognition functions arranged on the cloud 110 side in FIG. 1 in the terminal 100, it is also possible to configure a voice recognition device on the terminal 100 alone.

Fig. 2 is a diagram showing an example of the hardware configuration of a voice recognition device. For example, the terminal 100 shown in Fig. 1 has the configuration shown in Fig. 2. The terminal 100 includes a CPU 201, a ROM 202, a RAM 203, an external memory 204, a microphone 101, a keyboard 106, an input interface (I/F) 208, a video I/F 209, a display 107, a communication I/F 211, etc. The components 201 to 211 are connected to each other via a bus 220.

The CPU 201 functions as a control unit that controls the entire terminal 100. The ROM 202 stores a boot program for control. The RAM 203 is used as a work area for the CPU 201. That is, the CPU 201 controls the entire voice recognition device 100 by executing various programs stored in the ROM 202 while using the RAM 203 as a work area.

The external memory 204 may be a HDD, SSD, disk device, flash memory, etc., and stores data in a writable/readable manner under the control of the CPU 201.

The input I/F 208 is connected to a microphone 101 that captures the speaker's voice and a keyboard 106 that has multiple keys for inputting characters, numbers, various instructions, etc., and outputs data input from these to the CPU 201.

The video I/F 209 is connected to the display 107. Specifically, the video I/F 209 is composed of, for example, a graphics controller that controls the entire display 107, a buffer memory such as a VRAM (Video RAM) that temporarily records image information that can be displayed immediately, and a control IC that controls the display 107 based on image data output from the graphics controller.

Display 107 displays various data such as icons, cursors, menus, windows, or characters and images. For example, a TFT liquid crystal display or an organic EL display can be used as display 107.

The communication I/F 211 is connected to a network and functions as an interface for communication connection with the cloud 110. The network may be the Internet, a public line network, a mobile phone network, a LAN, a WAN, etc., which may be connected via a wired or wireless connection.

The terminal 100 shown in FIG. 1 realizes the functions of the terminal 100 by the CPU 201 executing a predetermined program using programs and data recorded in the ROM 202, RAM 203, external memory 204, etc. shown in FIG. 2. In addition, if the terminal 100 is a mobile device such as a smartphone or tablet, the keyboard 106 and the display 107 may be configured as a touch panel.

Furthermore, each of the servers 120 and 130 constituting the cloud 110 shown in FIG. 1 has a configuration similar to that shown in FIG. 2, with the CPU 201 functioning as a control unit and managing the overall processing.

Figure 3 is a flowchart showing an example of processing related to speech recognition. The speech recognition processing related to the above-mentioned speaker number estimation and speaker recognition is mainly performed by the cloud 110 (machine learning server 130).

First, the terminal 100 uploads a recorded audio file D to the cloud 110 (step S301). The terminal 100 may upload an already recorded audio file D. This audio file D is audio recorded by unspecified speakers, and the number of speakers is unknown. The terminal 100 uploads the audio file D to identify the number of speakers and the speakers. The cloud 110 stores the uploaded audio file D in the storage unit 121 of the storage server 120 (step S302).

Next, the control unit (number of speakers estimation unit 131, CPU 201) of the machine learning server 130 in the cloud 110 estimates the number of speakers included in the audio file D (step S303). The control unit assigns a unique ID to the audio in the audio file D according to the estimated number of speakers. By assigning the ID, the audio for each estimated speaker in the audio file D can be identified.

Next, the control unit accesses the trained model in trained model DB140 and determines whether a trained speaker model exists for audio file D (step S304).

If the determination result shows that a trained speaker model exists in audio file D (step S304: Yes), the control unit performs processing to recognize speakers for each of the estimated number of speakers contained in audio file D (step S305), and proceeds to processing in step S306. On the other hand, if the determination result shows that a trained speaker model does not exist in audio file D (step S304: No), the control unit proceeds to processing in step S306.

In step S306, the control unit generates a speaker recognition screen that reflects the results of the speaker recognition on the user interface (UI) (step S306). The control unit transmits this speaker recognition screen to the terminal 100.

As a result, a speaker recognition screen is displayed on the display 107 of the terminal 100. The speaker recognition screen has the number of speakers estimated by the above process and information on the recognized speakers (speaker candidates). In response to a user operation viewing the speaker recognition screen, the control unit of the terminal 100 sends feedback on the number of speakers (speaker recommendations) to the cloud 110 (machine learning server 130) (step S307).

In this feedback, the user operating the terminal 100 modifies and confirms the estimated number of speakers displayed on the speaker recognition screen, and modifies and confirms the recognized speakers (speaker candidates). Thus, in the embodiment, the number of speakers and speaker candidates estimated by the cloud 110 are modified and confirmed by the user of the terminal 100.

As a result, the cloud 110 (control unit of the machine learning server 130) identifies the speaker for the audio file D through the correction and confirmation operations by the user of the terminal 100, and performs tagging to identify the identified speaker (step S308). The tag information is transmitted from the cloud 110 to the terminal 100, and the terminal 100 stores the received tag information in association with the audio file D. Through the above processing, the terminal 100 can display the number of speakers and the speakers included in the audio file D on the display 107.

Figure 4 is a flowchart showing an example of processing for audio learning. The control unit of the cloud 110 machine learning server 130 performs the processing shown in Figure 4 for each processing for one audio file D shown in Figure 3, and constructs the trained model DB 140.

First, after processing step S308 (see FIG. 3), the control unit of the machine learning server 130 extracts speaker sound sources from the corresponding audio file D (step S401). The processing in FIG. 3 makes it possible to identify the sound sources for each speaker contained in the audio file D.

As a result, the machine learning server 130 performs learning on the identified speaker (step S402), and stores the speaker model, which is the learning result, in the trained model DB 140 (step S403). This allows learning of the voice of each speaker contained in the audio file D, and by repeating the learning, it becomes possible to improve the accuracy of speaker recognition.

Figure 5 is a diagram showing an example of a table structure used for speech recognition. These tables 501 to 504 are held by the control unit of the cloud 110 and are used for the above-mentioned speaker number estimation and speaker recognition processing.

Figure 5 (a) shows an audio table (audios) 501 that identifies audio files D uploaded from the terminal 100, and is held by the control unit of the storage server 120 of the cloud 110. The control unit of the storage server 120 assigns an identifier (id) to each uploaded audio file D and stores it as an audio table (audios). For example, the audio file D with id "0001" is "file0001.mp3".

Figures 5(b) to (d) show tables 502 to 504 held by the control unit of the machine learning server 130 in the cloud 110. The control unit of the machine learning server 130 reads the audio file D stored in the storage unit 121 of the storage server 120, and generates and references these tables when performing the above-mentioned speaker number estimation and speaker recognition processing.

Figure 5 (b) is a table (audio_predictions) 502 for speaker recognition. This table 502 includes information on the IDs for each audio file D (audio_id), the speaker IDs (speaker_id) estimated for each ID, and the recognition accuracy (confidence). For example, the ID "0001" indicates that the recognized speaker (speaker_id) included in the audio file D "file0001.mp3" is "0001", and the recognition accuracy (confidence) of this speaker "0001" is "0.8 (80% confidence)."

Figure 5 (c) is a table (speakers) 503 for speaker recognition. This table 503 contains information on the names of speakers recognized by ID. For example, the speaker (name) of ID "0001" is "Alice."

Figure 5 (d) is a table (audio_speakers) 504 for the estimated number of speakers/speakers after recognition for audio file D. This table 504 includes information on audio files D (audio_id) by ID and the estimated speakers (speaker_id). For example, for one audio file D (audio_id) "0001", the estimated speakers (speaker_id) for two people, "0001" and "0002", are "NULL". In this case, since both of these people are "NULL", the speakers have not been specifically recognized, and the number of speakers is only estimated to be two.

In addition, id "0003" indicates the state in which a speaker has been recognized. In this case, for one audio file D (audio_id) "0002", one speaker (speaker_id) "0001", i.e. "Alice" in Figure 5(c) is recognized with 80% confidence (see Figure 5(b)). In addition, in audio_id "0003", speaker "Charlie" is considered to exist with 90% confidence, and the state in which "Charlie" has actually been recognized is indicated.

The control unit of the machine learning server 130 updates the tables in Figures 5(a) to (d) when estimating the number of speakers and recognizing the speakers.

(Speaker number estimation and speaker recognition processing)
Next, the speech recognition process according to the embodiment will be described in order with reference to FIGS.
is a transition diagram of the processing of estimating the number of speakers and speaker recognition. In the example shown in FIG. 6, the terminal 100 is a mobile device such as a smartphone, and is equipped with a mobile application 601 having recording and voice recognition functions. The mobile application 601 corresponds to the control unit 105 of the terminal 100. The machine learning server 130 on the cloud 110 side performs different processing for the processing at the time of the first recording of voice on the terminal 100 (step S600) and the processing when at least one person has tagged the recording after the first recording (step S610).

In the process for the initial recording (step S600), the cloud 110 (machine learning server 130) performs learning using an unsupervised learning algorithm on the audio file D sent from the terminal 100 (mobile app 601) (step S601), and then estimates the total number of speakers contained in the audio file D (step S602). When estimating the number of speakers in this step S601, learning using a supervised learning algorithm can improve the accuracy of the speaker number estimation.

Figure 7 shows the display screen on the terminal when recording for the first time. When recording for the first time, the mobile app 601 shows a display screen 700 that is displayed on the display 107 of the terminal 100. When recording for the first time, the mobile app 601 displays the recording start date and time 701, recording duration 702, recording location 703, and title 704 on the display 107.

For example, the mobile app 601 obtains the recording start date and time 701 from a timer in the terminal 100, obtains the recording time 702 from timer measurement, which is the time from the start to the end of recording, obtains the recording location 703 from a positioning unit such as a GPS in the terminal 100, and sets the title 704 by user operation of the terminal 100, etc. During this initial recording, the speaker of the audio file D is not recognized, and the speaker 705 portion is not displayed.

Figure 8 is a diagram showing the recording screen on the terminal when recording. When recording audio, the mobile app 601 displays the recording screen 800 shown in Figure 8 on the display 107 of the terminal 100. The recording screen 800 displays a record/stop button 801, recording time 802, recorded audio (waveform) 803, and a recording transcription display section 804.

The mobile app 601 starts or stops recording each time the user operates the record/stop button 801. It also displays the time since recording started as recording time 802, and displays a waveform 803 corresponding to the audio being recorded. The recording transcription display section 804 displays text characters that have been generated in real time from the audio recorded by the transcription section 103 described above.

Figure 9 shows the display screen on the terminal after the number of speakers has been estimated. It shows the display screen 900 in a state where the mobile app 601 of the terminal 100 has received information on the number of speakers estimated by the cloud 110 (the number-of-speakers estimation unit 131 of the machine learning server 130).

Here, let us assume that the number of speakers included in the audio file D is two, based on speaker estimation on the cloud 110 side. In this case, the mobile app 601 displays "Two speakers estimated" in the speaker number estimation display area 711 showing the estimated number of speakers in the speaker 705 portion, as shown in FIG. 9. In addition, two speaker display areas 712 are displayed corresponding to the two estimated speakers.

The cloud 110 (speaker recognition unit 132 of machine learning server 130) then refers to the trained model and associates specific speakers (names) to identify who each of the two estimated speakers is. After associating the speakers for one of the two estimated speakers, the speaker recognition unit 132 similarly associates the remaining speaker.

Here, if the cloud 110 (speaker recognition unit 132 of the machine learning server 130) determines during speaker recognition that there is a high probability that a specific speaker is present in the audio file D (e.g., a confidence level of 70% or more), it displays the determined speaker candidate in the speaker display area 712 of the terminal 100.

Figure 10 is a diagram showing a display screen on the terminal of a speaker candidate. It shows a state when a notification of one speaker candidate (estimated number of speakers: two) is received from the cloud 110 (speaker recognition unit 132 of machine learning server 130). The mobile app 601 displays "Two speakers have been estimated. Please set a name" in the speaker number estimation display area 711. In addition, in the speaker display area 712 of the display screen 1000, the display of the first speaker's ID "0001" in the first speaker display area 712a is switched to the specific content of the speaker candidate "Are you Alice?". In addition, a confirmation display "Yes/No" is displayed. Also, the symbol 720 is a speaker number addition button for adding the number of speakers by a user operation.

In this way, the cloud 110 (the speaker recognition unit 132 of the machine learning server 130) specifically tags the speaker candidate and specifically displays the speaker candidate (name) "Alice" in the speaker display area 712a. This allows the user operating the terminal 100 to confirm whether the speaker candidate is correct or not. Then, when the user determines that the displayed speaker candidate is correct and operates the confirmation display "Yes," the mobile app 601 transmits information of the confirmation operation to the cloud 110 (the speaker recognition unit 132 of the machine learning server 130), and the cloud 110 (the speaker recognition unit 132 of the machine learning server 130) recognizes that the speaker of ID "0001" is "Alice" and that the speaker "Alice" exists in the audio file D.

This tagging process corresponds to step S610 in FIG. 6. In step S610, after estimating the speaker in the process at the time of initial recording (step S600), if a speaker is present in the audio file D with a certain degree of reliability or higher, the cloud 110 (speaker recognition unit 132 of the machine learning server 130) performs learning on this speaker using a supervised learning algorithm (step S611), and then performs processing to recommend (confirm) the speaker candidates to the user of the terminal 100 (step S612).

In FIG. 10, the second speaker display area 712b of the speaker display area 712 on the display screen 1100 displays the speaker ID "0002", indicating that only a second speaker is present. If a different speaker exists in this second speaker display area 712b with a certain degree of reliability or higher, the cloud 110 (the speaker recognition unit 132 of the machine learning server 130) will recommend speaker candidates to the user, just as in the case of the first speaker.

FIG. 11 is a diagram showing a display screen showing a list of speaker selections on a terminal. In the description of FIG. 10, the cloud 110 (the speaker recognition unit 132 of the machine learning server 130) notifies the terminal 100 of the speaker candidates, but may also transmit information of a list of speaker candidates included in the audio file D and having a predetermined reliability to the terminal 100. In this case, the mobile app 601 displays a display screen 1100 of the speaker selection list. The user can check this list display screen 1100 and select a speaker from among multiple speaker candidates. In addition, if the speaker candidates displayed in the speaker display area 712a of FIG. 10 are different, the user can select "No" to cause the mobile app 601 to display the display screen 1100, and the user can select other speaker candidates "Bob" and "Charlie".

As described above, in the embodiment, speaker recognition is not performed solely based on the judgment of the cloud 110 (the speaker recognition unit 132 of the machine learning server 130), but speaker candidates are first sent to the terminal 100 and then a process is performed in which the user confirms them, thereby making it possible to improve the accuracy of speaker recognition with a simple operation.

(Transcription function)
Here, a description will be given of the function of the transcription unit 103 of the terminal 100. The transcription unit 103 generates text characters corresponding to the speech while recording or playing back the speech.

Figure 12 is a diagram showing the display screen of the transcription on the terminal. This figure shows the state when audio file D is being played back. The control unit (mobile app 601) of the terminal 100 displays a transcription display screen 1200 on the display 107 when the transcription unit 103 is functioning. On the display screen 1200, the mobile app 601 displays recording information (title, recording date and time, recording duration, recording location) 1201, transcription content 1202, a playback position display bar 1203, and playback operation (play/stop, rewind, memo operation) buttons 1204.

The transcription unit 103 records the generated text characters in conjunction with the timing of speech, and displays the "confidence" of the transcription accuracy in an identifiable manner. For example, in the text characters displayed in the transcription content 1202, characters with less confidence are displayed lighter than the standard darkness (area 1210), and characters with more confidence than the standard are displayed darker (area 1211). This allows the user to easily grasp the conversion accuracy of each character from the darkness of the text displayed in the transcription content 1202. In addition, the playback position on the characters is highlighted (area 1221) so that the playback position can be identified.

In this way, the transcribed text can be displayed in different formats according to the estimated confidence of the transcription (corresponding to the accuracy of the transcription), which can improve the convenience of playing back the recording or searching the text after it has been recorded.

The speaker recognition information described above can also be effectively used for this transcription function. For example, the speaker recognition results for the original transcribed audio file D can be displayed on the display screen 1200. In the illustrated example, the speaker 1230 information "Alice" after speaker recognition for audio file D is displayed as part of the recording information 1201. This makes it possible to notify the user of the speaker of audio file D itself.

(Display when playing an audio file)
13 is a diagram showing a display screen on the terminal when playing an audio file. A large number of audio files D are stored in the terminal 100, and the control unit (mobile application 601) displays a screen on the display screen 1300 to make it easier to find a desired audio file D when playing an audio file. For example, as shown in FIG. 13, a calendar 1301 is displayed during playback on the terminal 100. On the calendar 1301, the recording date portion assigned to the recorded audio file D is displayed in an identifiable manner (in the illustrated example, the recording date is displayed as a circle). This allows the user to easily play a desired audio file D on the calendar 1301 based on the recording date.

Although not shown, by selecting a recording date on the calendar 1301, the above-mentioned speaker recognition information, i.e., the recorded speaker "Alice" etc., may be displayed as information on the recorded audio file D in a pop-up or the like. This makes it easier to search for the required audio file D.

(Example of speaker count estimation and speaker recognition correction)
Next, an example of estimating the number of speakers and correcting speaker recognition will be described with reference to Figs. 14 to 18. In the embodiment, a voice activity detection (VAD: Voice.Act) is used as a mechanism for distinguishing between a voice activity segment that is the voice of a person to be recognized and a non-voice activity segment that is noise, which are included in the voice file D.
The device uses Integrity Detection technology.

Figure 14 is a diagram showing an example of the waveform of the sound contained in the sound file. If the cloud 110 (machine learning server 130) estimates that the sound file D shown in Figure 14 is the sound registered as a single speaker, it will determine that the four VAD-extracted sound sources S1 to S4 are all the voices of the same person, and will use them for future learning.

Figure 15 is a diagram showing grouping of speakers included in an audio file. For convenience, the audio waveforms S1 to S4 shown in Figure 15 are the same as those in Figure 14. Here, assume that the cloud 110 (machine learning server 130) estimates that there are two speakers for audio file D and labels one of them as a speaker candidate with the name "Alice". In this case, the cloud 110 (machine learning server 130) performs clustering processing on the four audio sources S1 to S4 obtained by VAD audio source extraction, thereby dividing the audio sources S1 to S4 into two groups.

In the example of Figure 15, group 1 (G1) has sound sources S1 and S4, and group 2 (G2) has sound sources S2 and S3. This means that there is a 50% probability that the speaker in group G1 is the voice of "Alice." If it is recognized that there are four people, then the probability that the voice of "Alice" is in all four groups is 25%.

Cloud 110 (machine learning server 130) learns the sound source of the training data that is weighted by reliability during training, such as a voice that is 100% "Alice" and a voice that is 50% "Bob", and uses this trained model to determine whether there is an approximate voice for the sound source of a new audio file D. During training, for example, for a sound source with a reliability of 50%, if there are other trained models, cloud 110 (machine learning server 130) can obtain a reliability of 50% or more by performing an approximation determination from the stage of selecting the training data and acquiring it.

Figure 16 is a diagram showing changes in the estimated number of speakers. For convenience, the audio waveforms S1 to S4 shown in Figure 16(a) are the same as those in Figure 15. Assume that the cloud 110 (machine learning server 130) labels one speaker candidate "Alice" through the process shown in Figure 15, and recognizes group 1 (G1) as the sound sources S1 and S4 of speaker candidate "Alice."

Here, if the number of speakers included in the audio file D is presented to the user of the terminal 100 as two, and then the user operates to correct this to three, the cloud 110 (machine learning server 130) performs clustering of the VAD-extracted audio sources again to estimate the number of speakers, adapting them to three people. In the illustrated example, the cloud 110 (machine learning server 130) performs clustering of the VAD-extracted audio sources for the audio file D, adapting them to three people, and thereby changing the audio source S4 of group 1 (G1) shown in FIG. 16(a) to group 3 (S3) of the third person, as shown in FIG. 16(b).

Figure 17 shows changes to the estimated number of speakers. For convenience, the audio waveforms S1 to S4 shown in Figure 17(a) are the same as those in Figure 15. Assume that the cloud 110 (machine learning server 130) labels one speaker candidate "Alice" through the process shown in Figure 15, and recognizes group 1 (G1) as the sound sources S1 and S4 of speaker candidate "Alice."

Here, if the number of speakers included in the audio file D is presented to the user of the terminal 100 as two, and then the user operates to correct the number to one, the cloud 110 (machine learning server 130) performs clustering of the VAD-extracted audio source again to estimate the number of speakers, adapting it to one person. In the illustrated example, the cloud 110 (machine learning server 130) learns that all of the audio files D are audio sources for one person, and thereby creates a group 2 (G
2) The sound sources S2 and S3, and the sound sources S1 to S4 of the audio file D as shown in FIG. 17(b) are all changed to the same group 1 (G1).

Figure 18 is a diagram showing a display screen for adding speaker candidates on a terminal. As described above with reference to Figure 10, this shows a display screen 1800 when a speaker is added by the user after a notification of one speaker candidate (estimated number of speakers: two) has been received from the cloud 110 (speaker recognition unit 132 of the machine learning server 130).

In FIG. 10, the terminal 100 displays two speaker display areas 712a and 712b on the display screen 1800 to the user in response to an estimated number of speakers of two. If the user subsequently corrects the number of speakers included in the audio file D to three, the cloud 110 (machine learning server 130) will additionally display a third speaker display area 712c by operating the add speaker number button 720 shown in FIG. 10.

In the display example of FIG. 18, the third speaker display area 712c displays only the third speaker's ID "0003". The second speaker display area 712b displays the ID "0002", but if a specific speaker candidate "Bob" can be presented, it is displayed together with a confirmation message "Yes/No".

In this way, the cloud 110 (machine learning server 130) presents speaker candidates to the user in the number of estimated speakers, and based on the user's operation to modify the number of speakers, it becomes possible to accurately and efficiently estimate the number of speakers contained in the audio file D and subsequently recognize the speakers.

Furthermore, the voice recognition processing according to the embodiment can be effectively used to protect the rights of individuals. For example, it is useful for individuals who are in a weak position in contracts, and in the current situation where contracts have not yet taken root as a business practice, such as for artists such as actors and individuals working freelance, it can deal with problems such as verbal promises, "said/not said (something to each other)" problems, and broken promises when verbal promises are pretended to be forgotten. By recording conversations using the voice recognition processing described in this embodiment, it becomes possible to automatically generate simple memoranda and contracts. As for the format of contracts, many of them are basically patterned so that they can be generated by answering multiple questions (e.g. 20 questions), and can be handled by simple natural language processing techniques such as the transcription described above.

In many cases, there is strong psychological resistance to even confirming the contract with the other party, so audio recordings are important for recording the promises or facts. It is desirable to guarantee that audio recordings cannot be deleted or tampered with. For example, even if an abused child manages to record the scene of the abuse, it is meaningless if the audio is discovered and intentionally deleted.

In response to this, in the above embodiment, the audio file recorded by the terminal is not limited to being recorded/saved only on the terminal, but the recorded audio file may be uploaded to the cloud in real time and saved on the cloud side. Note that in order to maintain security and prevent tampering with the audio file, the audio file may be distributed and saved on the cloud side, or the hash value of the audio file may be recorded on the blockchain, etc.

According to the above-described embodiment, the speech recognition device estimates the number of speakers included in the speech file, refers to a trained model for each speaker prepared in advance, recognizes each speaker of the estimated number of speakers, and tags the speakers included in the speech file. This makes it possible to specifically present the number of speakers and the speakers included in the speech file to the user. Therefore, it becomes possible to easily find a desired speech from among a large number of speech files.

The recognition process presents the user with information on the estimated number of speakers and speaker candidates corresponding to the number of speakers, and recognizes each speaker based on the user's operation to identify a speaker from the speaker candidates, and the tagging process tags the speaker based on the user's operation. In this way, by simply adding a simple operation that allows the user to identify the speaker candidates estimated by the device, speakers can be identified with greater accuracy.

The estimation process also presents the estimated number of speakers to the user, and re-estimates the number of speakers included in the audio file based on the user's operation to change the number of speakers. This makes it possible to accurately estimate the number of speakers included in the audio file with simple user operations.

Furthermore, learning is performed to learn and accumulate speaker information after tagging, and the recognition process recognizes each speaker of the estimated number of speakers based on the trained model. This makes it possible to improve the recognition accuracy of speakers contained in audio files by repeating the learning process.

The voice recognition process may be performed by the terminal alone, or may be shared by a system using the terminal and the cloud. In the case of a system configuration, in a voice recognition system in which a terminal and a cloud are connected for communication, the terminal has a voice recording unit and a communication unit that uploads recorded or played voice files to the cloud, and the cloud estimates the number of speakers who made the voices included in the voice files, refers to a trained model for each speaker prepared in advance, recognizes each speaker of the estimated number of speakers, and notifies the terminal of information tagged with the speakers included in the voice files. In this way, by processing the voice recognition for the voice files, i.e., the processing for estimating the number of speakers and recognizing the speakers, on the cloud side, it is possible to improve the accuracy of the voice recognition while reducing the processing burden on the terminal side. In addition, it is possible to collectively process the voice recognition of multiple terminals on the cloud side. In addition, it is also possible to configure the terminal to collectively upload multiple voice files accumulated while offline between the terminal and the cloud to the cloud when online, and the cloud to collectively voice recognize the multiple files.

The control unit of the terminal also generates the text contained in the audio file in real time when recording or playing back the audio. This makes it possible to present the content of the audio file specifically to the user. In addition, the number of speakers and speaker information described above can be presented to the user, making it easier to find the desired audio file.

The cloud also has a storage unit that stores audio files uploaded from the terminal. This allows the audio files to be protected by externally storing them as described above, without having to be stored only on the terminal, and improves the effectiveness of the audio files.

As described above, in the embodiment, the audio files that are the subject of voice recognition can be recognized not only during recording, but also during playback. Therefore, it is possible to easily estimate the number of speakers and recognize the speakers without the need for complicated tasks such as presetting the number of speakers before recording, preparing microphones for each speaker, and detecting the direction of each speaker. Furthermore, according to the embodiment, even if a large number of audio files are created due to repeated recording, the number of speakers and the speaker can be added to the search for audio files, making it easy to find the desired audio file.

The program for speech recognition described in the present embodiment can be realized by executing a prepared program on a computer. This program is recorded on a computer-readable recording medium such as a semiconductor memory, a hard disk, a flexible disk, a CD-ROM, or a DVD, and is executed by being read from the recording medium by the computer. This program may also be distributed via a network such as the Internet.

As described above, the present invention is useful for application to devices that recognize voice, including IC recorders that record and play back, and smartphones equipped with recording apps.

REFERENCE SIGNS LIST 100 Terminal 101 Microphone 102 Recording unit 103 Transcription unit 104 Speaker tagging unit 105 Control unit 106 Keyboard 107 Display 110 Cloud 120 Storage server 130 Machine learning server 131 Speaker number estimation unit 132 Speaker recognition unit 140 Trained model DB
201 CPU
202 ROM
203 RAM
204 External memory 211 Communication I/F
601 Mobile application 711 Speaker number estimation display area 712 Speaker display area D Audio file

Claims (7)

On the computer,
Estimate the number of speakers for each speaker included in the audio file;
Refer to a pre-prepared trained model for each speaker to recognize each speaker of the estimated number of speakers;
tagging speakers contained in said audio files ;
The estimation process includes:
presenting the estimated number of speakers to a user;
re-estimating the number of speakers included in the audio file based on a change operation of the number of speakers by the user.
A speech recognition program for executing a process. Further, learning and storing the tagged speaker information;
The recognition process includes:
Recognizing each speaker of the estimated number of speakers based on the trained model.
2. The speech recognition program according to claim 1 , 3. The speech recognition program according to claim 1, further comprising: generating characters contained in the voice file in real time when the voice file is being recorded or reproduced . The computer
Estimate the number of speakers for each speaker included in the audio file,
Refer to a pre-prepared trained model for each speaker to recognize each speaker of the estimated number of speakers;
tagging speakers contained in said audio files;
The estimation process includes:
presenting the estimated number of speakers to a user;
re-estimating the number of speakers included in the audio file based on a change operation of the number of speakers by the user.
23. A speech recognition method comprising: A control unit for recognizing the number and speakers included in the audio file,
The control unit is
Estimate the number of speakers for each speaker included in the audio file;
Refer to a pre-prepared trained model for each speaker to recognize each speaker of the estimated number of speakers;
tagging speakers contained in said audio files;
The control unit, as the estimation process,
presenting the estimated number of speakers to a user;
re-estimating the number of speakers included in the audio file based on a change operation of the number of speakers by the user.
1. A speech recognition device comprising: In a voice recognition system in which a terminal is connected to the cloud,
The terminal includes:
A voice recording section;
A communication unit that uploads the recorded or played audio file to the cloud,
The cloud includes:
Estimating the number of speakers for each speaker included in the audio file;
Refer to a pre-prepared trained model for each speaker to recognize each speaker of the estimated number of speakers;
notifying the terminal of speaker tagged information included in the audio file;
The terminal includes:
presenting the number of speakers estimated by the cloud to a user via a display unit;
The cloud includes:
re-estimating the number of speakers included in the audio file based on the operation of changing the number of speakers by the user received from the terminal, and transmitting the result to the terminal.
A speech recognition system comprising: The cloud includes:
The voice recognition system according to claim 6, further comprising a storage unit for storing the voice file uploaded from the terminal.

Images