JP7664549B2 - Speech classification device and method - Google Patents

Description

This disclosure relates to a speech classification device and a speech classification method.

Patent Document 1 discloses a minutes output device that outputs minutes in which the speaker in a meeting is identified with high accuracy. This minutes output device acquires information related to the number of participants in a meeting and data related to the audio in the meeting, recognizes audio based on the audio data, and converts it into text as the speaker's remarks. The minutes output device identifies the speaker based on the information related to the number of participants and the audio data, and outputs minutes in which a label indicating the speaker is associated with the content of the converted remarks.

JP 2020-95210 A

In the configuration of Patent Document 1, information regarding the number of participants in a meeting is provided as prerequisite information for identifying speakers. For this reason, if information regarding the number of speakers in a meeting is not provided, it is assumed that the minutes output device of Patent Document 1 would have difficulty estimating how many people were speaking during the meeting. For example, when there is audio data for which minutes of a meeting are to be created, the longer the time that has passed since the audio data was recorded, the more fuzzy the memory of the number of speakers becomes, and furthermore, it is often impossible to accurately determine the number of speakers who actually spoke among the participants. For this reason, it can be said that there is room for improvement in terms of automatically classifying the number of participants in a meeting without having to input the number of participants in advance.

The present disclosure has been devised in light of the conventional situation described above, and aims to provide an utterance classification device and an utterance classification method that accurately classifies utterances while estimating the number of speakers participating in a conference, regardless of whether information regarding the number of speakers in the conference is input in advance.

The present disclosure provides a speech classification device including an input unit that inputs an audio file including speech from multiple speakers, an utterance extraction unit that analyzes the audio file and extracts the number of utterances and the length of each utterance included in the audio file, a parameter determination unit that determines multiple parameters for grouping the multiple speakers based on at least one of the number of utterances and the length of each utterance, a feature extraction unit that extracts features of each utterance within multiple speech sections included in the audio file based on the parameters, and a classification unit that groups the utterances by speaker based on the features of each utterance.

The present disclosure also provides a speech classification method executed by a speech classification device that is a computer, the speech classification method comprising the steps of: inputting an audio file including speeches of multiple speakers; analyzing the audio file and extracting the number of utterances included in the audio file and the utterance length of each utterance; determining a plurality of parameters for grouping the multiple speakers based on at least one of the number of utterances and the utterance length of each utterance; extracting, based on the parameters, features of each utterance in a plurality of speech sections included in the audio file; and grouping the utterances by speaker based on the features of each utterance.

According to the present disclosure, it is possible to accurately classify utterances while estimating the number of speakers participating in a meeting, regardless of whether information regarding the number of speakers in the meeting is input in advance.

FIG. 1 is a diagram showing an example of a system configuration of a speaker classification system according to a first embodiment; Block diagram showing an example of the hardware configuration of a PC FIG. 1 is a diagram showing an example of a procedure of an overall operation of a minutes generating device; A schematic diagram showing an example of the output of minutes FIG. 13 is a diagram showing an example of parameters determined according to conditions; A flowchart showing an example of an operation procedure of the minutes generating device according to the first embodiment. A diagram showing an example of a speech waveform based on continuous speech by one speaker. FIG. 2 is a diagram illustrating a first example of a segmentation of an utterance section used for extracting features; FIG. 13 is a diagram illustrating a second example of a segmentation of an utterance section used for extracting features; FIG. 8 shows an example of a speech waveform based on the continuous speech of one speaker shown in FIG. 7 followed by speech of another speaker. A flowchart showing an example of an operation procedure of the minutes generating device according to the second embodiment. A diagram showing an example of a feature file used for feature extraction. A diagram showing an example of the procedure for clustering operations to estimate the number of speakers included in a feature file Graph showing an example of clustering results without data division A table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering. Graph showing the behavior of the similarity between each utterance and each cluster based on the calculation results of FIG. 15. A flowchart showing an example of a basic operation procedure of the minutes generating device according to the third embodiment in the case where data division processing is not performed. A flowchart showing an example of the operation procedure of a subroutine of the cluster synthesis process shown in FIG. Graph showing examples of clustering process results when data division process is performed 20 is a graph showing an example of a combination of the results of the clustering processes shown in FIG. 19 . A table showing an example of calculating the similarity between clusters A graph showing the behavior of the similarity between each utterance and each cluster obtained by clustering A table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering. A flowchart showing an example of a basic operation procedure of the minutes generating device according to the third embodiment in the case where data division processing is performed

Below, with reference to the drawings as appropriate, an embodiment that specifically discloses an utterance classification device and an utterance classification method according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters and duplicate explanation of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following explanation are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

In each of the following embodiments, a conference is used as an example of an event in which speech from multiple speakers is expected, and the speech classification device according to the present disclosure uses audio files in which speech from multiple speakers during a conference is recorded, groups each speech by speaker (hereinafter referred to as "clustering"), and outputs minutes obtained by performing voice recognition on the speech of each grouped speaker.

(Embodiment 1)
1 is a diagram showing an example of a system configuration of a speaker classification system 50 according to the first embodiment. The speaker classification system 50 is used, for example, during a conference in which a plurality of speakers participate, and includes a sound collection device MIC, a PC (Personal Computer) 10, and a minutes-of-meeting creating device 20. The sound collection device MIC and the PC 10 are both placed in a conference room and used, and are connected, for example, by wire so that data can be input and output between them. The PC 10 and the minutes-of-meeting creating device 20 are connected via a network NW1 so that data can be transmitted and received. Note that the network NW1 may be omitted, in which case the PC 10 and the minutes-of-meeting creating device 20 are directly connected.

The network NW1 may be a wired network or a wireless network. The wired network may be, for example, a wired LAN (Local Area Network) such as Ethernet (registered trademark), or may be of the USB (Universal Serial Bus) standard. On the other hand, the wireless network may be, for example, a wireless LAN such as Wi-fi (registered trademark), Bluetooth (registered trademark), or WiGig using a high frequency such as the 60 GHz band, and the type is not particularly limited.

There are two possible use cases for the speaker classification system, for example. Note that the use cases are not limited to these two.

In the first use case U1 shown in FIG. 1, for example, four speakers A, B, C, and D participate in a conference, and each speaker speaks toward the sound collection device MIC. The sound collection device MIC is equipped with one or more sound collection devices such as microphones, collects the voices uttered by each speaker during the conference, and sends the collected voice data to the PC 10. The PC 10 uses the voice data sent from the sound collection device MIC to generate a voice file containing the voice data of the speech content of the speaker during the conference, and sends it to the minutes generation device 20 via the network NW1. The minutes generation device 20 receives the voice file from the PC 10, analyzes the voice file (see below for details), and clusters (groups) each utterance included in the voice file by speaker. The minutes generation device 20 recognizes the voice data of each clustered utterance, converts it into text, and outputs the text (see FIG. 4). A detailed configuration example of the PC 10 will be described with reference to FIG. 2 below.

In the second use case U2 shown in FIG. 1, for example, two speakers E and F each participate in a conference, and speaker E speaks toward PC 30, and speaker F speaks toward PC 40. As will be described later with reference to FIG. 2, PCs 30 and 40 are equipped with one or more microphones or other sound collection devices 17, and collect the voices uttered by the corresponding speakers E and F during the conference. Using the collected voice data, the PCs 30 and 40 generate voice files each containing the voice data of the contents of the speeches by speakers E and F during the conference, and send the voice files to the minutes-generating device 20 via the network NW2. The minutes-generating device 20 receives the voice files from PCs 30 and 40, synthesizes both voice files, analyzes the synthesized voice file (see below for details), and clusters (groups) each utterance included in the voice file by speaker. The minutes-generating device 20 recognizes the voice data of each clustered utterance, converts it into text, and outputs the text (see FIG. 4). Like network NW1, network NW2 may be a wired network (see above) or a wireless network (see above). Network NW2 may be omitted, in which case PCs 30 and 40 are directly connected to the minutes generating device 20. A detailed configuration example of PCs 30 and 40 will be described later with reference to FIG. 2.

The minutes-creating device 20 is connected to PC 10 via network NW1 so that data can be sent and received, and is further connected to PCs 30 and 40 via network NW2 so that data can be sent and received, and is configured using a server computer such as a cloud server with high-performance specifications. Note that the minutes-creating device 20 may be configured using a PC, may be realized by one computer, or may be realized by multiple different computers. Also, the minutes-creating device 20 may be realized by each or any one of PCs 10, 30, and 40.

The minutes generating device 20, which is an example of a speech classification device according to the present disclosure, includes a communication unit 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage 24, and a processor 25.

The communication unit 21 is configured using a communication circuit capable of transmitting and receiving data to and from external devices. The communication unit 21 is connected, for example, via network NW1 to enable data transmission and reception to and from PC 10, and is further connected via network NW2 to enable data transmission and reception to and from PCs 30 and 40. The communication unit 21 receives audio file data from PC 10 via network NW1 and sends it to the processor 25. Similarly, the communication unit 21 receives audio file data from PCs 30 and 40 via network NW2 and sends it to the processor 25.

The ROM 22 stores programs and data necessary for the processor 25 to execute operations (processing). In other words, the processor 25 executes processes defined in the programs using the programs and data stored in the ROM 22, thereby enabling the minutes generating device 20, which is a computer, to execute various processes related to the speech classification method of the present disclosure.

RAM 23 temporarily stores data or information generated or acquired during operation (processing) of processor 25.

The storage 24 is configured using a storage medium such as a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), and stores data or information generated by the processor 25.

The processor 25 is configured using, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit) or an FPGA (Field Programmable Gate Array). The processor 25 functions as a controller that manages the overall operation of the minutes generating device 20, and performs control processing for managing the operation of each part of the minutes generating device 20, data input/output processing between each part of the minutes generating device 20, data arithmetic processing, and data storage processing. The processor 25 can operate functionally as a voice input unit 251, a feature extraction unit 252, a voice clustering unit 253, and a minutes generating output unit 254 according to the programs and data stored in the ROM 22. The processor 25 uses the RAM 23 during operation, and temporarily stores data or information generated or acquired by the processor 25 in the RAM 23.

The audio input unit 251, which is an example of an input unit, acquires and inputs audio file data from the PC 10 and audio file data from the PCs 30 and 40 via the communication unit 21.

The feature extraction unit 252, which is an example of a speech extraction unit, analyzes the audio file (e.g., a data file composed of audio data of the contents of speech by multiple speakers during a conference) input by the audio input unit 251 to detect speech sections, and extracts the total number of utterances and the total speaking time of the audio data included in the audio file. For example, a sound pressure threshold (e.g., a sound pressure value) used to detect the speech section may be stored in advance in the ROM 22 or the RAM 23, and the feature extraction unit 252 may detect a section in which a sound pressure equal to or greater than the sound pressure threshold is detected as the speech section. Note that the method of detecting the speech section by the feature extraction unit 252 is not limited to the above-mentioned method, and may be realized by, for example, the feature extraction unit 252 reading out from the ROM 22 or the storage 24 a trained model of AI (artificial intelligence) (e.g., a trained model that performs VAD (Voice Active Detector) processing) that has already been generated based on the execution of a learning process, and executing it.

The feature extraction unit 252, which is an example of a division unit, determines the number of divisions (an example of a parameter, see FIG. 5) of the audio data included in the audio file based on the extraction results of the total number of utterances and the total total speaking time, and divides the audio data included in the audio file according to the determined number of divisions.

The feature extraction unit 252 also extracts features of the voice data generated by the speakers participating in the conference for each speech section of the voice data detected for the entire voice file, or for each speech section of the divided voice data. Here, the features of the voice data are indicated by a string of numbers, and are, for example, the Mel-Frequency Cepstrum Coefficient (MFCC), formant frequency, the frequency spectrum of the voice (for example, the amplitude spectrum or phase spectrum), and the voiceprint value shown in the spectrogram. Furthermore, the features of the voice data need not be limited to those described above.

The feature extraction unit 252, which is an example of a parameter determination unit, determines parameters (see FIG. 5) for clustering performed by the voice clustering unit 253 using at least one extraction result of the total number of utterances and the total total speech time of the voice data detected for the entire voice file, or at least one extraction result of the total number of utterances and the total total speech time of the divided voice data, and the table of FIG. 5. Note that, if the feature extraction unit 252 cannot extract (including not extracting) at least one of the total number of utterances and the total total speech time of the voice data detected for the entire voice file, it may use a fixed parameter value corresponding to at least one of the total number of utterances and the total total speech time that cannot be extracted.

Here, the parameters will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of parameters determined according to conditions. The table shown in FIG. 5 is stored, for example, in storage 24, and specifies the number of data divisions, the maximum number of classifications, and the number of data N additions that are determined for each condition (specifically, the total speech length and the number of speech sections). The number of data divisions, the maximum number of classifications, and the number of data N additions are each an example of a parameter. Note that the parameters are not limited to these, and may also include, for example, a lower limit value for the time of data (voice data) required for the learning process of the learned model used in clustering, an initial value for the classification of clustering, etc.

The total speech length is the total duration of the speaking parts contained in the audio file, and is different from the duration of the audio data contained in the audio file. The number of speech sections is the number of speaking parts (speech sections) contained in the audio file.

The number of data divisions is determined based on the total length of the total speech length of the audio files input to the audio input unit 251, with the aim of improving the accuracy of the clustering process by the audio clustering unit 253, and indicates a value for dividing the audio data contained in the audio files into time lengths suitable for clustering.

The maximum number of categories indicates the maximum number of categories resulting from clustering performed by the voice clustering unit 253, with the aim of suppressing an increase in the clustering processing load on the voice clustering unit 253.

The number of data N additions indicates the value of N when the number of pieces of data required for the learning process of the trained model of AI (artificial intelligence) performed before clustering by the voice clustering unit 253 can be prepared by multiplying the number of pieces of voice data in the speech section by N.

For example, if the extraction results of the total speech time and the number of speech sections included in the audio file input to the audio input unit 251 are "1 minute, 10", the feature extraction unit 252 determines the parameters as "none" for the number of data divisions, "2" for the maximum number of classifications, and "50" for the number of data N additions. In other words, if a short audio file with a total time of 1 minute is input, the feature extraction unit 252 calculates the features of the speech sections of the audio data included in the audio file without dividing the data, and performs a learning process by increasing the number of audio data included in the audio file by 50 times before clustering, and then uses the trained model obtained by the learning process to perform clustering so that each utterance included in the audio file corresponds to at most two speakers.

For example, if the extraction results of the total speech time and the number of speech sections included in the audio file input to the audio input unit 251 are "10 minutes, 100", the feature extraction unit 252 determines the parameters as "none" for the number of data divisions, "10" for the maximum number of classifications, and "10" for the number of data N additions. In other words, if an audio file with a total time of 10 minutes is input, the feature extraction unit 252 calculates the features of the speech sections of the audio data included in the audio file without dividing the data, and performs a learning process by increasing the number of audio data included in the audio file by 10 times before clustering, and then uses the trained model obtained by the learning process to perform clustering so that each utterance included in the audio file corresponds to a maximum of 10 speakers.

For example, if the extraction results of the total speech time and the number of speech sections included in the audio file input to the audio input unit 251 are "100 minutes, 1000", the feature extraction unit 252 determines the number of data divisions to be "10", the maximum number of classifications to be "2", and the number of data N additions to be "50" as parameters. In other words, if a long audio file with a total time of 100 minutes is input, the feature extraction unit 252 divides the audio data included in the audio file into 10 parts, calculates the features of the speech sections of the audio data divided into 10 parts, and performs a learning process without increasing the number of audio data as learning data included in the audio file before clustering, and then clusters the utterances included in the audio file to correspond to a maximum of 20 speakers using the trained model obtained by the learning process.

The voice clustering unit 253, which is an example of a classification unit, uses the output of the feature extraction unit 252 (specifically, the features of the utterances in the speech section, the parameters described with reference to FIG. 5) to group (cluster) each utterance of the detected voice data for the entire voice file or each utterance of the divided voice data by speaker. For example, the voice clustering unit 253 classifies utterances with similar features (in other words, utterances whose difference between the features is less than a predetermined value registered in ROM 22) into the same group (hereinafter also referred to as a "cluster"). In addition, the voice clustering unit 253 may calculate the similarity between the clusters, and combine both clusters if the calculation results are similar (in other words, if the difference in similarity is less than a predetermined value registered in ROM 22).

The minutes generation output unit 254, an example of an output processing unit, performs speech recognition on the voice data of each utterance by each speaker grouped (clustered) by the voice clustering unit 253, converts it into text, and generates the converted text (e.g., words spoken during the meeting) as minutes (see FIG. 4). The minutes generation output unit 254 displays the generated minutes data on PC 10 via network NW1, or on PCs 30 and 40 via network NW2.

Figure 4 is a diagram showing a schematic example of an output of minutes. As shown in Figure 4, the display 18 (see Figure 2) of the PCs 10, 30, and 40 displays the minutes data sent from the minutes generating device 20 in a manner that allows the speaker to be identified. For example, utterance RT1 of "Let's start the meeting. First, please tell me the progress of project Pa." utterance RT2 of "Yes, I understand. This week, the progress of Pa is..." utterance RT3 of "Thank you. Next, please tell me the progress of project Pb." utterance RT4 of "Yes, I understand. This week, the progress of Pb is..." are displayed. In particular, since the voice clustering unit 253 has identified that the speakers of utterances RT1 and RT3 are the same, utterances RT1 and RT3 are displayed in a manner that allows the speaker to be identified as being the same (for example, in the same color).

In the minutes-taking device 20 according to the first embodiment, it is not necessary for information regarding the number of speakers to be registered in advance as in Patent Document 1. Information regarding the number of speakers may be registered, but the minutes-taking device 20 according to the first embodiment can perform clustering (in other words, classifying the voice data uttered by each speaker) using the voice data of each utterance by each speaker by analyzing the input voice file, regardless of whether information regarding the number of speakers is registered.

Figure 2 is a block diagram showing an example of the hardware configuration of PCs 10, 30, and 40. PCs 10, 30, and 40 include an interface 11, an imaging unit 12, a processor 13, a storage 14, a communication device 15, an operation device 16, a sound collection device 17, a display 18, and an external interface 19. Since PCs 10, 30, and 40 have the same configuration, the following explanation of Figure 2 will use PC 10 as an example, but the same explanation can also be applied to PCs 30 and 40.

The interface 11 connects each component of the PC 10 so that data can be input and output. In Figure 2, the interface is abbreviated as IF.

The imaging unit 12 includes at least a lens and an image sensor. The lens includes, for example, a focus lens and a zoom lens, and forms an optical image on the light receiving surface (in other words, the imaging surface) of the image sensor by receiving light from at least one speaker in front of the PC 10. The image sensor performs photoelectric conversion to convert the light received on the light receiving surface (in other words, the imaging surface) into an electrical signal. The image sensor is configured using, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor sends an electrical signal (analog signal) corresponding to the light received on the light receiving surface (in other words, the imaging surface) to the processor 13 at predetermined time intervals. This analog signal is converted into digital captured image data by the processor 13. As a result, the captured image data is generated by the processor 13. The conversion from the analog signal to digital format may be performed by the image sensor. In this way, the PC 10 can acquire digitally formatted captured image data at predetermined intervals. Note that the imaging unit 12 may be omitted from the configuration of the PC 10.

Processor 13 is configured using, for example, a CPU, DSP, GPU or FPGA. Processor 13 functions as a controller that manages the overall operation of PC 10, and performs control processing to integrate the operation of each part of PC 10, data input/output processing between each part of PC 10, data arithmetic processing, and data storage processing. Processor 13 operates according to the programs and data stored in storage 14. Processor 13 uses storage 14 during operation, and temporarily stores data or information generated or acquired by processor 13 in storage 14.

Storage 14 includes, for example, a ROM, and stores in the ROM the programs and data required for the processor 13 to execute operations (processing). In other words, the processor 13 can execute various processes related to the speech classification method of the present disclosure on the computer PC 10 by using the programs and data stored in storage 14 to execute processes defined in the programs. Storage 14 also includes, for example, a RAM, and temporarily holds in the RAM data or information generated or acquired during the operation (processing) of processor 13. Storage 14 also includes a storage medium, such as a flash memory, HDD, or SSD, and stores in the storage medium data or information generated by processor 13.

The communication device 15 is configured using a communication circuit capable of transmitting and receiving data to and from an external device. The communication device 15 is connected, for example, via a network NW1, so as to be capable of transmitting and receiving data to and from the minutes generation device 20. The communication device 15 receives minutes data from the minutes generation device 20 via the network NW1 and sends it to the processor 13.

The operation device 16 is a device used by the speaker during operation, and may be, for example, a mouse, a keyboard, a touchpad, or a touch panel, or a combination of multiple of these. A signal from the operation device 16 based on the speaker's operation is input to the processor 13 via the interface 11.

The sound collection device 17 is, for example, composed of one or more microphones, and collects the voices of the speakers in front of or around the PC 10 during the conference, and sends the collected voice data to the processor 13.

The display 18 is configured, for example, using an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) device, and outputs (for example, displays) data of the speaker's face image captured by the imaging unit 12 and data of the minutes from the minutes generating device 20.

The external interface 19 inputs and outputs data to and from an external medium (e.g., an external storage medium such as an SD card) via a cable conforming to the USB standard. The external interface 19 may also be connected to the minutes generating device 20.

Figure 3 is a diagram showing an example of the overall operation procedure of the minutes-generating device 20. Each process shown in Figure 3 is a schematic and chronological diagram of the operation procedure of the minutes-generating device 20, and is mainly executed by the processor 25 of the minutes-generating device 20.

In FIG. 3, when processor 25 receives an audio file from PC 10, for example, it analyzes (e.g., VAD processing) the audio file (e.g., a data file composed of audio data of speech contents of multiple speakers during a conference) in feature extraction unit 252 to detect speech sections (Step 1). Based on the speech sections detected in Step 1, processor 25 extracts the total number of utterances and the total total speech time of the audio data included in the audio file. Based on the extraction results of the total number of utterances and the total total speech time, processor 25 determines the number of divisions K (an example of a parameter) of the audio data included in the audio file in feature extraction unit 252, and divides the audio data included in the audio file according to the determined number of divisions K in feature extraction unit 252.

The processor 25 generates an audio file (e.g., a wav file) consisting of the speech section of each of the K divided pieces of audio data in the feature extraction unit 252 (Step 2), and extracts the features of the audio data included in the audio file in the feature extraction unit 252 (Step 3). The processor 25 determines parameters (see FIG. 5) for clustering performed by the audio clustering unit 253 for each of the K divided pieces of audio data, using the extraction results of the total number of utterances and the total total speech time of the audio data and the table of FIG. 5.

The processor 25 uses the output of the feature extraction unit 252 (specifically, the features of the utterance in the speech section, the parameters described with reference to Figure 5) to group (cluster) each utterance of the divided speech data by speaker in the speech clustering unit 253 (Step 4). The processor 25 uses the results of the clustering process in Step 4 to perform a cluster synthesis process as necessary in the speech clustering unit 253 (Step 5). The processor 25 outputs the results of the synthesis process in Step 5 (Step 6, see Figure 4).

Next, the operation procedure of the minutes-taking device 20 according to the first embodiment will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of the operation procedure of the minutes-taking device 20 according to the first embodiment. Each process shown in FIG. 6 is mainly executed by the processor 25 of the minutes-taking device 20.

In FIG. 6, when the processor 25 receives audio file data from the communication unit 21, for example, the processor 25 analyzes (e.g., VAD processing) the audio file (e.g., a data file composed of audio data of speech contents of multiple speakers during a conference) in the feature extraction unit 252 (St1). This allows the processor 25 to detect sections of audio data (speech sections) of each speech by multiple speakers contained in the entire audio file input from the communication unit 21 in the feature extraction unit 252. Based on this detected speech section, the processor 25 extracts the total number of utterances and the total total speech time of the audio data contained in the entire audio file input from the communication unit 21 in the feature extraction unit 252.

The processor 25 determines the number of divisions (one example of a parameter) of the voice data contained in the entire voice file in the feature extraction unit 252 based on the extraction results of the total number of utterances and the total total speaking time, and divides the voice data contained in the entire voice file in the feature extraction unit 252 according to the determined division number K (St2). For example, K is an integer equal to or greater than 2 and indicates the number of divisions. The processor 25 acquires one piece of voice data divided in step St2 in the feature extraction unit 252 (St3), and cuts out the speech section of the voice data (St4). In step St4, the processor 25 detects, as the speech section, a section in which a sound pressure equal to or greater than a sound pressure threshold (see above) stored in, for example, the ROM 22 or RAM 23 is detected.

For each speech section cut out in step St4, the processor 25 calculates and extracts the features of the voice data of that speech section in the feature extraction unit 252 (St5). Using the extraction results of the total number of utterances and the total speech time of the voice data of the speech section cut out in step St4 and the table of FIG. 5, the processor 25 determines parameters (see FIG. 5) for clustering performed by the voice clustering unit 253 in the feature extraction unit 252 (St6).

The processor 25 uses the parameters determined in step St6 and the features of the voice data of each speech section extracted in step St5 to group (cluster) each utterance of the single piece of voice data acquired in step St3 by speaker in the voice clustering unit 253 (St7). At this time, the processor 25 performs a synthesis process (e.g., an addition process) of the features of the voice data of each utterance included in each cluster indicating the grouping result of each utterance, and stores the result in the RAM 23.

The processor 25 determines whether or not to perform a synthesis process between clusters indicating the grouping results of each utterance (St8). For example, when the number of speakers information from the PCs 10, 30, and 40 is specified in advance (i.e., when the number of speakers is known), clustering is performed in step St7 to obtain a number of clusters that matches the specified number of speakers, and therefore synthesis process between clusters is not executed. When synthesis process between clusters is not performed (St8, NO), the processor 25 proceeds to step St11.

On the other hand, when the processor 25 determines that the clusters are to be synthesized (St8, YES), the processor 25 performs a similarity calculation between the clusters obtained by the clustering in step St7 in the voice clustering unit 253 (St9). This calculation of the similarity between the clusters may be realized by an existing technique (see FIG. 13), such as the cosine distance. For example, a similarity threshold (similarity value) used in the synthesis process between the clusters is stored in advance in the ROM 22 or RAM 23, and the processor 25 performs synthesis process between the clusters in the voice clustering unit 253 only when the similarity between the clusters calculated in step St9 is equal to or greater than the similarity threshold (St10). On the other hand, when the similarity between the clusters calculated in step St9 is less than the similarity threshold, the processor 25 does not perform synthesis process between the clusters.

The processor 25 determines whether or not the series of processes from step St3 to step St10 has been performed on each of all of the voice data obtained by the division process in step St2 (St11). The processor 25 repeats the series of processes from step St3 to step St10 on one piece of voice data obtained in step St3 until it determines that the series of processes from step St3 to step St10 has been performed on each of all of the voice data (St11, NO).

On the other hand, if the processor 25 determines that the series of processes from step St3 to step St10 have been performed on each of all the voice data obtained by the splitting process in step St2 (St11, YES), it determines whether or not the splitting process in step St2 was effective in order to determine whether or not a cross-cluster synthesis process is required between the clusters obtained from each of the K pieces of voice data split in step St2 (St12). If the splitting process has not been performed in step St2 (St12, NO), the process of the processor 25 in FIG. 6 ends.

On the other hand, if the processor 25 determines that the splitting process in step St2 was effective (i.e., splitting process was performed in step St2) (St12, YES), it performs a similarity calculation across the clusters obtained from each of the voice data split in step St2 (St13). This similarity calculation in step St13 may be the same as or different from the method in step St9. The processor 25 synthesizes the clusters in the voice clustering unit 253 only when the similarity between the clusters calculated in step St13 is equal to or greater than the similarity threshold (see above) (St14). Furthermore, the processor 25 performs a synthesis process (e.g., an addition process) on the features of the voice data of each utterance that constitutes the synthesized cluster, and stores the result in the RAM 23.

As described above, the minutes generating device 20 according to the first embodiment includes an input unit (e.g., the audio input unit 251) that inputs an audio file containing speech from multiple speakers, an utterance extraction unit (e.g., the feature extraction unit 252) that analyzes the audio file and extracts the number of utterances and the length of each utterance contained in the audio file, a parameter determination unit (e.g., the feature extraction unit 252) that determines multiple parameters for grouping multiple speakers based on at least one of the number of utterances and the length of each utterance, the feature extraction unit 252 that extracts the feature of each utterance within multiple speech sections contained in the audio file based on the parameters, and a classification unit (e.g., the audio clustering unit 253) that groups each utterance by speaker based on the feature of each utterance.

As a result, the minutes generating device 20 can derive clustering parameters from the number of utterances and the length of each utterance contained in the input audio file, regardless of whether information about the number of speakers at the meeting has been input in advance, and can perform clustering using these parameters, thereby enabling the utterances to be classified with high accuracy while estimating the number of speakers participating in the meeting.

The minutes generating device 20 further includes a division unit (e.g., feature extraction unit 252) that divides the audio file into multiple speech sections based on at least one of the number of utterances and the length of each utterance. The feature extraction unit 252 extracts features of the utterances in each speech section divided by the division unit. This allows the minutes generating device 20 to divide the audio data to find appropriate parameters for clustering when the total speech length included in the input audio file is long.

The multiple parameters also include a first parameter determined based on the number of utterances (e.g., how many seconds or more of data to use) and a second parameter determined based on the length of each utterance (e.g., the maximum number of classifications). This allows the minutes generation device 20 to quickly and efficiently perform the learning process of the trained model used for clustering by taking into account the number of utterances in the audio data contained in the input audio file and the length of each utterance, and as a result, can quickly and efficiently perform clustering using the trained model obtained by the learning process.

The parameter determination unit also determines the first parameter (e.g., the number of data N additions) by inputting the number of utterances into a predetermined calculation formula (e.g., if the number of utterances in the voice data is "100" and the number of data required for the learning process of the trained model is M (an integer), then "M/100" becomes N) or a predetermined table (see FIG. 5). Similarly, if the number of utterances in the voice data is "1000" and the number of data required for the learning process of the trained model is M (an integer), then "M/1000" becomes N. This allows the minutes generation device 20 to easily determine the first parameter for classifying (clustering) speakers with a certain degree of accuracy.

The parameter determination unit also determines the second parameter (e.g., the maximum number of classifications) by inputting the speech length of each utterance into a predetermined formula (e.g., if 60 seconds or more of data is required to classify the utterances with a certain degree of accuracy while estimating the number of speakers, then "L/60" is used if the total speech length of the audio data is L (an integer)) or into a predetermined table (see FIG. 5). This allows the minutes generation device 20 to easily determine the second parameter for classifying (clustering) speakers with a certain degree of accuracy.

The minutes generating device 20 also includes an output processing unit (e.g., the minutes generating output unit 254) that performs voice recognition on the voice data of each utterance grouped by speaker, and outputs the voice recognition results in a manner that allows each speaker to be identified. This allows the minutes generating device 20 to output the contents of utterances made by multiple speakers in a meeting in such a way that it is intuitively clear whether the speakers are the same or different.

(Embodiment 2)
In the second embodiment, an example of a minutes generating device will be described as an example of an utterance feature extraction device that improves the accuracy of feature extraction processing compared to the first embodiment. Since the configuration of the speaker classification system according to the second embodiment is the same as the configuration of the speaker classification system 50 according to the first embodiment, the same reference numerals are used to simplify or omit the description of the same contents, and the different contents will be described.

When extracting features, it is considered desirable to extract them on an utterance-by-utterance basis rather than on a word-by-word basis. For example, assume that speaker A (see Figure 1) utters "Good morning." In this case, in extracting features of the voice data of speaker A's "Good morning," a first method is to extract four features for each of "o," "ha," "yo," and "u," while a second method is to extract a feature for "Good morning" alone. As a result, the second method is considered to be superior. This is because, for example, even if the human ear hears a single sound, it is difficult to tell who is speaking, and a relatively long sound such as "Good morning" is more likely to reveal the atmosphere of the speaker's voice (in other words, the features of the voice data).

Figure 7 is a diagram showing an example of a voice waveform based on continuous speech by one speaker A. The voice waveform RWD1 shown in Figure 7 is the voice waveform of speaker A's speech, "Good morning, how are you?". If features can be extracted on an utterance-by-utterance basis, such as "Good morning, how are you?", it is preferable because it makes it easier to detect the atmosphere of the speech that is characteristic of that speaker (see above). However, with conventional sound pressure analysis, it has been difficult to detect when the speaker switches midway. The longer the detected speech section is, the higher the possibility that multiple speakers are speaking, so it is preferable to divide it.

For example, the audio waveform RWD1 shows a section P1 including an utterance section P2 of "Good morning," a silent section N1, and an utterance section P3 of "How are you?". The minutes creation device 20 according to the second embodiment adopts the utterance sections P2 and P3 as valid speech sections for feature extraction, rather than the entire section P1. In other words, the minutes creation device 20 does not extract features of the speech portion of the audio waveform RWD1 for the entire section P1, in which the utterance sections P2 and P3 continue almost continuously, but performs feature extraction processing for each of the utterance sections P2 and P3.

Figure 8 is a diagram showing a first example of a segmentation of a speech section used for feature extraction. Figure 9 is a diagram showing a second example of a segmentation of a speech section used for feature extraction. Here, it is assumed that, for example, speech data (speech waveform) of a 10-second speech section is obtained. As a first example of segmentation (segmentation), the minutes generating device 20 according to the second embodiment may split the entire 10-second speech waveform RWD2 in the middle in time into a first 5-second segment P11 and a last 5-second segment P12. As a second example of segmentation (segmentation), the minutes generating device 20 according to the second embodiment may split the entire 10-second speech waveform RWD3 into a first 6-second segment P13 and a last 4-second segment P14 so as not to include the silent segment N2 (i.e., the segment where the sound pressure is considerably reduced) in the middle in the feature extraction segment.

Figure 10 shows an example of a speech waveform based on the continuous speech by one speaker A shown in Figure 7 followed by the speech of another speaker B. When cutting out a speech section to extract features of the speech data, ideally the features of one speaker should be included in one speech section, but if the end of the speech (i.e., the end of the speech) is not detected easily and the speech section becomes too long, there is a possibility that another speaker will be mixed in.

For example, the audio waveform RWD4 shows a section P21 including a speech section P22 of "Good morning," a speech section P23 of "How are you?", and a speech section P24 of "How are you?". The minutes generating device 20 according to the second embodiment does not use the entire section P21 as a feature extraction section, but uses the speech sections P22, P23, and P24, which are defined by dividing section P21 by a predetermined number of seconds (e.g., 5 seconds), as valid speech sections. In other words, the minutes generating device 20 does not extract features of the speech portion of the audio waveform RWD4 for the entire section P21, in which the speech sections P22, P23, and P24 continue almost continuously, but performs feature extraction processing for each of the speech sections P22, P23, and P24 in the feature extraction unit 252.

Next, the operation procedure of the minutes-taking device 20 according to the second embodiment will be described with reference to FIG. 11. FIG. 11 is a flowchart showing an example of the operation procedure of the minutes-taking device 20 according to the second embodiment. Each process shown in FIG. 11 is mainly executed by the feature extraction unit 252 of the processor 25 of the minutes-taking device 20. In addition, the minutes-taking device 20 may execute each process shown in FIG. 11 as a subroutine of the feature extraction of step St5 shown in FIG. 6.

In FIG. 11, when the processor 25 inputs the voice data to be subjected to the feature extraction process in the feature extraction unit 252, it detects the speech section of the voice data (St21) and analyzes the speech time in the feature extraction unit 252 (St21). This analysis can be realized, for example, by using a trained model that performs VAD processing and that can be executed by the feature extraction unit 252. The processor 25 determines in the feature extraction unit 252 whether the length of the speech time (i.e., the speech length) extracted by the analysis in step St21 is equal to or greater than the time length threshold (e.g., 5 seconds, see FIG. 10) stored in the ROM 22 or RAM 23 (St22). If it is determined that the length of the speech time (i.e., the speech length) is equal to or less than the time length threshold (St22, NO), the process of the processor 25 proceeds to step St24.

On the other hand, if the processor 25 determines that the length of the speech time (i.e., the speech length) is equal to or greater than the time length threshold (St22, YES), the processor 25 divides the voice data input in step St21 into multiple pieces in the feature extraction unit 252 (St23). The processor 25 extracts and detects the voice data divided in step St23 or the voice data input in step St21 as a feature extraction section in the feature extraction unit 252 (St24). The processor 25 calculates and extracts the features of the voice data of the feature extraction section cut out in step St24 in the feature extraction unit 252 (St25).

After step St25, the processor 25 determines whether or not the division process was performed in step St23 (St26). If it is determined that the division process was not performed in step St23 (St26, NO), the process of the processor 25 in FIG. 11 ends.

On the other hand, if the processor 25 determines that the splitting process has been performed in step St23 (St26, YES), the feature extraction unit 252 calculates the similarity between the split voice data (e.g., the difference value between the features of the voice data) (St27). If the splitting process has been performed in step St23, there is a high probability that the same speaker originally spoke in one speech section. For this reason, the processor 25 calculates the similarity between the voice data before and after the splitting process (i.e., before and after the break), and if this similarity is equal to or greater than a fourth threshold (see below), combines the features of each voice data.

When the processor 25 determines that the similarity between the voice data calculated in step St27 is equal to or greater than the fourth threshold (similarity value) stored in the ROM 22 or RAM 23 (St28, YES), it synthesizes the features of each voice data (St29). On the other hand, when the processor 25 determines that the similarity between the voice data is equal to or less than the fourth threshold (St28, NO), the process of the processor 25 in FIG. 11 ends.

As described above, the minutes generating device 20 according to the second embodiment includes an input unit (e.g., audio input unit 251) that inputs an audio file containing speech from multiple speakers, a speech section detection unit (e.g., feature extraction unit 252) that analyzes the audio file using a first algorithm for detecting speech sections and detects multiple speech sections contained in the audio file, a feature extraction section detection unit (e.g., feature extraction unit 252) that extracts feature extraction sections in each of the multiple speech sections detected using a second algorithm for detecting speech sections, and a feature extraction unit 252 that extracts features of the extracted extraction sections.

As a result, the minutes generating device 20 can use a second algorithm different from the first algorithm for detecting speech sections included in an audio file to appropriately extract sections (extraction sections) for extracting features from each of the multiple speech sections detected using the first algorithm. Therefore, the minutes generating device 20 can appropriately extract features of the audio data of each utterance by multiple speakers, which can contribute to improving the accuracy of the clustering process.

The minutes generating device 20 also includes a classification unit (e.g., the voice clustering unit 253) that groups each utterance in multiple speech sections included in the audio file by speaker based on the feature amount of the extracted extraction section. This allows the minutes generating device 20 to group (cluster) each utterance by speaker with high accuracy while estimating the number of multiple speakers.

In addition, in the minutes generating device 20, the speech section detection unit divides a detected speech section into multiple speech sections when the speech section is longer than a predetermined time (for example, the above-mentioned time length threshold). This allows the minutes generating device 20 to reduce the possibility of another speaker being mixed in due to the speech section becoming too long, and appropriately extract the features of the audio data of an utterance by one speaker.

The first algorithm detects a detection section of a sound pressure equal to or greater than a first sound pressure threshold (an example of a sound pressure threshold) as a speech section. The second algorithm has stricter detection conditions than the first algorithm for detecting a speech section contained in an audio file in order to detect an extraction section of a feature, and detects a detection section of a sound pressure equal to or greater than a second sound pressure threshold that is greater than the first sound pressure threshold as an extraction section of a feature. For example, if the first sound pressure threshold is a predetermined reference sound pressure (e.g., 20 μPa), the first sound pressure threshold is 10 dB, and the second sound pressure threshold is 15 dB. This allows the minutes generating device 20 to further carefully select the audio data of the speech section detected by the first algorithm and detect the extraction section of a feature with high accuracy.

The first algorithm detects the detection section of the sound pressure equal to or greater than the first sound pressure threshold (one example of a sound pressure threshold) as the speech section. The second algorithm detects the detection section of the audio signal in the frequency band included in human voices (e.g., 100 to 1000 Hz) as the feature extraction section. This enables the minutes generation device 20 to further carefully select audio data having the frequency band of human voices from the audio data of the speech section detected by the first algorithm, and detect the feature extraction section with high accuracy.

The second algorithm further detects the detection section of the sound pressure equal to or greater than a second sound pressure threshold (see above) that is greater than the first sound pressure threshold as the feature extraction section. As a result, the minutes generating device 20 not only carefully selects sound data having a human voice frequency band from the voice data of the speech section detected by the first algorithm, but also carefully selects a second sound pressure threshold greater than the first sound pressure threshold from the voice data of the speech section detected by the first algorithm, so that the feature extraction section can be detected with higher accuracy.

In addition, in the minutes-of-meetings generating device 20, when one speech section detected using the first algorithm contains multiple (e.g., two) extraction sections detected using the second algorithm, the feature extraction unit 252 extracts the averaged result of each feature corresponding to the multiple extraction sections as a feature corresponding to the multiple extraction sections. This allows the minutes-of-meetings generating device 20 to obtain features of voice data that are more likely to represent the speaker by averaging (smoothing) each feature in the multiple extraction sections.

(Embodiment 3)
In embodiment 3, a further improvement example of the clustering according to embodiment 1 will be described. Since the configuration of the speaker classification system according to embodiment 3 is the same as the configuration of the speaker classification system 50 according to embodiment 1, the same reference numerals are used to simplify or omit the description of the same contents, and only the differences will be described.

Unlike Patent Document 1, the minutes-taking device 20 according to each embodiment groups (clusters) each utterance by multiple speakers participating in a meeting by speaker, even if information regarding the number of speakers is not available in advance. In embodiment 3, in order to perform such clustering (i.e., to appropriately narrow down the speakers), the minutes-taking device 20 narrows down the speakers by synthesizing clusters as necessary based not only on the similarity between clusters obtained by clustering (see FIG. 13 described later), but also on correlation coefficients based on the similarity between the speech data and the clusters obtained by clustering (see later).

Figure 12 is a diagram showing an example of a feature file used for feature extraction. For example, audio waveform RWD5 shows section 31 including feature extraction section P32, feature extraction section P33, and feature extraction section P34. Each of extraction sections P32, P33, and P34 is obtained by the detection method described with reference to embodiment 2, for example, but may be obtained by a detection method other than this detection method. Features are calculated and extracted for each of extraction sections P32, P33, and P34, and in Figure 12, for example, a feature file storing the values of three features is obtained.

If information about the number of speakers (e.g., X people) is given in advance as in Patent Document 1, the minutes-taking device 20 can simply cluster the feature files into X pieces as a result of the clustering process so that they match the given number (X people). However, if information about the number of speakers (e.g., X people) is not given in advance (so-called unsupervised), the minutes-taking device 20 must perform clustering while estimating how many people are speaking.

For this purpose, one possible method (comparison example) is to first perform clustering using appropriate initial values (classification initial values), and then use the similarity between each cluster obtained as a result of the processing to synthesize clusters with high similarity (see Figure 13). Figure 13 is a diagram showing an example of the procedure for the outline of the clustering operation for estimating the number of speakers included in a feature file.

In FIG. 13, each of a plurality of (e.g., 36) feature files (see FIG. 12) extracted from an audio file is plotted in a two-dimensional coordinate system (Step 1). This plot can be realized by, for example, arranging the values of two types of feature calculated based on the feature files at positions Q1, Q2, Q3, ..., Q36, which are specified as coordinate values. After Step 1, clustering is performed with an appropriate initial value (classification initial value, for example, 5), resulting in classification into five clusters CL1, CL2, CL3, CL4, and CL5 (Step 2). Note that this initial value (classification initial value) is an example of a parameter determined by the feature extraction unit 252 described with reference to the first embodiment.

After Step 2, the similarity between the clusters CL1 to CL5 obtained in Step 2 is calculated (Step 3). In Step 3, for example, the distance between the center positions (coordinate values) of the clusters CL1 to CL5 is calculated as the similarity. Note that the similarity does not have to be calculated based on the distance between the center positions of the clusters. If it is determined that the similarity calculated in Step 3 is equal to or greater than a predetermined threshold (for example, the similarity threshold described with reference to the first embodiment), the corresponding clusters are combined (Step 4). For example, in FIG. 13, clusters CL2 and CL3 are combined to generate cluster CL6, and clusters CL4 and CL5 are combined to generate cluster CL7.

However, there was a high possibility that appropriate synthesis would not be possible based solely on the similarity between clusters. This was because, for example, when the similarity between clusters was close to the above-mentioned predetermined threshold, synthesis could result in erroneous classification of speech from different speakers as being in the same cluster.

Therefore, in the third embodiment, the results are narrowed down by combining clusters as necessary based on not only the similarity between clusters obtained by clustering (see FIG. 13 described below), but also on correlation coefficients based on the similarity between the speech data and the clusters obtained by clustering (see below) (see FIG. 14 to FIG. 24).

First, an overview of the processing of the minutes generating device 20 according to embodiment 3 will be described by exemplifying a case where the speech sections contained in the audio file input to the processor 25 are not very long (i.e., a case where data division processing is not necessary). Figure 14 is a graph showing an example of the clustering processing result when data division processing is not performed. Figure 15 is a table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering. Figure 16 is a graph showing the behavior of the similarity between each utterance and each cluster based on the calculation result of Figure 15.

For example, as shown in FIG. 14, assume that utterances corresponding to feature files (see FIG. 12) are plotted two-dimensionally and classified into three clusters CLx, CLy, and CLz through clustering by the speech clustering unit 253 according to embodiment 3. The numbers in circles in FIG. 14 correspond to the utterance numbers, and it is shown here that a total of 10 utterances (feature files) have been classified. Cluster CLx has three utterances: utterances (1), (2), and (3). Cluster CLy has three utterances: utterances (4), (5), and (6). Cluster CLz has four utterances: utterances (7), (8), (9), and (10).

The speech clustering unit 253 calculates the similarity between each cluster CLx, CLy, and CLz and all of the utterances (1) to (10) (see FIG. 15). Here, the similarity is calculated to have a value between 0 and 100, and so on. As shown in FIG. 14, cluster CLx has three utterances, utterances (1), (2), and (3), so the similarity between cluster CLx and each of the utterances (1) to (3) is very high (see dotted line frame). Similarly, cluster CLy has three utterances, utterances (4), (5), and (6), so the similarity between cluster CLy and each of the utterances (4) to (6) is very high (see dotted line frame). Similarly, cluster CLz has four utterances, utterances (7), (8), (9), and (10), so the similarity between cluster CLz and each of the utterances (7) to (10) is very high (see dotted line frame).

Furthermore, the voice clustering unit 253 calculates a correlation coefficient G indicating the degree of correlation between the clusters, using the calculation results of the similarity between each cluster CLx, CLy, CLz and all utterances (1) to (10) and a predetermined evaluation function ES (cluster, utterance number, similarity). The correlation coefficient G is calculated from the output value of the predetermined evaluation function ES (cluster, utterance number, similarity), and takes a value of, for example, 0 to 1. In other words, if the correlation coefficient G is equal to or greater than a predetermined correlation threshold stored in advance in the ROM 22 or RAM 23, the voice clustering unit 253 performs synthesis processing on each cluster corresponding to the correlation coefficient G. On the other hand, if the correlation coefficient G is less than the above-mentioned predetermined correlation threshold, the voice clustering unit 253 does not perform synthesis processing on each cluster corresponding to the correlation coefficient G.

Figure 16 shows a graph illustrating the characteristics of the vertical axis (similarity) against the horizontal axis (utterance number), using the similarity described with reference to Figure 15. As can be seen from the behavior of the similarity (score) between clusters CLx, CLy, and CLz and each utterance, there was no correlation coefficient G between clusters that exceeded a predetermined correlation threshold. In other words, the voice clustering unit 253 does not perform synthesis processing on each cluster corresponding to the correlation coefficient G (specifically, the pair of clusters CLx, CLy, the pair of clusters CLy, CLz, and the pair of clusters CLz, CLx).

Next, an example of the operation procedure of the minutes generating device 20 according to the third embodiment when data division processing is not required will be described with reference to FIG. 17 and FIG. 18. FIG. 17 is a flowchart showing an example of the basic operation procedure of the minutes generating device according to the third embodiment when data division processing is not performed. FIG. 18 is a flowchart showing an example of the operation procedure of the subroutine of the cluster synthesis processing shown in FIG. 17. Each process shown in FIG. 17 and FIG. 18 is mainly executed by the voice clustering unit 253 of the processor 25 of the minutes generating device 20. The minutes generating device 20 may execute each process shown in FIG. 17 as a subroutine of the clustering of step St7 shown in FIG. 6. In addition, the minutes generating device 20 executes each process shown in FIG. 18 as a subroutine of the cluster synthesis processing of step St33 shown in FIG. 17.

In FIG. 17, the processor 25 acquires data to be input for clustering (e.g., voice data of all utterances (in other words, data of feature files), parameters described with reference to embodiment 1, and a trained model for clustering) in the voice clustering unit 253 (St31). The processor 25 uses the data acquired in step St31 to perform clustering in the voice clustering unit 253 (St32). After step St32, the processor 25 performs cluster synthesis processing in the voice clustering unit 253 (St33).

In FIG. 18, the processor 25 calculates the similarity between each cluster classified by the clustering in step St32 in the voice clustering unit 253 (St41). If it is determined that the similarity between each cluster calculated in step St41 is less than the similarity threshold (see embodiment 1) (St42, NO), the processing of the processor 25 proceeds to step St44. On the other hand, if the processor 25 determines that the similarity between each cluster calculated in step St41 is equal to or greater than the similarity threshold (see embodiment 1) (St42, YES), the processor 25 performs a synthesis process on the corresponding clusters in the voice clustering unit 253 (St43).

The processor 25 also calculates the similarity between each cluster classified by the clustering in step St32 and the voice data of all utterances acquired in step St31 in the voice clustering unit 253 (St44, see FIG. 15). The processor 25 calculates the correlation coefficient G indicating the degree of correlation between the clusters in the voice clustering unit 253 using the calculation result of the similarity between each cluster and all utterances and a predetermined evaluation function ES (cluster, utterance number, similarity) (St45). If the processor 25 determines that the correlation coefficient G calculated in step St44 is equal to or greater than a predetermined correlation threshold previously stored in the ROM 22 or RAM 23 (St46, YES), the voice clustering unit 253 performs a synthesis process on each cluster corresponding to the correlation coefficient G (St47). On the other hand, if it is determined that the correlation coefficient G is less than the above-mentioned predetermined correlation threshold (St46, NO), the process of the processor 25 in FIG. 18 ends.

Next, an overview of the processing of the minutes generating device 20 according to embodiment 3 will be described by exemplifying a case where the speech section included in the audio file input to the processor 25 is long (i.e., a case where data division processing is required). Figure 19 is a graph showing an example of each clustering processing result when data division processing is performed. Figure 20 is a graph showing an example of synthesis of each clustering processing result of Figure 19. Figure 21 is a table showing an example of calculation of the similarity between each cluster. Figure 22 is a graph showing the behavior of the similarity between each utterance and each cluster obtained by clustering. Figure 23 is a table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering.

For example, as shown in FIG. 19, when the speech section included in the audio file is long (for example, when there is audio data for 20 utterances), the data is divided by the audio clustering unit 253 (see FIG. 24), and the classification results corresponding to the two clustering process results are shown by exemplifying the division into two pieces of audio data. Specifically, the cluster CLx1 in FIG. 19 has three utterances, utterances (1), (2), and (3). The cluster CLy1 has three utterances, utterances (4), (5), and (6). The cluster CLz1 has four utterances, utterances (7), (8), (9), and (10). Similarly, the cluster CLx2 in FIG. 19 has three utterances, utterances (11), (12), and (13). The cluster CLy2 has three utterances, utterances (14), (15), and (16). Cluster CLz2 has four utterances: (17), (18), (19), and (20). Figure 20 shows the clusters shown in Figure 19 superimposed on each other.

The speech clustering unit 253 calculates the similarity between each cluster (specifically, clusters CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2). The results of this similarity calculation are shown in FIG. 21. According to FIG. 21, the similarity (98) between clusters CLx1 and CLx2 and the similarity (91) between clusters CLy1 and CLy2 are very high, so they may be subject to synthesis processing (in other words, clusters CLx1 and CLx2, and clusters CLy1 and CLy2 may be identified as the same speaker) (see FIG. 20). However, although the similarity (62) between clusters CLy1 and CLz2 and the similarity (58) between clusters CLy2 and CLz2 appear to be high, it is difficult to know whether they may be subject to synthesis processing.

Therefore, the speech clustering unit 253 calculates the similarity between each cluster CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2 and all of the utterances (1) to (20) (see FIG. 23). As shown in FIG. 23, since cluster CLx1 has three utterances, utterances (1), (2), and (3), the similarity between cluster CLx1 and each of the utterances (1) to (3) is very high (see dotted line frame). Similarly, the similarity between cluster CLx2 and each of the utterances (11) to (13) is very high (see dotted line frame). Similarly, since cluster CLy1 has three utterances, utterances (4), (5), and (6), the similarity between cluster CLy1 and each of the utterances (4) to (6) is very high (see dotted line frame). Similarly, cluster CLy2 has three utterances, utterances (14), (15), and (16), so the similarity between cluster CLy2 and each of utterances (14) to (16) is very high (see dotted line frame). Similarly, cluster CLz1 has four utterances, utterances (7), (8), (9), and (10), so the similarity between cluster CLz1 and each of utterances (7) to (10) is very high (see dotted line frame). Similarly, cluster CLz2 has four utterances, utterances (17), (18), (19), and (20), so the similarity between cluster CLz2 and each of utterances (17) to (20) is very high (see dotted line frame).

Furthermore, the voice clustering unit 253 calculates a correlation coefficient G indicating the degree of correlation between the clusters using the calculation results of the similarity between each cluster CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2 and all utterances (1) to (20) and a predetermined evaluation function ES (cluster, utterance number, similarity). The correlation coefficient G is calculated from the output value of the predetermined evaluation function ES (cluster, utterance number, similarity), and takes a value of, for example, 0 to 1. In other words, if the correlation coefficient G is equal to or greater than a predetermined correlation threshold value stored in advance in the ROM 22 or RAM 23, the voice clustering unit 253 performs synthesis processing on each cluster corresponding to the correlation coefficient G. On the other hand, if the correlation coefficient G is less than the above-mentioned predetermined correlation threshold value, the voice clustering unit 253 does not perform synthesis processing on each cluster corresponding to the correlation coefficient G.

Figure 22 shows a graph illustrating the characteristics of the vertical axis (similarity) against the horizontal axis (utterance number), using the similarity explained with reference to Figure 23. As can be seen from the behavior of the similarity (score) between clusters CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2 and each utterance, the correlation coefficients G between clusters that exceed a predetermined correlation threshold are the correlation coefficient between clusters CLx1 and CLx2 and the correlation coefficient between clusters CLy1 and CLy2. In other words, the voice clustering unit 253 performs cluster synthesis processing only for the pair of clusters CLx1 and CLx2 and the pair of clusters CLy1 and CLy2.

Next, an example of the operation procedure of the minutes generating device 20 according to embodiment 3 when data division processing is required will be described with reference to FIG. 24. FIG. 24 is a flowchart showing an example of the basic operation procedure of the minutes generating device 20 according to embodiment 3 when data division processing is required. The processes shown in FIG. 24 are mainly executed by the voice clustering unit 253 of the processor 25 of the minutes generating device 20. The minutes generating device 20 executes the processes shown in FIG. 18 as subroutines of the cluster synthesis process of steps St33 and St53 shown in FIG. 24. In the description of FIG. 24, the same processes as those in FIG. 18 are given the same step numbers, and the description is simplified or omitted, and the different contents are described.

In FIG. 24, the processor 25 divides the audio data contained in the entire input audio file in the audio clustering unit 253 (St51). The processor 25 executes the series of processes from steps St31 to St33 described with reference to FIG. 18 in the audio clustering unit 253 for each piece of audio data divided in step St51. The processor 25 determines whether or not the series of processes from steps St31 to St33 has been executed for all of the audio data divided in step St51 (St52). The processor 25 repeats the execution of the series of processes from steps St31 to St33 for the remaining audio data divided in step St51 until the series of processes from steps St31 to St33 has been executed for all of the audio data divided in step St51.

On the other hand, if the processor 25 determines that the series of processes from step St31 to step St33 have been performed on all of the audio data divided in step St51 (St52, YES), it performs a cluster synthesis process (see Figure 18) between each piece of audio data that constitutes all of the audio data divided in step St51 (St53, see Figures 19, 20, 22 and 23).

As described above, the minutes generating device 20 according to the third embodiment includes an input unit (e.g., a voice input unit 251) that inputs an audio file containing speech from multiple speakers, a feature extraction unit 252 that extracts features of each utterance within multiple speech sections included in the audio file, and a classification unit (e.g., a voice clustering unit 253) that groups each utterance by speaker based on the features of each utterance. When a predetermined condition is met, the classification unit performs group synthesis to synthesize the multiple grouped groups into a group of one speaker.

As a result, the minutes generation device 20 can perform clustering using the features of the audio data of each utterance contained in the input audio file, regardless of whether information regarding the number of speakers at a meeting has been input in advance, and can perform cluster synthesis processing when certain conditions are met, thereby performing appropriate clustering (speaker classification) to narrow down and estimate the number of speakers with a high degree of accuracy.

In addition, in the minutes-of-meetings generating device 20, the classification unit derives the similarity (e.g., brute-force calculation of similarity, see FIG. 23) between multiple groups, each having audio data of one or more utterances for each speaker, and the audio data of multiple speech sections included in the audio file, as group synthesis, and derives a correlation coefficient between the groups using the derived similarity. In addition, when the correlation coefficient is equal to or greater than a predetermined threshold, the classification unit determines that a predetermined condition is met, and synthesizes two or more groups whose correlation coefficient is equal to or greater than a predetermined threshold into a group of one speaker (i.e., cluster synthesis processing). This allows the minutes-of-meetings generating device 20 to perform cluster synthesis processing with high accuracy using not only the similarity between clusters, but also the correlation coefficient based on the similarity between the clusters and the audio data of the utterances.

The minutes generating device 20 further includes an utterance extraction unit (e.g., feature extraction unit 252) that analyzes the audio file and extracts the number of utterances and the utterance length of each utterance contained in the audio file, and a division unit (e.g., feature extraction unit 252) that divides the audio file into multiple utterance sections based on the number of utterances and the utterance length of each utterance. As a result, even if the utterance sections contained in the audio file are long, the minutes generating device 20 can adaptively divide the audio data of the input audio file based on the number of utterances and the utterance length of each utterance, thereby performing highly accurate clustering.

In addition, in the minutes-of-meetings generating device 20, the classification unit groups each utterance within each divided speech section by speaker for that speech section. When a predetermined condition is met, the classification unit uses the grouping processing results for each divided speech section to perform group synthesis between the grouping processing results for different speech sections. As a result, even when the same speaker speaks in a time-discontinuous manner, the minutes-of-meetings generating device 20 can appropriately perform synthesis processing using the clustering processing results (i.e., clusters) between different divided voice data, thereby accurately improving the accuracy of estimating the number of speakers (in other words, the accuracy of narrowing down the number of speakers).

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, amendments, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components in the various embodiments described above may be combined in any manner as long as it does not deviate from the spirit of the invention.

The present disclosure is useful as a speech classification device and method that accurately classifies utterances while estimating the number of speakers participating in a conference, regardless of whether information about the number of speakers in the conference is input in advance.

10, 30, 40 PCs
11 Interface 12 Imaging unit 13, 25 Processor 14, 24 Storage 15 Communication device 16 Operation device 17 Sound collection device 18 Display 19 External interface 21 Communication unit 22 ROM
23 RAM
50 Speaker classification system 251 Voice input unit 252 Feature extraction unit 253 Voice clustering unit 254 Minutes generation output unit MIC Sound collection device

Claims (7)

an input unit for inputting an audio file including speech from multiple speakers;
an utterance extraction unit that analyzes the audio file and extracts the number of utterances and the length of each utterance included in the audio file;
a parameter determination unit that determines a plurality of parameters for grouping the plurality of speakers based on at least one of the number of utterances and the utterance length of each utterance;
a feature extraction unit that extracts a feature of each utterance in a plurality of speech sections included in the audio file based on the parameters;
A classification unit that classifies each of the utterances into groups for each speaker based on the feature amount of each of the utterances.
Speech classifier. a division unit that divides the audio file into a plurality of speech sections based on at least one of the number of utterances and the length of each utterance,
The feature extraction unit extracts features of utterances in each utterance section divided by the division unit.
The speech classification device of claim 1 . the plurality of parameters include a first parameter determined based on the number of utterances and a second parameter determined based on an utterance length of each of the utterances;
The speech classification device of claim 1 . the parameter determination unit determines the first parameter by inputting the number of utterances into a predetermined formula or a predetermined table;
The speech classification device of claim 3 . the parameter determination unit determines the second parameter by inputting the utterance length of each utterance into a predetermined formula or a predetermined table.
The speech classification device of claim 3 . an output processing unit that performs speech recognition on the speech data of each of the utterances grouped by speaker, and outputs a speech recognition result in a manner that allows each speaker to be identified,
The speech classification device of claim 1 . 1. A method for speech classification executed by a speech classification device that is a computer, comprising:
A step of inputting an audio file including utterances from multiple speakers;
analyzing the audio file and extracting the number of utterances contained in the audio file and the length of each utterance;
determining a plurality of parameters for grouping the plurality of speakers based on at least one of a number of utterances and an utterance length of each utterance;
extracting a feature amount of each utterance in a plurality of speech sections included in the audio file based on the parameters;
and classifying the utterances into groups for each speaker based on the feature amount of each utterance.
Speech classification methods.

Images