JP6440967B2 - End-of-sentence estimation apparatus, method and program thereof - Google Patents

Description

  The present invention relates to a technique for estimating a sentence end symbol for giving semantic information to a speech recognition result.

  There is a demand for applying speech recognition technology to automatically create an utterance memo for each utterance of a conference participant. The utterance memo created by voice recognition enables the reflection of individual utterance contents during the conference and facilitates discussion. It also has the advantage of facilitating reviewing the meeting after the meeting and reducing the time and effort of creating the minutes.

  In order to realize automatic creation of an utterance memo, it is not sufficient to make text only phonemic information by speech recognition. One of the reasons is that it is not possible to determine where the sentence breaks based on phonological information alone. As a result, there are disadvantages such as an increase in the time for creating the minutes due to a decrease in the readability of the text, and the creation of the minutes including a meaning error due to a misunderstanding.

  For this reason, Non-Patent Document 1 discloses a technique for automatically giving punctuation marks to a speech recognition result. FIG. 13 shows a flow of processing of the automatic punctuation technology in Non-Patent Document 1. The corresponding technology uses a method of machine learning called conditional random field using words, parts of speech, clause boundaries, dependency information obtained by morphological analysis of speech recognition results, and time information between utterances. To estimate the position where punctuation marks should be added. Thereafter, punctuation marks are assigned to the punctuation mark assignment positions estimated from the previous processing in the speech recognition result.

  A dialogue between two or more people such as a conference includes many utterances including meanings of questions and emphasis. As a result, there are utterances having the same phonological information but different meanings. For example, the phonological information “is it?” Can be extracted from both the utterance “is it?” In the meaning of the question and the utterance “is it?” In the sense of convincing. On the other hand, when the same punctuation mark is given to the end of the sentence according to the conventional technique, information on the meaning of the utterance is lost, and the meaning of the utterance may be misunderstood. In the above example, there is a possibility that all utterances having phonological information “Is that so?” Are considered to be satisfactory. As a result, the minutes with the meaning of the utterance are created, and there is a disadvantage that the human being who read the minutes has a misunderstanding. From the above, when two or more conversations such as a conference are assumed, it is necessary to have semantic information in the speech recognition result.

  One way to give semantic information to sentences is to use sentence endings. For example, it is possible to give the meaning of a question by adding a question mark “?” At the end of the sentence. Therefore, if a plurality of sentence ending symbols are prepared and a sentence ending symbol that matches the meaning of the utterance can be automatically given, it can be said that the semantic information can be provided.

  When automatically assigning an end-of-sentence symbol, it is difficult to assign the symbol with high accuracy even if the subject of the prior art automatic punctuation-point assignment is expanded from the punctuation mark to the end-of-sentence symbol. This is because the utterance tendency of the speakers participating in the dialogue and the appearance tendency of the end-of-sentence symbols depend on the dialogue situation, that is, the location of the dialogue and the position of the speaker.

  FIG. 14 shows an example of the relationship between the dialogue status, the utterance tendency, and the appearance tendency of sentence ending symbols. For example, speakers in lectures tend to have less prosodic fluctuations in utterances, and the utterance contents are calm and many question utterances. Therefore, there are many punctuation marks and few question marks. Therefore, when automatically assigning a sentence ending symbol to a speaker's utterance in a lecture, a small change in prosodic variation is detected and used for adding a sentence ending symbol. It is valid. However, participants in free discussions tend to have large prosodic fluctuations in utterances, and there are many emotional utterances such as questions and emphasis, so question marks and exclamation marks are likely to appear at the end of sentences. Therefore, when automatically assigning end-of-sentence symbols to utterances of participants in free discussions, a standard that detects only large changes in prosodic changes and uses them to add end-of-sentence marks, making it difficult for question marks and exclamation points to appear It is more effective to provide

Yuya Akita and Tatsuya Kawahara, “Automatic Insertion Based on Multiple Annotation of Reading Marks for Lectures”, IPSJ Journal, Vol.54, No.2, pp.463-470, Feb. 2013.

  From the above, in order to achieve high-accuracy sentence ending symbol assignment, a criterion for assigning sentence ending symbols according to the conversation situation should be given. However, since it is assumed in the prior art that there is only one speaker, the conversation situation is not considered and the end-of-sentence symbol is always given according to the same standard. As a result, there is a possibility that the accuracy of automatic assignment of sentence ending symbols is lowered.

The purpose of this invention, the end of the sentence symbol estimator for estimating the end of the sentence symbols using dialogue situation wherein, to provide a method and a program.

The sentence ending symbol estimation device according to one aspect of the present invention uses the central speaker degree of each speaker of a dialog carried out by a plurality of speakers as an index indicating the ratio of each speaker's utterance in the dialog. The speaker bias is used as an index representing the length of the speaker's utterance in the dialogue, the dialogue severity is used as an index representing the tone of the speaker during the dialogue, A dialog situation feature calculation device for calculating a dialog situation feature of a dialogue using at least one of speaker bias and dialogue severity as a dialogue situation feature, and a plurality of dialogue situation features calculated by the dialogue situation feature calculation device. Select the end-of-sentence giving standard corresponding to the dialogue from the end-of-sentence giving norms, and use the selected end-of-sentence giving norm, the acoustic features and language features of the dialogue to end the text And a, and endnotes symbol estimator for estimating the.

  It is possible to calculate the dialogue situation feature used to estimate the sentence ending symbol corresponding to the meaning of the utterance. Moreover, the sentence end symbol corresponding to the meaning of the utterance can be estimated.

The block diagram for demonstrating the example of the sentence ending symbol estimation apparatus. The flowchart for demonstrating the example of the sentence ending symbol estimation method. The block diagram for demonstrating the example of a dialog condition characteristic calculation apparatus. The flowchart for demonstrating the example of the dialog condition feature calculation method. The figure which shows the example of the relationship between the degree of dialogue severity and speaker bias, and the place of dialogue in the real world. The figure which shows the example of the center speaker degree for every field of dialogue, and the positioning of a speaker. The figure which shows the example of the speech area of the audio | voice for every speaker. The figure which shows the example of a center speaker degree and a speaker bias degree. The figure which shows the example of the whole speech area and non-utterance area in a dialog. The block diagram for demonstrating the example of the sentence end symbol provision model production | generation part. The block diagram for demonstrating the example of the sentence end symbol provision model production | generation part. The block diagram for demonstrating the example of the regression coefficient learning part 17. FIG. The block diagram for demonstrating the conventional punctuation mark automatic provision technique. The figure which shows the example of the relationship of a dialog situation, the tendency of speech, and the appearance tendency of a sentence end symbol.

[Overall flow]
First, using the voice of each speaker who participated in the dialogue, we estimated the place of dialogue based on the scale representing the degree of bias of the speaker's utterance length and the severity of the speaker's tone during the dialogue, Analyzing the utterance rate of each speaker during the dialogue, the position of the speaker in the dialogue is estimated.

  Next, the sentence ending symbol assignment model is learned for each dialogue place and speaker position. First, a speech database including speeches of various dialogue places / speaker positions and transcript text data with sentence ending symbols corresponding to each speech data are prepared. Based on the estimated dialogue field and speaker positioning information, the speech database is divided so that the voices having the same dialogue field and speaker positioning are in the same set. For each divided speech database, a sentence ending symbol assignment model is learned by using, as learning data, acoustic data of conversational speech and text data to which a sentence ending symbol is assigned for each word boundary of the speech recognition result. This end-of-sentence symbol assignment model is a norm for the end-of-sentence symbol in a certain dialogue place or speaker position. Note that learning of the sentence ending symbol assignment model does not necessarily have to be performed, but in that case, a threshold value of an acoustic feature or a language feature is used as a sentence ending symbol assignment criterion.

  When automatically assigning end-of-sentence symbols, the location of the dialogue and the position of the speaker are estimated from the input dialogue voice, and the end-of-sentence symbol is assigned using the end-of-sentence assignment standard that is close to the location of the dialogue and the speaker. Thus, it is possible to obtain a speech recognition result text to which a sentence ending symbol is automatically assigned in accordance with the position of each speaker during the conversation.

[Dialogue status features]
Three scales are defined: "Dialogue severity", "Speaker bias degree", and "Central speaker degree" that can express the conversation situation in the real world and can be calculated from the speech of the conversation participants. These three measures are collectively called “dialogue situation characteristics”.

  “Dialogue severity” is a scale indicating whether the tone of dialogue participants is strict. In other words, the dialogue severity of a dialogue is an index representing the severity of the dialogue. This is determined based on, for example, the size of the prosody change of the conversation participant and the length of the non-speech interval in the entire conversation.

  The “speaker bias degree” is a scale indicating whether or not there is a bias in the length of the utterance section for each speaker in the dialogue. In other words, the speaker bias degree of a certain dialog is an index representing the speaker's utterance length bias degree in the dialog. This is calculated from the size of the utterance ratio of the speaker who speaks most in the dialogue.

  These measures of “dialogue severity” and “speaker bias” express the place where the dialogue took place. For example, in the lecture, the dialogue severity and speaker bias are high, and in the parliament, the dialogue severity is high but the speaker bias is low. FIG. 5 shows an example of the relationship between the degree of dialogue severity and speaker bias and the real-world dialogue field.

  “Central speaker degree” is a scale indicating the size of a speaker's utterance ratio in a dialog, and is related to the position of the speaker in a certain dialog. In other words, the central speaker degree of a speaker having a certain dialogue is an index indicating the ratio of the speaker's utterance in the dialogue. This is obtained from the size of the utterance ratio in the entire conversation participant. For example, a speaker having a high degree of central speaker at a lecture is a speaker, and a speaker having a low degree of central speaker is a questioner.

  FIG. 6 shows an example of the degree of central speaker and the position of the speaker for each dialogue place. One degree of dialogue severity and one degree of speaker bias are required for the entire dialogue, whereas a central speaker degree is obtained for each speaker participating in the dialogue. In addition, these dialogue situation features are assumed to be invariant in the dialogue, and the single speaker severity, speaker bias, and the number of speakers as many as the number of speakers are calculated using all intervals from the beginning of the dialogue to the end of the dialogue. Suppose you want.

  Prepare multiple end-of-sentence assignment rules based on the value of the dialog status feature. As will be described later, the ending symbol assignment norm may be, for example, a probability value of a ending symbol assignment model obtained by collecting only voices having similar values of dialogue situation characteristics and learning in advance using those voices. It may be threshold processing of features and language features. When automatically assigning end-of-sentence symbols, the conversation situation feature is automatically estimated from the input speech, and the end-of-sentence estimation is selected by selecting the end-of-sentence giving standard when the values are close. As described above, by changing the sentence ending symbol provision norm based on the conversation situation characteristics, sentence ending symbols can be assigned according to the conversation situation, and the sentence ending symbol assignment accuracy is improved.

[Embodiment]
Hereinafter, embodiments of the sentence ending symbol estimation apparatus and method will be described.

  As shown in FIG. 1, the sentence ending symbol estimation device includes, for example, a dialog situation feature calculation device 1, a speech recognition unit 2, an acoustic feature extraction unit 3, a text analysis unit 4, a sentence ending symbol estimation unit 5, and a sentence ending symbol addition unit 6. ing. The sentence ending symbol assigning unit 6 may not be provided.

  The sentence ending symbol estimation method is realized, for example, by the sentence ending symbol estimation device performing the processing from step S1 to step S6 in FIG.

  In this embodiment, it is assumed that a voice recording dialogues of a plurality of speakers is input. At this time, it is assumed that the sound for each speaker is individually recorded. This input may be a voice recorded with a close-up microphone such as a headset attached to each speaker, or a technique for speaker classification and sound source separation using a single or a plurality of microphones. For example, the voice may be separated for each speaker using Japanese Patent No. 4964204). Note that the number of speakers recording voice is the number N of speakers used in the dialog situation feature extraction unit. At this time, it is assumed that speakers who existed in the dialog but never spoken or speakers whose individual voices were not recorded are not included in the number of speakers. The input voice for each speaker is input to the dialog situation feature calculation apparatus 1 and the voice recognition unit 2.

<Dialogue situation feature calculation device 1 (FIGS. 1 and 3)>
The dialog situation feature calculation device 1 calculates the dialog situation feature using the input voice for each speaker (step S1). The calculated dialog situation feature is output to the sentence ending symbol estimation unit 5.

  As shown in FIG. 3, the conversation situation feature calculation apparatus 1 includes an utterance interval detection unit 11, a fundamental frequency extraction unit 12, an entire utterance interval detection unit 13, a central speaker degree speaker bias degree calculation unit 14, and dialogue severity estimation. For example, a feature calculation unit 15 and a dialogue severity calculation unit 16 are provided.

  The dialog situation feature calculation method is realized, for example, by the dialog situation feature calculation apparatus performing the processing from step S11 to step S17 in FIG.

  Hereinafter, details of each part in the dialog situation feature calculation apparatus 1 will be described. Note that, in the following utterance section detection unit 11 and fundamental frequency extraction unit 12, for example, a method of dividing and analyzing input speech every short time of about 10 msec, for example, is taken.

<< Speech section detector 11 (FIG. 3) >>
The utterance section detection unit 11 detects the utterance section for each speaker using the inputted voice for each speaker (step S11). Information about the detected utterance period is output to the entire utterance period detection unit 13, the fundamental frequency extraction unit 12, and the central speaker degree speaker bias degree calculation unit 14.

  The utterance section refers to a section from the start time to the end time of one utterance of the speaker, and the voice for each speaker includes one or more utterance sections. Further, it is assumed that a short period such as breathing is included in the utterance section, but a long period such as a listening section of another person's utterance is not included in the utterance section. The determination of whether the interval is included in the utterance section is realized by threshold processing of the time between utterances, for example. For example, suppose that it is included in the utterance interval for less than 1 second and is not included in the utterance interval for longer than 1 second. FIG. 7 shows an example of a speech utterance section for each speaker. In this embodiment, the speech segment detection is performed by the threshold processing of the short time voice power, but any existing speech segment detection method may be used.

<< Basic frequency extraction unit 12 (FIG. 3) >>
The fundamental frequency extraction unit 12 extracts the fundamental frequency for each speaker using the input voice for each speaker and the information about the input speech period (step S12). As a result, a time series of fundamental frequencies is generated. Information about the extracted fundamental frequency is output to the dialogue severity estimation feature calculation unit 15.

  The fundamental frequency extraction process is performed for each utterance section of the voice for each speaker. For example, the fundamental frequency extraction is performed using an autocorrelation method. Of course, any existing fundamental frequency extraction method may be used.

<< Whole utterance section detector 13 (FIG. 3) >>
The entire utterance section detection unit 13 detects the entire utterance section using the input utterance sections of all the speakers (step S13). Information about the detected entire utterance section is output to the central speaker degree speaker bias degree calculation unit 14 and the dialogue severity estimation feature calculation unit 15.

  The entire utterance interval refers to an interval that is an utterance interval of one or more speakers in the dialogue. FIG. 7 shows an example of detecting the entire utterance section. Thus, the entire utterance interval is an interval obtained by combining the utterance intervals of one or more speakers.

<< Center speaker degree Speaker bias calculator 14 (Fig. 3) >>
The central speaker degree speaker bias degree calculation unit 14 calculates the central speaker degree and the speaker bias degree by using the information about the input utterance intervals of all the speakers and the information about the input whole utterance intervals. Calculate (step S14, step S15). Since the central speaker degree is calculated for each speaker, the central speaker degree may be referred to as “central speaker degree for each speaker”.

First, the central speaker degree speaker bias degree calculation unit 14 obtains an utterance ratio for each speaker. This is obtained by dividing the sum of the lengths of the utterance intervals of a certain speaker by the sum of the lengths of the entire utterance intervals. The utterance ratio r _n is expressed as follows. N is the number of speakers, n = 1, ..., T n is the sum of the length of the speech segment of speaker n as N, T is the sum of the overall length of the speech period.

Next, the central speaker degree is obtained from the utterance ratio for each speaker (step S14). This is obtained by dividing the utterance ratio for each speaker by the maximum value of the utterance ratio. The central speaker degree c _n is expressed as follows.

  Finally, the central speaker degree speaker bias degree calculation unit 14 obtains the speaker bias degree (step S15). This can be obtained by scaling a value obtained by subtracting a ratio when the participants speak on average from a maximum value of the utterance ratio from 0 to 1. Speaker bias degree B is expressed as follows.

  The central speaker degree represents the utterance ratio for each speaker, where the speaker who has spoken the longest in a certain dialogue is 1. The degree of speaker bias represents the utterance ratio in the entire utterance section of the speaker who spoke the longest in a certain dialogue. When the speaker bias degree is 0, it means that all the speakers who participated in the dialogue spoke equally. When the speaker bias is 1, it means that one speaker has spoken from the beginning. FIG. 8 shows an example of the central speaker degree and the speaker bias degree.

<< Dialogue severity estimation feature calculator 15 (Fig. 3) >>
The dialogue severity estimation feature calculation unit 15 uses the time series of the fundamental frequencies of all the speakers input and the entire speech utterances input, the average fundamental frequency time change, the average fundamental frequency acceleration, and the ratio of non-speech intervals Is calculated (step S16). The calculated average fundamental frequency time change, average fundamental frequency acceleration, and ratio of the non-speech interval are output to the dialogue severity calculator 16.

  The dialogue severity estimation feature calculation unit 15 performs feature extraction for dialogue severity estimation at a later stage. First, the time change and acceleration of the fundamental frequency are obtained from the time series of the fundamental frequency for each speaker. Since the fundamental frequency is given in discrete time, the first-order difference is used for calculating the time change, and the second-order difference is used for calculating the acceleration. The absolute values of these temporal changes and accelerations are averaged over all utterance sections and all speakers, and are used as values representing the magnitude of prosodic change in the entire dialogue. The former is called average fundamental frequency time change, and the latter is called average fundamental frequency acceleration.

  Of the entire utterance section, the start time of the first utterance is the dialog start time, and the end time of the last utterance is the dialog end time. Among the sections from the dialog start time to the dialog end time, a section where no one utters is defined as a non-speaking section. The ratio of the total length of the non-speech section to the length from the conversation start time to the conversation end time is defined as the ratio of the non-speech section. FIG. 9 shows an example of the entire utterance interval and non-utterance interval in the dialogue.

<< Dialog severity calculator 16 (Fig. 3) >>
The dialogue severity calculator 16 uses the inputted average fundamental frequency time change, the inputted average fundamental frequency acceleration, the inputted ratio of the non-speech interval, and the inputted regression coefficient for estimating the dialogue severity, The dialogue severity is calculated (step S17).

  In general, the stricter the dialogue (such as parliament), the smaller the fluctuation of the fundamental frequency and the longer the non-speech interval. The dialogue severity is a scale expressing the above, and takes a value from 1 to 0. If the dialogue severity is 1, it represents a strict dialogue, and if it is 0, it represents a less strict dialogue (such as free discussion).

  The calculation of the dialogue severity can be realized by threshold processing. For example, when the average fundamental frequency time variation and the average fundamental frequency acceleration are smaller than a certain value and the non-speech interval is larger than another certain value, the dialogue severity is set to 1. Of course, the dialogue severity may be calculated by a statistical regression model such as logistic regression. However, when a statistical regression model is applied, processing for normalizing the output value from 0 to 1 is added. In addition, when estimating the severity of dialogue using a statistical regression model, it is necessary to learn the regression coefficient in advance. The prior learning method of the regression coefficient will be described later.

<Voice recognition unit 2 (Fig. 1)>
The speech recognition unit 2 outputs a speech recognition result text using the input speech for each speaker (step S2). The speech recognition result text is output to the text analysis unit 4 and the sentence end symbol giving unit 6.

  The speech recognition result text is generated, for example, by applying speech recognition to the speech for each speaker and converting the speech waveform into characters.

<Acoustic feature extraction unit 3 (FIG. 1)>
The acoustic feature extraction unit 3 extracts an acoustic feature using the input voice for each speaker (step S3). The extracted acoustic features are output to the sentence ending symbol estimation unit 5.

  The acoustic feature is at least one of a fundamental frequency, a short time signal power, a speech spectrum envelope, and a length between.

  The acoustic feature extraction unit 3 extracts a fundamental frequency, a short-time signal power, and a speech spectrum envelope (MFCC) for speech at each time. Also, the length between utterances is extracted using utterance interval detection. The interval length refers to a time “a short period included in the utterance interval such as breathing” in the utterance interval detection unit 11. When a human gives semantic information to an utterance, the basic frequency of the utterance and the short-time power are often changed, but it is known that the change also appears in the speech spectrum envelope. For example, a difference appears in the speech spectrum envelope between when the voice is relaxed and when the voice is nervous. In addition, the information between them is a large criterion for judging whether or not the end of the sentence. From the above, for example, these four types of acoustic features are used in the case of sentence ending symbol estimation.

<Text analysis unit 4 (Fig. 1)>
The text analysis unit 4 obtains language features using the input speech recognition result text (step S4). The obtained language feature is output to the sentence ending symbol estimation unit 5.

  The language feature is at least one of a word, a part of speech, and a dependency structure. For example, words, parts of speech, and dependency structures are all language features.

  The text analysis unit 4 divides the text of the speech recognition result for each word using a morphological analyzer, and obtains a part of speech for each word. It is also possible to create a chain of all three words and a chain of three parts of speech included in the speech recognition result, and use this as a linguistic feature of the words and parts of speech. Also, the entire text is parsed to obtain a dependency structure for each word, which may also be a language feature. It is assumed that time information is assigned to each word and part of speech, and temporal correspondence with the acoustic feature is taken.

<End-of-sentence estimation unit 5 (Fig. 1)>
The sentence ending symbol estimation unit 5 performs sentence ending symbol assignment determination for each word boundary using the input acoustic feature, the inputted language feature, and the inputted dialogue situation feature (step S5). The sentence ending symbol assignment determination for each word boundary is output to the sentence ending symbol assignment unit 6.

  The sentence ending symbol estimation unit 5 estimates the optimum sentence ending symbol for each word boundary by using the sentence ending symbol assignment norm selected based on the conversation situation feature. The end-of-sentence adding standard is selected based on the dialog situation feature. The sentence ending symbol provision norm is a sentence ending symbol provision standard or a sentence ending symbol provision model. As the sentence end symbol assignment standard, for example, a rule-based method for estimating an optimum sentence end code by threshold processing of acoustic features and language features is used. The sentence ending symbol assignment model represents a model and a classifier that represent the appearance probability of a sentence ending symbol learned by machine learning such as a conditional random field or a support vector machine.

  It is known that there is a strong relationship between the end-of-sentence symbol and the acoustic and language features. For example, when a question mark is given, there is a tendency that the fundamental frequency increases and the appearance of particles / case particles increases. However, the relationship between the sentence ending symbol and the acoustic or language features changes depending on the conversation situation. For example, in a strict meeting, there is little change in the fundamental frequency in scenes other than questions, so the question mark should be estimated mainly using the fundamental frequency. However, in the case of a non-strict meeting, the fundamental frequency changes in various situations, so the question mark should be estimated mainly using language features. For the purpose of enabling automatic response to the above changes, we introduce automatic estimation of dialogue situation features and selection of sentence ending symbol estimation criteria using dialogue situation features.

  In many cases, a sentence ending symbol is determined based on a plurality of factors of acoustic features and language features. For example, a question mark is given when an increase in fundamental frequency and the appearance of a particle indicating a question occur simultaneously. For this reason, there is a possibility that false detections frequently occur by simply thresholding acoustic features and language features. Therefore, it is also effective to perform sentence ending symbol estimation using a sentence ending symbol estimation model learned by machine learning, which can take into account complex factors.

  Note that when an optimal sentence ending symbol is estimated using a sentence ending symbol assignment model, prior learning of the model is required. The outline of the pre-learning at this time will be described later.

  For example, there are six types of sentence ending symbols to be selected: question mark “?”, Exclamation mark “!”, Three-point reader “…”, laugh symbol “(laugh)”, punctuation mark “.”, And punctuation mark “,”. On the basis of the end-of-sentence symbol assignment norm, it is classified for each word boundary which of the six types of end-of-sentence symbol and the seven types of “don't give anything” is appropriate.

  As described above, the sentence ending symbol estimation unit 5 selects a sentence ending symbol provision norm corresponding to the conversation situation from a plurality of sentence ending symbol provision norms based on the conversation situation feature calculated by the conversation situation feature calculation device 1. The end-of-sentence symbol for the speech recognition result text of the dialog is estimated using the selected end-of-sentence adding standard, the acoustic feature and the language feature of the dialog.

<End-of-sentence adding unit 6 (FIG. 1)>
The sentence ending symbol assigning unit 6 generates a speech recognition result with a sentence ending symbol using the sentence ending symbol assignment determination for each input word boundary and the input speech recognition result text (step S6).

  Specifically, the sentence ending symbol assigning unit 6 generates a speech recognition result with a sentence ending symbol by assigning a sentence ending symbol to the speech recognition result text. At that time, sentence end assignment assignment determination for each word boundary is used as a reference for sentence end assignment.

<End-of-sentence assignment model generator 7>
A function for generating a sentence ending symbol assignment model by prior learning may be provided in the sentence ending symbol estimation device.

  As shown in FIGS. 10 and 11, the sentence ending symbol assigning model generation unit 7 includes a dialog situation feature calculation device 71, a speech database division unit 72, a sentence ending symbol correct label creation unit 73, a speech recognition unit 74, and an acoustic feature extraction unit 75. For example, a text analysis unit 76 and a sentence ending symbol addition model generation unit 77 are provided.

  The pre-learning by the sentence ending symbol assigning model generation unit 7 uses a speech database in which speech for each speaker is recorded and a transcript with a sentence ending symbol corresponding to each speech data. This speech database may be a speech database used for pre-learning of regression coefficients for estimating dialogue severity described later. Transcripts with end-of-sentence text are text data that a person listens to and creates, and each word boundary is given time information that can be associated with the voice in the voice database for each speaker. To do.

  The dialogue situation feature calculation device 71 calculates the dialogue situation feature in the same manner as the dialogue situation feature calculation device 1. The calculated dialog situation feature is output to the voice database dividing unit 72.

  The voice database dividing unit 72 uses the input voice database for each speaker, the input transcript with the sentence ending symbol, and the input dialog status feature, and the dialog status feature threshold processing of the dialog status feature makes it possible to generate a voice with a close dialog status feature. Output the database. For example, threshold values such as a central speaker degree of 0.7 or more, a speaker bias degree of 0.5 or more, and a dialogue severity of 0.5 or more are set, and speech satisfying them is set as one database. In the case of the above example, the intention is to divide from the database the speech of the “speaker” having a high central speaker level of the “speaker” which is a place of dialogue having a high level of dialog severity and speaker bias. Each database divided based on the conversation situation feature can be regarded as a set of voices having similar utterance contents and utterance methods. It is assumed that the transcript with a sentence ending symbol is also divided in such a way that it can correspond to the speech included in each database.

  In this way, speech databases with similar dialog status features and corresponding transcripts with end-of-sentence symbols are grouped together. The following processing is performed for each of the speech database included in each group and the corresponding transcript with a sentence ending symbol, and a “sentence ending symbol assignment model in a certain dialog situation” of each group is generated.

  The sentence ending symbol correct label creating unit 73 generates a sentence ending symbol correct label using the input transcript with the sentence ending symbol. The generated sentence ending symbol correct label is output to the sentence ending symbol assignment model generation unit 77.

  The correct ending symbol label indicates the type of ending symbol that falls on a word boundary. For example, a question mark “?”, An exclamation point “!”, A three-point reader “…”, a laugh symbol “(laugh)”, and a phrase “.”. It is assumed that any of the seven types of reading marks “,” and nothing is given.

  Specifically, the sentence ending symbol correct label creating unit 73 performs morphological analysis of the transcription with the sentence ending symbol and divides it into words. After that, the end-of-sentence symbol is found at the word boundary for all words excluding the end-of-sentence symbol, and the correct end-sentence label is used.

  The processes of the speech recognition unit 74, the acoustic feature extraction unit 75, and the text analysis unit 76 are the same as the processes of the speech recognition unit 2, the acoustic feature extraction unit 3, and the text analysis unit 4, respectively. . The acoustic features extracted by the acoustic feature extraction unit 3 and the language features obtained by the text analysis unit 4 are output to the sentence ending symbol addition model generation unit 77.

  The sentence ending symbol assignment model generation unit 77 generates a sentence ending symbol assignment model in a certain dialogue situation using the input acoustic features, the input language features, and the input sentence ending symbol correct labels.

  The sentence ending symbol adding model generating unit 77 learns the sentence ending symbol assigning model by machine learning using the acoustic features and language features of each speech included in the speech database having similar conversation situation features as input data and the sentence ending symbol correct answer label as teacher data. To do. As the machine learning method, it is assumed that a conditional random field or a support vector machine is used, but any machine learning method may be used as long as it can solve the classification problem.

  In this way, the sentence ending symbol assignment model generation unit 77 estimates each conversation situation based on the conversation situation feature, and uses the acoustic feature, language feature, and sentence ending symbol correct label of the dialogue as teacher data for each estimated conversation. By learning, a plurality of sentence ending symbol assignment models, which are a plurality of sentence ending symbol assignment norms, are generated.

<Regression coefficient learning unit 17 for dialogue severity estimation>
The regression coefficient learning unit 17 for estimating the dialogue severity may be provided in the dialogue situation feature calculation device and the sentence ending symbol estimation device.

  An example of the regression coefficient learning unit 17 is shown in FIG. The regression coefficient learning unit 17 includes, for example, an utterance section detection unit 171, a fundamental frequency extraction unit 172, an entire utterance section detection unit 173, a dialogue severity estimation feature calculation unit 174, and a regression analysis unit 175.

  For pre-learning, prepare a voice database that contains various dialogues. However, it is assumed that the individual recording of speech for each speaker and the assignment of the correctness string for dialogue severity are performed in each dialogue included in the database. The dialogue strictness correct answer label is manually assigned, and 1 is given if a person feels strictness after hearing the dialogue, and 0 if not. It is assumed that the dialogue severity correct answer label is given in units of dialogue. Regression analysis is performed using all dialogues included in the speech database and all dialogue severity correct answer labels, and a regression coefficient for dialogue severity estimation is obtained.

  The processing of the utterance interval detection unit 171, the fundamental frequency extraction unit 172, the entire utterance interval detection unit 173, and the dialogue severity estimation feature calculation unit 174 is performed in the utterance interval detection unit 11, the fundamental frequency extraction unit 12, and the entire utterance interval detection unit 13, respectively. And since it is the same as the process of the dialog strictness estimation feature calculation unit 15, a duplicate description thereof will be omitted. Here, only the regression analysis unit 175 will be described.

  The regression analysis unit 175 calculates a regression coefficient for estimating the dialogue severity using the input average fundamental frequency change amount, the ratio of the non-speech interval, and the dialogue severity correct answer label.

  Specifically, the regression analysis unit 175 performs pre-learning of regression coefficients for estimating dialogue severity, for example, as follows. The regression coefficient is obtained by applying regression analysis with the average fundamental frequency time change, the average fundamental frequency acceleration and the ratio of the non-speech interval obtained from the speech for each speaker as explanatory variables and the correct label as a dependent variable. In the regression analysis, it is necessary to use the same regression model (logistic regression model or the like) as the dialogue severity calculation unit 16.

[Variations]
The processes described in the apparatus and method are not only executed in chronological order according to the order of description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary.

  Further, when each process in each device is realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, each process is realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  Each processing means may be configured by executing a predetermined program on a computer, or at least a part of these processing contents may be realized by hardware.

  Needless to say, other modifications are possible without departing from the spirit of the present invention.

  By adding a sentence ending symbol corresponding to the meaning of the utterance, it is possible to prevent the misunderstanding of the meaning and to create a minutes that can understand the atmosphere of the place. Further, as a secondary effect, it is possible to search for a specific utterance using a sentence end code (for example, to search only a question part), and the efficiency of making minutes is improved.

DESCRIPTION OF SYMBOLS 1 Dialogue state characteristic calculation apparatus 11 Speech area detection part 12 Fundamental frequency extraction part 13 Whole speech area detection part 14 Central speaker degree Speaker bias degree calculation part 15 Dialogue severity estimation characteristic calculation part 16 Dialogue severity calculation part 17 Regression coefficient Learning section 171 Speaking section detection section 172 Fundamental frequency extraction section 173 Whole utterance section detection section 174 Dialogue severity estimation feature calculation section 175 Regression analysis section 2 Speech recognition section 3 Acoustic feature extraction section 4 Text analysis section 5 End of sentence symbol estimation section 6 End of sentence Symbol assignment unit 7 End-of-sentence symbol addition model generation unit 71 Dialogue state feature calculation device 72 Speech database division unit 73 End-of-word symbol correct label creation unit 74 Speech recognition unit 75 Acoustic feature extraction unit 76 Text analysis unit 77 End-of-sentence symbol addition model generation unit

Claims (5)

The central speaker degree of each speaker of a dialog conducted by a plurality of speakers is used as an index indicating the proportion of each speaker's utterance in the dialog, and the speaker bias of the dialog is the speaker in the dialog. The degree of utterance length of the utterance, and the degree of dialogue severity of the dialogue as the index of strictness of the speaker's tone during the dialogue, the central speaker degree, the speaker bias degree and the dialogue A dialog situation feature calculation device for calculating a dialog situation feature of the dialogue, using at least one of the strictness as a dialogue situation feature;
Based on the dialog situation feature calculated by the above dialog situation feature calculation device, a sentence ending symbol assignment standard corresponding to the dialogue is selected from a plurality of sentence ending symbol assignment standards, and the selected sentence ending symbol provision standard and the sound of the dialogue are selected. A sentence ending symbol estimation unit for estimating a sentence ending symbol for a text representing the utterance content of the dialogue using the feature and the linguistic feature;
End-of-sentence estimation device. In the sentence ending symbol estimation apparatus of Claim 1,
The situation of each dialogue is estimated based on the dialogue situation feature, and for each estimated dialogue, the acoustic features, language features, and sentence ending symbol correct labels of the dialogue are learned as teacher data. A sentence ending symbol addition model generating unit for generating a plurality of sentence ending symbol assignment models;
End-of-sentence estimation device. The central speaker degree of each speaker of a dialog conducted by a plurality of speakers is used as an index indicating the proportion of each speaker's utterance in the dialog, and the speaker bias of the dialog is the speaker in the dialog. The degree of utterance length of the utterance, and the degree of dialogue severity of the dialogue as the index of strictness of the speaker's tone during the dialogue, the central speaker degree, the speaker bias degree and the dialogue With at least one of the strictness as a dialogue situation feature,
A dialog situation feature calculating step for calculating a dialog situation feature of the dialog;
The sentence ending symbol estimation unit selects a sentence ending symbol assignment criterion corresponding to the dialogue from a plurality of sentence ending symbol provision criteria based on the conversation situation feature calculated by the dialog situation feature calculation device, and assigns the selected sentence ending symbol An end-of-sentence estimation step for estimating an end-of-sentence for a text representing the utterance content of the dialogue using the norm, the acoustic features and language features of the dialogue;
Sentence ending symbol estimation method. The sentence ending symbol estimation method according to claim 3 ,
The sentence ending symbol assignment model generation unit estimates the situation of each dialogue based on the dialogue situation feature, and learns the acoustic feature, language feature and sentence ending symbol correct label of the dialogue as teacher data for each estimated dialogue, A sentence ending symbol giving model generating step for generating a plurality of sentence ending symbol giving models as the plurality of sentence ending symbol giving norms;
End-of-sentence estimation method.   The program for functioning a computer as each part of the sentence ending symbol estimation apparatus of Claim 1 or 2.

Images