JP7611744B2 - Signal processing device and program - Google Patents

Description

The present invention relates to a signal processing device and a program.

To realize text-to-speech synthesis, which uses statistical information to synthesize speech for any text, it is necessary to prepare a large amount of speech signal data, the text of the speech, reading, accent, and other information as a preliminary step. Correctly assigning reading and accent information to a large amount of speech signal data requires the work of an expert who can accurately understand and distinguish accents, such as an announcer, which is costly. Therefore, there is a demand for technology that can determine the accent type for speech signal data, and research is being conducted.

The technology described in Non-Patent Document 1 is intended to identify which accent type given speech signal data corresponds to. Specifically, this technology determines in advance the pitch that represents each mora (consonant + vowel or vowel, geminated consonant, glottal stop) within an accent phrase. It also constructs a (number of moras - 1)-dimensional feature vector from the difference in values between adjacent moras. It then uses the feature vectors of accent phrases with the same accent type to determine a normal distribution of (number of moras - 1)-dimensional feature vectors for each accent type. It then uses these modeled normal distributions to identify which accent type the (number of moras - 1)-dimensional feature vector calculated for the accent phrase to be determined corresponds to.

Patent Document 1 also describes a technology for determining the type of accent for a speech signal. The technology in Patent Document 1 aims to support text-to-speech synthesis. To that end, this technology divides the speech corresponding to a target phrase into multiple accent phrases. Then, for the target accent phrase, the slope of the time change in pitch and the final pitch are found on a mora-by-mora basis. Then, based on that information, this technology calculates the values of four types of evaluation functions, high, low, up, and down, on a mora-by-mora basis. The type of accent is determined by finding the combination that maximizes the sum of the evaluation function values for all the moras that make up the accent phrase.

Patent No. 4129989

Toshinori Ishii, Nobuaki Minematsu, Keiichi Hirose, Accent type determination of continuous Japanese speech considering pitch perception, Technical Report of the Institute of Electronics, Information and Communication Engineers, SP, Speech, 101(270), 23-30, 23 August 2001, The Institute of Electronics, Information and Communication Engineers.

However, the technology described in Non-Patent Document 1 uses the difference in pitch between moras, and is therefore only able to determine the accent type based on the difference values of (the number of moras minus 1). Furthermore, the technology described in Patent Document 1 classifies accent types as a combination of the calculation results of the evaluation function.
All of these conventional methods are premised on pitch extraction, and there is a possibility that pitch extraction errors may affect the accuracy of recognition. Furthermore, all of these techniques are based on analysis of accent phrases, and do not take into account pitch fluctuations in the entire utterance or in phrases. Therefore, there is a problem in that the accent cannot be estimated with high accuracy. Thus, the conventional techniques have a problem in that the accent cannot be estimated with high accuracy from a speech signal.

The present invention was developed based on the above problem recognition, and aims to provide a signal processing device and a program that can estimate accent labels with high accuracy.

[1] In order to solve the above problem, a signal processing device according to one aspect of the present invention includes an acoustic analysis unit that determines, for each predetermined unit of text, acoustic features on a frame-by-frame basis based on a speech signal corresponding to the predetermined unit of text, a language analysis unit that determines linguistic features of the text on a frame-by-frame basis based on the predetermined unit of text, and a label estimation unit that inputs the frame-by-frame acoustic features and the linguistic features into a pre-trained statistical model, and estimates an accent label corresponding to the text as an output from the statistical model.

[2] In one aspect of the present invention, the signal processing device further includes a training acoustic analysis unit that determines, for each training text of a predetermined unit, training acoustic features on a frame-by-frame basis based on a training speech signal corresponding to the training text of the predetermined unit, a training language analysis unit that determines training language features on a frame-by-frame basis of the training text of the predetermined unit, a correct label extraction unit that determines a correct label of an accent corresponding to the text based on the training language features, and a model learning unit that inputs the frame-by-frame training acoustic features and the training language features to a statistical model to obtain a training estimated label of an accent corresponding to the training text as an output from the statistical model, and learns the statistical model based on an error between the training estimated label and the correct label.

[3] A signal processing device according to an aspect of the present invention includes an acoustic analysis unit that determines, for each predetermined unit of text, acoustic features on a frame-by-frame basis based on a speech signal corresponding to the predetermined unit of text; a language analysis unit that determines linguistic features of the text on a frame-by-frame basis based on the predetermined unit of text; a correct label extraction unit that determines a correct label for an accent corresponding to the text based on the linguistic features; and a model learning unit that inputs the frame-by-frame acoustic features and the linguistic features into a statistical model to obtain an estimated label for an accent corresponding to the text as an output from the statistical model, and learns the statistical model based on an error between the estimated label and the correct label.

[4] In one aspect of the present invention, in the signal processing device described above, the predetermined unit is a unit of accent phrase.

[5] In one aspect of the present invention, in the above-mentioned signal processing device, the linguistic features are a breath paragraph position in an utterance, an accent phrase position in the breath paragraph, a mora position in the accent phrase, and a frame position in the mora.

[6] In one aspect of the present invention, in the signal processing device, the statistical model is realized using a deep neural network.

[7] In another aspect of the present invention, in the signal processing device, the acoustic feature is a mel spectrogram.

[8] Another aspect of the present invention is a program for causing a computer to function as a signal processing device according to any one of [1] to [7] above.

According to the present invention, by using a statistical model trained on frame-by-frame acoustic features and linguistic features, it is possible to accurately estimate accent, taking into account the influence of speech based on linguistic features.

1 is a block diagram showing a schematic functional configuration of an accent estimation device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a detailed functional configuration of a pre-processing unit (32) of the accent estimation device according to the embodiment. FIG. 13 is a schematic diagram (1/3) showing a format of a full context label used by the accent estimation device according to the embodiment. FIG. 2 is a schematic diagram (2/3) showing a format of a full context label used by the accent estimation device according to the embodiment. FIG. 3 is a schematic diagram (3/3) showing a format of a full context label used by the accent estimation device according to the embodiment. 13 is a schematic diagram (1/2) showing an example of a full context label extracted by a language analysis unit in a preprocessing unit according to the embodiment performing language analysis processing on a text. FIG. 13 is a schematic diagram (2/2) showing an example of a full context label extracted by a language analysis unit in a preprocessing unit according to the embodiment performing language analysis processing on a text. FIG. 13 is a flowchart showing a processing procedure for a first correct label extraction unit and a second correct label extraction unit according to the embodiment to obtain a correct label for each accent phrase. 13 is a flowchart showing a procedure of a process in which a second correct label extraction unit according to the embodiment creates a tone for each accent phrase. 13 is a schematic diagram showing rules for converting the tones of an accent phrase into numerical values of correct labels in the accent estimation device of the embodiment. FIG. 11 is a schematic diagram showing a process flow when a statistical model is learned by a model learning unit of the embodiment. FIG. 11 is a schematic diagram showing a process flow for estimating a label by a label estimation unit according to the embodiment. FIG. FIG. 2 is a block diagram showing an example of an internal configuration for realizing functions of the accent estimation device according to the embodiment.

Next, an embodiment of the present invention will be described with reference to the drawings. First, some concepts that this embodiment deals with will be described. An utterance is speech that corresponds to a set of sentences. A breath group is a unit of speech that is separated by a pause during speech. A unit in a written sentence that is separated by a comma corresponds approximately to a breath group. An accent phrase is usually a unit of a Japanese word, or a word plus an attached word. A mora is a linguistic element consisting of one consonant and one vowel, or one vowel, mora, or mora. When a mora does not contain a yōon or a small vowel that is mainly used in foreign words, it is represented by one kana (such as katakana). When a mora contains a yōon or a small vowel that is mainly used in foreign words, it is represented by one kana plus a kana that represents the yōon. For example, "chi," "o," "sha," and "fi" are each units of a mora. A phoneme is an even smaller unit than a mora, consisting of only consonants or only vowels.

In this embodiment, in pre-learning, the accent estimation device divides the text of each utterance into accent phrase units and obtains acoustic features for each frame within an accent phrase from the speech signal. The accent estimation device also obtains linguistic features and correct accent labels for each frame within an accent phrase based on the text. The accent estimation device also uses the acoustic features and linguistic features for each frame within an accent phrase as input data and learns a statistical model so as to minimize the error between the estimated value of the statistical model and the correct label.
In label estimation processing, the accent estimation device divides the text of each utterance into accent phrase units and obtains acoustic features for each frame within the accent phrase from the speech signal. The accent estimation device also obtains linguistic features for each frame within the accent phrase based on the text. The accent estimation device then estimates accent labels by inputting the acoustic features and linguistic features for each frame within the accent phrase into a trained statistical model.

FIG. 1 is a block diagram showing a schematic functional configuration of an accent estimation device according to this embodiment. As shown in the figure, the accent estimation device 1 includes a model storage unit 11, an estimation speech corpus storage unit 21, a preprocessing unit 22, a label estimation unit 23, a training speech corpus storage unit 31, a preprocessing unit 32, and a model learning unit 33. These functions of the accent estimation device 1 can be realized, for example, by a computer and a program. Furthermore, each function has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used as necessary. Furthermore, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

The accent estimation device 1 shown in the figure has a function of constructing a statistical model (accent estimation model) by machine learning based on a training speech corpus. The accent estimation device 1 writes the constructed statistical model to a model storage unit 11. The accent estimation device also has a function of estimating and outputting a label representing the accent of unknown text using the above statistical model. The unknown text is obtained, for example, from the estimation speech corpus.

Of the functions of the accent estimation device 1, only the function of estimating a label representing an accent of unknown text based on a pre-constructed statistical model may be implemented as the accent estimation device 2. In this case, the accent estimation device 2 is configured to include the estimation speech corpus storage unit 21, the pre-processing unit 22, and the label estimation unit 23 among the functional units listed above. The accent estimation device 2 can perform label estimation processing by referring to the statistical model stored in the model storage unit 11. Learning of the statistical model at this time has been completed. Of the functions of the accent estimation device 1, only the function of constructing a statistical model may be implemented as the accent estimation model learning device 3. In this case, the accent estimation model learning device 3 is configured to include the training speech corpus storage unit 31, the pre-processing unit 32, and the model learning unit 33 among the functional units listed above. The accent estimation model learning device 3 writes information on the statistical model obtained by learning to the model storage unit 11. In other words, the accent estimation model learning device 3 can learn the statistical model. By using this statistical model, it becomes possible to estimate the accent label corresponding to the input data.

Each of the accent estimation device 1, the accent estimation device 2, and the accent estimation model learning device 3 may be called a "signal processing device." The speech signal, text, sutra features, and language features described below are each a "signal" that is processed by the signal processing device.

The model storage unit 11 stores information about the statistical model. The statistical model is a model that can be machine-learned. The statistical model is realized, for example, by using a neural network. The statistical model may be realized, in particular, by using a deep neural network. However, the statistical model may be realized by a method other than a neural network. In either case, the statistical model outputs data (output data) that statistically corresponds to input data (input data). Specifically, the model storage unit 11 stores values of internal parameters of the statistical model. Note that the machine learning method of the statistical model itself can be implemented using existing technology.

The training speech corpus storage unit 31 stores a training speech corpus used to construct a statistical model. The training speech corpus contains a large number of pairs of speech signals and text. The speech signals are sampled, for example, at a sampling frequency of 16,000 Hertz (Hz) and a conversion bit rate of 16 bits (bit). However, the sampling frequency and conversion bit rate do not need to be limited to the values exemplified here. The text contained in the training speech corpus corresponds to the speech signals and is text that has been created manually or appropriately using speech recognition processing, etc.

The preprocessing unit 32 processes pairs of speech signals and texts read from the training speech corpus storage unit 31, and outputs pairs of input data and correct labels for the training process. The preprocessing unit 32 divides the input data into accent phrase units, and outputs input data and correct labels for each accent phrase. The input data includes acoustic features and linguistic features on a frame-by-frame basis for each accent phrase. The preprocessing unit 32 passes the pairs of input data and correct labels to the model training unit 33. The detailed functional configuration for the processing of the preprocessing unit 32 will be described later with reference to FIG. 2.

The model learning unit 33 performs machine learning processing using pairs of input data and correct labels passed from the preprocessing unit 32. Specifically, the model learning unit 33 uses a statistical model to estimate labels based on the input data, calculates the difference (loss) between the estimated labels (also called "estimated labels for learning") and the correct labels, and adjusts the internal parameters of the statistical model to minimize the loss. The model learning unit 33 repeatedly performs processing to adjust (update) the internal parameters for many pairs of input data and correct labels as learning data. The machine learning method itself used by the model learning unit 33 belongs to existing technology. For example, when the statistical model is realized by a neural network, the model learning unit 33 uses the backpropagation method to learn the statistical model.

In other words, the model learning unit 33 inputs frame-by-frame acoustic features (training acoustic features) and language features (training language features) into a statistical model, obtains an estimated label (training estimated label) of an accent corresponding to text (training text) as output from the statistical model, and learns the statistical model based on the error (loss) between the estimated label (training estimated label) and the ground truth label. In other words, the model learning unit 33 adjusts (updates) the internal parameters of the statistical model in the direction that reduces the error.

The estimation speech corpus storage unit 21 stores an estimation speech corpus that includes text that is the subject of label estimation. The estimation speech corpus includes pairs of text and speech signals.

The preprocessing unit 22 performs preprocessing on the text read from the estimation speech corpus storage unit 21 and passes the result to the label estimation unit 23 as input data.

The label estimation unit 23 uses a statistical model to estimate an accent label corresponding to the input data passed from the preprocessing unit 22. The label estimation unit 23 estimates the label by referring to the model storage unit 11. At that point, the model storage unit 11 has already stored information about the trained statistical model. Specifically, the label estimation unit 23 inputs frame-by-frame acoustic features and linguistic features into a pre-trained statistical model, thereby estimating an accent label corresponding to the text as an output from the statistical model.

Fig. 2 is a block diagram showing a more detailed functional configuration of the preprocessing unit 32 of the accent estimation device 1. As shown in the figure, the preprocessing unit 32 includes an acoustic analysis unit 351, a phoneme segmentation unit 352, a central phoneme extraction unit 353, a language analysis unit 354, a mora unit integration unit 361, an accent phrase unit integration unit 362, an accent phrase division unit 371, a first language feature extraction unit 381, a second language feature extraction unit 382, a first correct label extraction unit 386, a second correct label extraction unit 387, and an acoustic feature language feature integration unit 391.

The preprocessing unit 32 performs preprocessing for learning. The preprocessing unit 32 may be called a "learning preprocessing unit." The acoustic analysis unit 351 included in the preprocessing unit 32 may be called a "learning acoustic analysis unit." The audio signal that the acoustic analysis unit 351 analyzes may be called a "learning audio signal." The acoustic features determined by the acoustic analysis unit 351 may be called "learning acoustic features." The language analysis unit 354 included in the preprocessing unit 32 may be called a "learning language analysis unit." The text that the language analysis unit 354 analyzes may be called "learning text." The language features determined by the language analysis unit 354 may be called "learning language features."

The acoustic analysis unit 351 determines frame-by-frame acoustic features for each predetermined unit of text based on the speech signal corresponding to the predetermined unit of text. In other words, the training acoustic analysis unit determines frame-by-frame training acoustic features for each predetermined unit of training text based on the training speech signal corresponding to the predetermined unit of training text. Here, "predetermined unit of text" refers to, for example, text in units of accent phrases.

The phoneme segmentation unit 352 performs phoneme segmentation on the speech signal. In other words, the phoneme segmentation unit 352 identifies intervals (start and end positions) for each phoneme in the speech signal.

The central phoneme extraction unit 353 extracts central phonemes based on the language analysis results (full context labels).

The language analysis unit 354 determines frame-by-frame language features of the text based on the above-mentioned predetermined unit of text. In other words, based on the training text of the predetermined unit, it determines frame-by-frame training language features of the training text. Examples of language features determined by the language analysis unit 354 are the breath paragraph position in the utterance, the accent phrase position within the breath paragraph, the mora position within the accent phrase, and the frame position within the mora.

The mora-based integration unit 361 integrates the frame-based acoustic feature data output by the acoustic analysis unit 351 into mora-based acoustic feature data.

The accent phrase unit integration unit 362 integrates the acoustic feature data into acoustic feature data for each accent phrase.

The accent phrase division unit 371 divides the full context label, which is the result of the above-mentioned language analysis process, into accent phrase units.

The first language feature extraction unit 381 extracts language features on a frame-by-frame basis within a mora.

The second language feature extraction unit 382 extracts language features on a frame-by-frame basis within an accent phrase.

The first correct label extraction unit 386 extracts correct labels on a frame-by-frame basis within a mora.

The second correct label extraction unit 387 extracts correct labels on a frame-by-frame basis within an accent phrase.

The first correct label extraction unit 386 and the second correct label extraction unit 387 may be collectively referred to simply as the "correct label extraction unit." In other words, the correct label extraction unit finds a correct label for the accent corresponding to the text based on the language features (training language features).

The acoustic feature/language feature integration unit 391 integrates the acoustic feature data output by the accent phrase unit integration unit 362 and the language feature data output by the second language feature extraction unit 382. The data integrated by the acoustic feature/language feature integration unit 391 becomes input data to the statistical model.

Figures 3, 4 and 5 are schematic diagrams showing the format of full context labels (context-dependent phoneme labels) used by the accent estimation device 1. A full context label is obtained as a result of analyzing a text, and is a phoneme label that depends on the context of the text. Figures 3 to 5 show one data structure of a full context label. Note that the method in which the accent estimation device 1 uses the full context label during processing will be explained later. As shown in Figures 3, 4 and 5, a full context label includes the data listed below.

The following _p1 to _p5 are phoneme information.
_p1 : Identification information of the phoneme preceding the preceding phoneme _p2 : Identification information of the preceding phoneme _p3 : Identification information of the current phoneme (central phonemes from _p1 to _p5 ) _p4 : Identification information of the next phoneme _p5 : Identification information of the phoneme following the next phoneme

The following _a1 to _a3 are accent information.
_a1 : Difference between accent type (mora position of accent nucleus) and position of current mora identification information _a2 : Position of current mora identification information in current accent phrase (forward)
a ₃ : Position of current mora identification information in the current accent phrase (backwards)

The following _b1 to _b3 , _c1 to _c3 , and _d1 to _d3 are part-of-speech information.
_b1 : Part of speech of the previous word _b2 : Conjugated form of the previous word _b3 : Conjugated type of the previous word

_c1 : Part of speech of the current word _c2 : Conjugation form of the current word _c3 : Conjugation type of the current word

_d1 : Part of speech of the next word _d2 : Conjugated form of the next word _d3 : Conjugated type of the next word

The following _e1 to _e5 , _f1 to _f8 , and _g1 to _g5 are accent information.
_e1 : number of moras in the previous accent phrase _e2 : accent type in the previous accent phrase (mora position of accent nucleus)
_e3 : whether the previous accent phrase is an interrogative word or not; _e4 : undefined context; _e5 : whether a pause is inserted between the previous accent phrase and the current accent phrase or not;

_f1 : number of moras in the current accent phrase _f2 : accent type in the current accent phrase (mora position of the accent nucleus)
_f3 : Whether the current accent phrase is an interrogative word or not _f4 : Undefined context _f5 : Position of the current accent phrase identification information in the current breath paragraph in terms of accent phrase units (forward)
_f6 : Position of the current accent phrase identification information in the current breath paragraph in terms of accent phrase units (backwards)
_f7 : Position of the current accent phrase identification information in the current breath paragraph in mora units (forward)
_f8 : Position of the current accent phrase identification information in the current breath paragraph in mora units (backwards)

_g1 : number of moras in the next accent phrase _g2 : accent type in the next accent phrase (mora position of accent nucleus)
_g3 : Whether the next accent phrase is an interrogative word _g4 : Undefined context _g5 : Whether a pause is inserted between the next accent phrase and the current accent phrase

The following _h1 to _h2 , _i1 to _i8 , and _j1 to _j2 are breath phase information.
_h1 : number of accent phrases in the previous breath group _h2 : number of moras in the previous breath group

_i1 : number of accent phrases in the current breath group _i2 : number of moras in the current breath group _i3 : position of the breath group unit of the current breath group identification information (forward)
_i4 : Position of the current breath paragraph identification information in units of breath paragraphs (backward)
_i5 : Position of accent phrase unit of current breath paragraph identification information (forward)
_i6 : Position of accent phrase unit of current breath paragraph identification information (backward)
_i7 : Position of the current breath paragraph identification information in mora units (forward)
_i8 : Position of the current breath paragraph identification information in mora units (backwards)

_j1 : number of accent phrases in the next breath group _j2 : number of moras in the next breath group

The following _k1 to _k3 are total number information.
_k1 : Total number of breath paragraphs in this utterance _k2 : Total number of accent phrases in this utterance _k3 : Total number of moras in this utterance

6 and 7 are schematic diagrams showing examples of full context labels extracted by the language analysis unit 354 performing language analysis processing on text. This example is a full context label based on the text "Sunny, then cloudy." In this figure, one line of full context label data corresponds to one phoneme. However, one line of data is folded back halfway. FIG. 6 shows the first to eighth lines of data. FIG. 7 shows the ninth to fifteenth lines of data. The beginning of each line corresponds to the above-mentioned items _p1 to _p5 . Moreover, the part starting with "/A:" of each line corresponds to the above-mentioned items _a1 to _a3 . Moreover, the part starting with "/B:" of each line corresponds to the above-mentioned items _b1 to _b3 . Moreover, the part starting with "/C:" of each line corresponds to the above-mentioned items _c1 to _c3 . Moreover, the part starting with "/D:" of each line corresponds to the above-mentioned items _d1 to _d3 . Moreover, the portion beginning with "/E:" in each line corresponds to the above-mentioned items _e1 to _e5 . Moreover, the portion beginning with "/F:" in each line corresponds to the above-mentioned items _f1 to _f8 . Moreover, the portion beginning with "/G:" in each line corresponds to the above-mentioned items _g1 to _g5 . Moreover, the portion beginning with "/H:" in each line corresponds to the above-mentioned items _h1 to _h2 . Moreover, the portion beginning with "/I:" in each line corresponds to the above-mentioned items _i1 to _i8 . Moreover, the portion beginning with "/J:" in each line corresponds to the above-mentioned items _j1 to _j2 . Moreover, the portion beginning with "/K:" in each line corresponds to the above-mentioned items _k1 to _k3 .

Next, we will explain the pre-training process and the label estimation process in detail.

In pre-learning, the speech signal for each utterance and the text corresponding to that speech signal are used as learning data. In the pre-learning process, frame-by-frame data within an accent phrase is created from the data for each utterance, and a statistical model for estimating the correct accent label is trained using the frame-by-frame data within the accent phrase. Specifically, this is done as follows.

The preprocessing unit 32 processes pairs of speech signals and text read from the training speech corpus storage unit 31.

The acoustic analysis unit 351 performs acoustic analysis of the audio signal. That is, the acoustic analysis unit 351 performs analysis of the acoustic features for each frame while shifting a predetermined window width. Specifically, the acoustic analysis unit 351 extracts acoustic features for each frame, for example, in units of 5 milliseconds (ms). However, the length of the frame may be other than 5 milliseconds. As the acoustic features, a mel spectrogram is used. The mel spectrogram is information of an audio signal in the frequency domain. The mel spectrogram is a spectrogram in which the frequency is converted to the mel scale. The mel spectrogram itself is an acoustic feature by an existing method. As an example, the mel spectrogram can be extracted using a function provided at the following URL.
URL https://librosa.org/doc/latest/generated/librosa.feature.melspectrogram.html

In this embodiment, the acoustic analysis unit 351 extracts a mel spectrogram in the range of a minimum frequency of 80 Hz and a maximum frequency of 7600 Hz. The order is set to 80. The acoustic analysis unit 351 also standardizes the mel spectrogram for each utterance using the average value and standard deviation calculated for each dimension of the mel spectrogram in the sound section. The acoustic analysis unit 351 outputs this order 80 value as an acoustic feature for each frame. It is expected that standardization for each utterance as described above can absorb variations between utterances and between speakers.

The language analysis unit 354 performs language analysis processing on the text corresponding to the speech signal to obtain a full context label that depends on the context. The language analysis processing includes morphological analysis and syntactic analysis processing of the text (sentence). The format of the full context label is as described in Figs. 3, 4, and 5. An example of an extracted full context label is as described in Figs. 6 and 7. The full context label itself shown in these figures is also used in existing methods. The language analysis unit 354 extracts the full context label from the text by using, for example, the analysis technology of a speech synthesis processing toolkit provided at the following URL.
URL http://open-jtalk.sourceforge.net

As shown below, the full context label includes information such as the breath paragraph position within the sentence in which the phoneme is contained, the accent phrase position within the breath paragraph, and the mora position within the accent phrase.

The central phoneme extraction unit 353 extracts the central phoneme from the full context label output by the language analysis unit 354.

The phoneme segmentation unit 352 performs phoneme segmentation based on the speech signal and the central phoneme of the full context label extracted by the central phoneme extraction unit 353. The phoneme segmentation unit 352 obtains phoneme period information of each phoneme included in the speech signal by the phoneme segmentation process. The phoneme period information is information on the start time and end time of a phoneme. In other words, the phoneme segmentation unit 352 identifies the period of the phoneme in the speech signal. The phoneme segmentation unit 352 performs phoneme segmentation using, for example, a forced alignment technique of a speech recognition processing toolkit provided at the following URL.
URL http://htk.eng.cam.ac.uk

In addition to the phoneme segmentation process performed by the phoneme segmentation unit 352, it may be possible to manually check and correct the segmentation.

On the other hand, the accent phrase division unit 371 divides the full context label output by the language analysis unit 354 into accent phrase units. The accent phrase division unit 371 passes information on the full context label divided into accent phrases to the accent phrase unit integration unit 362.

The mora-unit integration unit 361 integrates the acoustic feature data output by the acoustic analysis unit 351 on a mora-by-mora basis. Specifically, the mora-unit integration unit 361 determines acoustic features on a mora-by-mora basis using the central phoneme of the full context label extracted by the central phoneme extraction unit 353 and the phoneme section information resulting from the processing of the phoneme segmentation unit 352. The acoustic features in a mora are a collection of acoustic features on a frame-by-frame basis within that mora.

The accent phrase unit integration unit 362 integrates the mora-based acoustic feature data output by the mora-based integration unit 361 into accent phrase units. Specifically, the accent phrase unit integration unit 362 integrates the mora-based acoustic feature data using the accent phrase information contained in the full context label. The acoustic feature data within an accent phrase is a collection of mora-based acoustic feature data contained in the accent phrase.

Meanwhile, the first language feature extraction unit 381 and the second language feature extraction unit 382 obtain language features from the full context label output by the language analysis unit 354. The first language feature extraction unit 381 outputs frame-by-frame language features in a mora. The second language feature extraction unit 382 outputs frame-by-frame language features in an accent phrase. The first language feature extraction unit 381 assigns the following four pieces of information to each frame in each mora in the target accent phrase:
(1) A numerical value representing the position of the breath paragraph in the sentence; (2) A numerical value representing the position of the accent phrase in the breath paragraph; (3) A numerical value representing the position of the mora in the accent phrase; (4) A numerical value representing the position of the frame in the mora.

Specifically, the four pieces of information are as follows: The numerical value of (1) above is a value obtained by dividing the breath group position in the sentence by the total number of breath groups in the sentence. When expressed as a formula using symbols in the full context label format of Figures 3 to 5, the breath group position BG (breath group) is BG = _i3 / _k1 . The numerical value of (2) above is a value obtained by dividing the accent phrase position in the breath group by the total number of accent phrases in the breath group. When expressed as a formula using symbols in the full context label format, the accent phrase position AP (accent phrase) is AP = _f5 / _i1 . The numerical value of (3) above is a value obtained by dividing the mora position in the accent phrase by the total number of moras in the accent phrase. When expressed as a formula using symbols in the full context label format, the mora position MR (mora) is MR = _a2 / _f1 . The numerical value in (4) above is the value obtained by dividing the frame position in the mora by the total number of frames in the mora. That is, the frame position FR(frame) is expressed as a formula: frame _i /frame_mora. Here, frame _i is the frame position (integer value) in the mora, and frame_mora is the total number of frames in the mora. These fourth-order numerical values are regarded as frame-by-frame language features.

The first language feature extraction unit 381 extracts and outputs a set of frame-by-frame language features within a mora. In other words, the language features within a mora are the frame-by-frame language features within the mora repeated the number of times for the total number of frames within the mora.

The second language feature extraction unit 382 extracts and outputs a set of language features on a frame-by-frame basis within an accent phrase. In other words, the language features within an accent phrase are the combination of all the language features within the moras in the accent phrase.

Then, the acoustic feature/language feature integration unit 391 integrates the acoustic features output by the accent phrase unit integration unit 362 and the language features output by the second language feature extraction unit 382 to obtain input data for training the model.

The first correct label extraction unit 386 and the second correct label extraction unit 387 extract correct labels based on the full context label by the following process. Japanese accents can be simulated by expressing high and low tones, with the pitch of each mora being either high (H) or low (L). Within one accent phrase, there can be up to one mora position where the pitch of the voice falls. Furthermore, standard Japanese (also known as the Tokyo dialect) is characterized by a low tone in the first mora and a high tone in the second mora, except for cases where the first mora in an accent phrase has a high tone and the second mora has a low tone (high-headed type).

Figure 8 is a flowchart showing the procedure of the process performed by the first correct label extraction unit 386 and the second correct label extraction unit 387 in the preprocessing unit 32 to find correct labels for each accent phrase. The premise is that the language analysis unit 354 has already found full context labels by analyzing the text. Below, the process of finding correct labels will be explained according to this flowchart.

In step S11, the first correct label extraction unit 386 converts the text into pronunciations in mora units based on the full context label obtained by the language analysis unit 354. For example, if the original text is "He twisted all reality in his direction," the processing in this step will result in the pronunciations "A/RA/YU/RU/GEN/N/JI/TS/O/SU/BE/TE/JI/BU/N/NO/HO/O/E/NE/JI/MA/GE/TA/NO/DA." In this pronunciation, the slash (/) indicates the division of a mora.

Next, in step S12, the second correct label extraction unit 387 refers to the full context label and integrates the above pronunciation of the mora unit into pronunciation of the accent phrase unit. Through the processing of this step, the pronunciation of the above example is converted into pronunciation of the accent phrase unit "arayuru/genjitsuo/subete/jibunnohooe/nejimagetanoda." Here, the slash (/) indicates the division of the accent phrase.

Next, in step S13, the second correct label extraction unit 387 refers to the full context label and creates tones for each accent phrase that correspond to the pronunciation of the accent phrase. Specifically, the second correct label extraction unit 387 creates tones for each accent phrase from the information on the mora position of the accent nucleus in the full context label. The accent nucleus is the mora immediately before the pitch drops. Specifically, the values of the mora position in the accent phrase (a ₂ represented by a symbol in the form of a full context label), the number of moras in the accent phrase (f ₁ represented by a symbol in the form of a full context label), and the accent type in the accent phrase (f ₂ represented by a symbol in the form of a full context label) are observed. When f ₂ = 1, it indicates a high-headed type, and the tone at the mora position a ₂ = 1 is H, and the tones at other positions are L. When f ₂ = f ₁ , it indicates a flat type or a high-tailed type, and the tone at a ₂ = 1 is L, and the tones at other positions are H. Cases other than the above two represent a middle-high type, with the tone for _a2 = 1 and _a2 > _f2 being L, and the tones at other positions being H. Details of the processing in this step will be explained later with reference to FIG. 9. By the processing in this step, tones for accent phrase units, "LHHL/LHHHH/HLL/LHHHHLL/LHHLLLL", corresponding to the pronunciation of the accent phrase units in the above example, are obtained. The slash (/) here also represents a division of accent phrases. Furthermore, L corresponds to low, and H corresponds to high.

Next, in step S14, the second correct label extraction unit 387 converts the tones of the accent phrase units into correct labels of accents. In this embodiment, the accent correct labels are a string of three numbers, 0, 1, and 2, corresponding to the tones of the accent phrase units described above. The meanings of these numbers are as follows. That is, the number 2 corresponds to the case where the pitch of the voice falls in the mora following the mora in the accent phrase. In other words, when the tone changes from H to L between adjacent moras, the part corresponding to the H corresponds to 2. The number 1 corresponds to the case where the pitch of the voice rises in the mora following the mora in the accent phrase. In other words, when the tone changes from L to H between adjacent moras, the part corresponding to the L corresponds to 1. The number 0 corresponds to the case where the pitch of the voice does not change in the mora following the mora in the accent phrase. In other words, when the tone changes from L-L or H-H between adjacent moras, the part corresponding to the mora before that corresponds to 0. The numerical value corresponding to the last mora in the accent phrase is also 0. Through the processing in this step, the correct label for the accent phrase unit, "1020/10000/200/1000200/1020000", is obtained based on the tones of the accent phrase unit in the above example. The slash (/) here also indicates the division of the accent phrase. Each numerical value here corresponds to a mora.

In other words, the number "2" contained in the correct label corresponds to the mora immediately before the pitch drops. The number "1" contained in the correct label corresponds to the mora immediately before the pitch rises. The number "0" contained in the correct label corresponds to the case where the pitch does not change, or the last mora in the accent phrase.

In this step, the second correct label extraction unit 387 further assigns the numerical value of the mora (correct label) to each frame in the mora. In other words, the second correct label extraction unit 387 copies the numerical value (0, 1, or 2) corresponding to the correct label of a certain mora so that it is repeated the number of times equal to the number of frames contained in the mora. In this way, the second correct label extraction unit 387 generates a string of numerical values on a frame-by-frame basis within the accent phrase. In other words, this numerical value of degree 1 is the correct label on a frame-by-frame basis within the accent phrase.

By the above-described processing, the acoustic feature/language feature integration unit 391 of the preprocessing unit 32 outputs data that integrates the acoustic feature and the language feature. In addition, the second correct answer label extraction unit 387 outputs a correct answer label related to accent. These two pairs of data are used by the model learning unit 33 to learn the statistical model.

Figure 9 is a flowchart showing the details of the process of step S13 shown in Figure 8. That is, it is a flowchart showing the procedure of the process in which the second correct label extraction unit 387 creates tones for each accent phrase. The following explanation will be given along with this flowchart.

In step S21, the second correct label extraction unit 387 judges whether or not _f2 = 1. That is, it judges whether or not the value of the accent type in the accent phrase is 1. If _f2 = 1 (step S21: YES), the process proceeds to the judgment in step S22. Otherwise (step S21: NO), the process proceeds to the judgment in step S25.

In step S22, the second correct label extraction unit 387 judges whether _a2 = 1. That is, it judges whether the mora position in the accent phrase is 1. If _a2 = 1 (step S22: YES), the tone is set to "H" (step S24) and the process of this flowchart ends. Otherwise (step S22: NO), the tone is set to "L" (step S23) and the process of this flowchart ends.

If the process proceeds to step S25, the second correct label extraction unit 387 judges whether or not _f2 = _f1 . That is, it judges whether or not the value of the accent type in the accent phrase is equal to the number of moras in the accent phrase. If _f2 = _f1 (step S25: YES), the process proceeds to the judgment in step S26. Otherwise (step S25: NO), the process proceeds to the judgment in step S29.

In step S26, the second correct label extraction unit 387 judges whether _a2 = 1. That is, it judges whether the mora position in the accent phrase is 1. If _a2 = 1 (step S26: YES), the tone is set to "L" (step S28) and the process of this flowchart ends. Otherwise (step S26: NO), the tone is set to "H" (step S27) and the process of this flowchart ends.

If the process proceeds to step S29, in this step the second correct label extraction unit 387 determines whether the condition ( _a2 = 1 or _a2 > _f2 ) is satisfied. _a2 = 1 corresponds to the case where the mora position in the accent phrase is 1. _a2 > _f2 corresponds to the case where the mora position in the accent phrase is larger than the accent type in the accent phrase. If ( _a2 = 1 or _a2 > _f2 ) is satisfied (step S29: YES), the tone is set to "L" (step S31) and the processing of this flowchart is terminated. Otherwise (step S29: NO), the tone is set to "H" (step S30) and the processing of this flowchart is terminated.

As explained above, the second correct label extraction unit 387 creates tones for each accent phrase through the process described in this flowchart.

Figure 10 is a schematic diagram showing the rules for converting the tones of an accent phrase into the numerical value of a correct label in the processing of step S14 shown in Figure 8. As shown in the figure, the second correct label extraction unit 387 determines the numerical value of the correct label corresponding to the earlier of two consecutive moras in the tone of the accent phrase, depending on the tone pattern of the two moras. However, for a mora that ends an accent phrase (pattern (E) or (F) in Figure 10), the second correct label extraction unit 387 determines the numerical value of the correct label corresponding to that mora, regardless of the tone of that mora.

Pattern (A) in FIG. 10 is a case where the tone of two consecutive moras changes from "H" to "L", and in this case the correct label value corresponding to the preceding mora is "2". Pattern (B) is a case where the tone of two consecutive moras changes from "H" to "L", and in this case the correct label value corresponding to the preceding mora is "1". Pattern (C) is a case where the tone of two consecutive moras goes from "H" to "H", and in this case the correct label value corresponding to the preceding mora is "0". Pattern (D) is a case where the tone of two consecutive moras goes from "L" to "L", and in this case the correct label value corresponding to the preceding mora is "0". In other words, patterns (C) and (D) are cases where the tone of two consecutive moras continues without changing. Patterns (E) and (F) correspond to the last mora in an accent phrase, and in this case, regardless of whether the tone of the mora is "H" or "L", the numerical value of the correct label corresponding to the mora is "0".

The model learning unit 33 learns a statistical model using input data (learning data combining acoustic features and linguistic features on a frame-by-frame basis) in units of accent phrases passed from the preprocessing unit 32 and the correct label. The statistical model is realized, for example, using a DNN (deep neural network). Specifically, the model learning unit estimates a label based on the above input data, finds the difference between the estimated label and the correct label, and updates the internal parameters of the statistical model using the backpropagation method based on the difference.

The statistical models may be constructed by distinguishing between a model for the voices of male speakers and a model for the voices of female speakers. Alternatively, a gender-independent model may be constructed as the statistical model without distinguishing between male and female speakers. When training a model for male speakers, the model training unit 33 uses training data for male speakers. When training a model for female speakers, the model training unit 33 uses training data for female speakers. When training a gender-independent model, the model training unit 33 uses a mixture of training data for both male and female speakers. In addition, a statistical model for a specific individual speaker may be constructed as the statistical model.

The input data in the training data is, for example, 84th-order data combining 80th-order acoustic feature data and 4th-order linguistic feature data. As the input data, for example, data on hundreds of thousands of accent phrases is used. The input data is standardized by calculating the average and standard deviation for each of the 84 dimensions. The correct label data can be used as is without standardization.

As described above, for example, hundreds of thousands of pieces of learning data are rearranged in a random order and divided into training data, validation data, and evaluation data in a ratio of, for example, 8:1:1 for use. The learning process by the model learning unit 33 is performed in mini-batch units. The mini-batch size handled at one time is, for example, 32 accent phrases. For each mini-batch, the shorter accent phrases are padded with zeros to match the accent phrase with the longest number of frames (number of frames: frame_max) and used for learning.

The process itself in which the model learning unit 33 learns the statistical model can be performed using existing technology. As an example, TensorFlow, an open source software machine learning toolkit, can be used. TensorFlow is described in the following URL:
URL https://tensorflow.org

Figure 11 is a schematic diagram showing the flow of processing when the model learning unit 33 learns a statistical model. As shown in the figure, the statistical model is configured to include the following layers: an input layer 411, a masking layer 412, a bidirectional LSTM 413, a normalization layer 414, a dropout layer 415, a fully connected layer 416, and a softmax function 417. Note that LSTM is an abbreviation for "Long short-term memory". Bidirectional LSTM is sometimes written as "BLSTM".

During learning, the model learning unit 33 inputs a tensor [32,frame_max,84] of 84th-order input data, which is adjusted to the longest number of frames per mini-batch, frame_max, with a mini-batch size of 32, to the input layer. The input data is processed by the input layer 411 and the masking layer 412. The masking layer 412 outputs a tensor [32,frame_max,84] with masking information that ignores the zero-padded portion. The masking information propagates to layers after the bidirectional LSTM 413, enabling calculations that ignore the zero-padded portion. The model learning unit 33 passes the output tensor [32,frame_max,84] from the masking layer 412 through a bidirectional LSTM layer 413 (adding the forward and backward outputs) with 64 units × 6 layers, and outputs a tensor [32,frame_max,64]. In the bidirectional LSTM layer 413, a 12-order regularization is applied to the kernel at 0.001 to prevent overlearning. Then, in order to further improve the generalization performance, the model learning unit 33 passes the output tensor [32,frame_max,64] from the six-layer bidirectional LSTM layer 413 through a normalization layer 414 and a dropout layer 415 with a drop rate of 0.5 to output the tensor [32,frame_max,64]. The model learning unit 33 passes the output tensor [32,frame_max,64] from the dropout layer 415 through a fully connected layer 416 with three units to output the tensor [32,frame_max,3]. Based on the output tensor [32,frame_max,3] from the fully connected layer 416, the model learning unit 33 performs the calculation of the softmax function 417 while ignoring values masked by masking for each frame. The model learning unit 33 outputs the tensor [32,frame_max,3] of the estimated value, which is the output of the model.

The model learning unit 33 calculates a loss function (difference, loss) using the correct label [32,frame_max,1] output from the preprocessing unit 32 and the above estimated value [32,frame_max,3]. Specifically, the model learning unit 33 obtains a mask value [32,frame_max,1] for ignoring the zero-padded portion of the correct label, and calculates the loss using the labels in multi-class classification with the correct label [32,frame_max,1] and the estimated value [32,frame_max,3]. To calculate the loss, the model learning unit 33 calculates the SparseCategoricalCrossentropy() function (sparse categorical cross entropy) and obtains the calculation result [32,frame_max]. Then, the model learning unit 33 expands the dimension with the expand_dims() function to [32,frame_max,1]. The model learning unit 33 takes the logical product (logical_and() function) of this value and the mask value [32,frame_max,1] obtained above, and calculates the average (reduce_mean() function) of the obtained value as the loss. Note that the reduce_mean() function is a function that takes the average of a specific dimension of a tensor.

The model learning unit 33 calculates the training loss of the model for each batch of training data, obtains the gradient of the model weights with respect to the training loss, and uses an optimizer to update the model weights based on the gradient. In this case, the model learning unit 33 uses SGD (Stochastic Gradient Descent) with learning_rate=0.00001, decay=1e-6, and momentum=0.9 as the optimizer.

The model learning unit 33 updates the retained model when the above training loss falls below the minimum training loss up to the previous epoch. The model learning unit 33 performs learning for up to 1000 epochs, with one epoch being the process of completing calculations for all mini-batches of training data. The model learning unit 33 calculates the validation loss and validation accuracy of the model using validation data for each epoch, and if the validation loss does not improve for 50 consecutive epochs, it terminates learning early. The model learning unit 33 adopts the retained model when the validation loss is minimized in this way.

The model learning unit 33 writes the parameter information of the adopted model to the model storage unit 11.

Next, the label estimation process will be described in detail. When performing label estimation, the data on which the estimation is based includes a voice signal for each utterance and a transcribed text corresponding to the voice signal. In this embodiment, the estimation voice corpus storage unit 21 holds the above pairs of voice signals and text.

The preprocessing unit 22 has a configuration similar to that of the preprocessing unit 32 described above. However, the preprocessing unit 22 does not need to have a configuration for extracting correct labels. In other words, the preprocessing unit 22 does not need to have the functions of the first correct label extraction unit 386 and the second correct label extraction unit 387 of the preprocessing unit 32 shown in FIG. 2. The preprocessing unit 22 creates and outputs input data consisting of acoustic features and language features on a frame-by-frame basis for each accent phrase based on the text and speech signal for each utterance, similar to the preprocessing unit 32. Since the preprocessing unit 32 has already been described with reference to FIG. 2, a detailed description of the procedure by which the preprocessing unit 22 creates input data consisting of acoustic features and language features will be omitted here.

The label estimation unit 23 estimates labels for accents based on the input data output by the preprocessing unit 22.

Figure 12 is a schematic diagram showing the flow of processing in which the label estimation unit 23 estimates a label. As shown in the figure, the configuration of the statistical model at this time is the same as the configuration shown in Figure 11 (reference numerals 411 to 417). However, during estimation, the dropout layer 415 is ignored. At the time of label estimation processing, the statistical model has already been trained. In other words, the values of the parameters (weights) of the statistical model have already been determined. During label estimation processing by the label estimation unit 23, no adjustment of parameters based on the correct label is performed.

The input data that the label estimation unit 23 receives from the preprocessing unit 22 is a tensor [1, frame, 84] of input data with a mini-batch size of 1, the number of frames frame, and a degree of 84. The label estimation unit 23 inputs the input data from the input layer 411. The processing in each layer from the input layer 411 to the softmax function 417 has already been described as the processing for model learning. The label estimation unit 23 obtains an estimated value [1, frame, 3] with a mini-batch size of 1, the number of frames frame, and a degree of 3 using this trained model. For each frame of the obtained estimated value [1, frame, 3], the label estimation unit 23 sets the maximum index (0, 1, or 2) among the three-degree values as the value of the estimated label for each frame. Since one mora includes many frames, the label estimation unit 23 determines the estimated label for each mora by taking a majority vote of the estimated label values (0, 1, or 2) for each frame in the mora.

The accent can be identified by the value of the estimated label for each mora in the accent phrase. When generating speech using synthetic speech, the pitch can be adjusted by the value of the estimated label for each mora. If the estimated label for each mora is "1", the pitch of the voice will rise in the mora following that mora. If the estimated label for each mora is "2", the pitch of the voice will fall in the mora following that mora. If the estimated label for each mora is "0", there will be no situation in which the pitch changes in the mora following that mora (including cases where there is no next mora in the accent phrase).

FIG. 13 is a block diagram showing an example of an internal configuration for realizing the functions of the accent estimation device 1. The accent estimation device 1 (or at least a part of it) can be realized using a computer. As shown in the figure, the computer is configured to include a central processing unit 901, a RAM 902, an input/output port 903, input/output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. According to each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic operations and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input/output port 903 is a port through which the central processing unit 901 exchanges data with external input/output devices, etc. Input/output devices 904 and 905 are input/output devices. Input/output devices 904 and 905 exchange data with central processing unit 901 via input/output port 903. Bus 906 is a common communication path used inside the computer. For example, central processing unit 901 reads and writes data in RAM 902 via bus 906. Also, for example, central processing unit 901 accesses the input/output port via bus 906.

At least some of the functions of the accent estimation device 1 in the above-mentioned embodiment can be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into computer systems. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above-mentioned program may be a program for realizing some of the above-mentioned functions, and may further be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

As described above, in this embodiment, by using a statistical model trained on acoustic features and linguistic features on a frame-by-frame basis within an accent phrase, it is possible to accurately estimate an accent by taking into account the influence of an utterance, such as the breath paragraph position within the utterance, the accent phrase position within the breath paragraph, the mora position within the accent phrase, and the frame position within the mora. Furthermore, in this embodiment, the statistical model can be trained based on the acoustic features and linguistic features on a frame-by-frame basis within the accent phrase, and the correct answer label.

The accent estimation device 1 or 2 and the accent estimation model learning device 3 may be implemented by the following modified examples. In the above embodiment, a statistical model was learned based on text and speech signals with accent phrases as units. Also, assuming that a learned statistical model exists, a process was performed to estimate accent labels based on text and speech signals with accent phrases as units. In modified examples, accent labels may be estimated or statistical models may be learned based on text and speech signals of other predetermined units, not limited to accent phrase units. Also, in modified examples, information on the position of a predetermined unit (not limited to accent phrase units, any unit) in an utterance, information on the position of a mora, or information on the position of a frame is used as a linguistic feature.

In addition, in the above embodiment, a mel spectrogram is used as the acoustic feature. As a modification, the accent estimation device 1 or 2 and the accent estimation model learning device 3 may be implemented using an acoustic feature other than a mel spectrogram.

The above describes in detail the embodiments and variations of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and includes designs that do not deviate from the gist of the present invention.

[Evaluation of Examples]
Evaluation of the examples will be described below. In the examples, the accent estimation model training device 3 trained an unspecified speaker statistical model for female voices and an unspecified speaker statistical model for male voices. As training data for the unspecified speaker statistical model for female voices, 289,197 accent phrases extracted from 37,226 utterances by eight female speakers were used. As training data for the unspecified speaker statistical model for male voices, 196,956 accent phrases extracted from 39,065 utterances by nine male speakers were used.

Furthermore, label estimation was evaluated using the trained speaker-independent statistical model for female voices and speaker-independent statistical model for male voices. The evaluation data used was data that was not used for training the models. As a result, the accuracy of the estimated accent labels was as follows. That is, the accuracy of the speaker-independent statistical model for female voices was 98.98%. The accuracy of the speaker-independent statistical model for male voices was 99.28%. In this way, this embodiment was able to estimate accent labels with high accuracy.

The present invention can be used, for example, to estimate an accurate accent label for a given accent phrase. Furthermore, the present invention can be used, for example, for speech synthesis. However, the scope of use of the present invention is not limited to the examples given here.

1, 2 Accent estimation device (signal processing device)
3. Accent estimation model learning device (signal processing device)
11 Model storage unit 21 Estimation speech corpus storage unit 22 Preprocessing unit 23 Label estimation unit 31 Training speech corpus storage unit 32 Preprocessing unit 33 Model training unit 351 Acoustic analysis unit (training acoustic analysis unit)
352 Phoneme segmentation unit 353 Central phoneme extraction unit 354 Language analysis unit (Learning language analysis unit)
361: mora unit integration unit 362: accent phrase unit integration unit 371: accent phrase division unit 381: first language feature extraction unit 382: second language feature extraction unit 386: first correct label extraction unit (correct label extraction unit)
387 Second correct label extraction unit (correct label extraction unit)
391 Acoustic feature and language feature integration unit 901 Central processing unit 902 RAM
903 Input/Output Port 904, 905 Input/Output Device 906 Bus

Claims (8)

an acoustic analysis unit that calculates, for each predetermined unit of text, an acoustic feature quantity for each frame based on a speech signal corresponding to the predetermined unit of text;
a language analysis unit that calculates language features of the text in units of frames based on the text in the predetermined units;
a label estimation unit that estimates an accent label corresponding to the text as an output from a pre-trained statistical model by inputting the frame-by-frame acoustic features and the linguistic features into the pre-trained statistical model;
A signal processing device comprising: a training acoustic analysis unit that calculates training acoustic features in units of frames for each training text of a predetermined unit based on a training speech signal corresponding to the training text of the predetermined unit;
a training language analysis unit that calculates training language features for each frame of the training text based on the training text of the predetermined unit;
a correct label extraction unit for determining a correct label of an accent corresponding to the text based on the training language features;
a model training unit that inputs the training acoustic features and the training linguistic features for each frame into a statistical model to obtain training estimated labels of accents corresponding to the training text as an output from the statistical model, and trains the statistical model based on an error between the training estimated labels and the ground truth labels;
The signal processing device according to claim 1 , further comprising: an acoustic analysis unit that calculates, for each predetermined unit of text, an acoustic feature quantity for each frame based on a speech signal corresponding to the predetermined unit of text;
a language analysis unit that calculates language features of the text in units of frames based on the text in the predetermined units;
a correct label extraction unit for determining a correct label of an accent corresponding to the text based on the linguistic feature;
a model learning unit that inputs the frame-by-frame acoustic features and the linguistic features into a statistical model to obtain an estimated label of an accent corresponding to the text as an output from the statistical model, and learns the statistical model based on an error between the estimated label and the ground truth label;
A signal processing device comprising: The predetermined unit is an accent phrase unit.
A signal processing device according to any one of claims 1 to 3. The linguistic feature amounts are a breath group position in an utterance, an accent phrase position in the breath group, a mora position in the accent phrase, and a frame position in the mora.
The signal processing device according to claim 4. The statistical model is realized using a deep neural network.
A signal processing device according to any one of claims 1 to 5. The acoustic feature is a mel spectrogram.
A signal processing device according to any one of claims 1 to 6. Computer,
A signal processing device according to any one of claims 1 to 7.
A program to function as a

Images