JPH077276B2 - Syllable recognizer - Google Patents

Description

The present invention relates to a syllable recognition device using a time delay neural network.

<Prior Art> Conventionally, there are the following syllable spotting devices using a time-delayed neural network (hereinafter referred to as TDNN) (Sawai, Alex Weibel, Kano: "Syllables by time-delayed neural network". Examination of spotting "Proceedings of the Acoustical Society of Japan 1988-10). This syllable spotting device consists of a TDNN consisting of an input layer, two intermediate layers and an output layer. As a sample for learning this TDNN, syllables / B
We selected 53 words including A / and used a voice sample that cut out 15 frames (10 ms period) of syllable / BA /.
The 16th-order fast Fourier transform mel spectrum of the audio signal is used as the input pattern. In addition, this TD
The units in the output layer of the NN are recognition categories “BA” and “non-B”.
Two are provided corresponding to A ". Learning is performed by the error back propagation method.

How to give the teacher data at the time of learning to the above TDNN is the boundary position between the phoneme / B / and the phoneme / A / of the input pattern and the TD
When the deviation from the center position of NN is within a certain time, "1" is given to the unit assigned to the recognition category "BA" of the output layer, and "1" is given to the unit assigned to the recognition category "non-BA". Give 0 ".

The unknown voice signal is input by scanning the above input pattern of the unknown syllable to each unit of the input layer of the TDNN while shifting by 3 frames. Then, when the output value of the unit assigned to “BA” in the output layer is larger than the output value of the unit assigned to “non-BA”, it is determined that the syllable of the input pattern is / BA /. In the opposite case, the syllable in the category of the input pattern is determined to be / non-BA /.

<Problems to be Solved by the Invention> As described above, the conventional syllable spotting device described above is
The input pattern input to the input layer of the TDNN is a 16th-order fast Fourier transform mel spectrum extracted from the speech signal, and the categories represented by the output data output from the output layer are "BA" and "non-BA". is there. That is, the feature pattern of the voice signal is input to the TDNN to directly recognize the syllable. Therefore, the progress of the syllable recognition process in TDNN (for example, the value of the weight between units in each layer) is unknown. Moreover, even if it is understood, the meaning of the value of the weight is unknown.

Therefore, if the TDNN learning does not converge easily, there is a problem that the cause of the non-convergence of learning is completely unknown.
Further, since the cause of the non-convergence of the learning is completely unknown in this way, there is a problem that the learning cannot be dealt with toward the convergence and the learning time becomes longer than necessary.

Therefore, an object of the present invention is to provide a syllable recognition device capable of knowing the progress of the syllable recognition process and shortening the learning time of the TDNN.

<Means for Solving the Problems> In order to achieve the above object, the syllable recognition device of the present invention is
A time-delay means is provided, and a signal obtained by combining a time series of signals representing sequentially input acoustic parameters and a signal obtained by delaying the time series of signals representing the acoustic parameters by a predetermined time by the time delay means A first time-delay neural network for converting a signal representing a single tone into a time series and outputting the time-series signal, and a time-delaying means are provided, and a signal representing a phoneme or a single tone output from the first time-delay neural network is output. A signal obtained by sequentially inputting a time series and combining a time series of signals sequentially representing the phonemes or single tones and a signal obtained by delaying the time series of the signals representing the phonemes or single tones by a predetermined time by the time delay means is generated. , A second time-delay neural network for converting a signal representing a syllable into a time series and outputting the time series.

<Operation> A time series of signals representing acoustic parameters extracted from unknown words is sequentially input to the first time delay neural network.
Then, the first time-delay neural network combines the time series of the signals sequentially representing the acoustic parameters and the signal obtained by delaying the time series of the signals representing the acoustic parameters by the time delay means. The signal is converted into a time series of a signal representing a phoneme or a single sound and output. Then, the time series of the signal representing the phoneme or the single sound output from the first time-delay neural network is input to the second time-delay neural network.

Then, the second time-delay neural network divides the time series of signals sequentially representing phonemes or single tones and the signals obtained by delaying the time series of signals representing the phonemes or single tones by a time delay means. The combined signals are converted into a time series of signals representing syllables and output. Therefore,
The syllable can be recognized by the time series of the signal representing the syllable output from the second time delay neural network.

At that time, by monitoring a signal representing a phoneme or a single sound output from the first time-delay neural network, the progress of the syllable recognition process (that is, the recognition result of the phoneme sequence forming the syllable) is known. It is possible.

<Examples> The present invention will be described in detail below with reference to illustrated examples.

FIG. 1 is a block diagram of the syllable recognition device of the present invention.
This syllable recognizer is roughly composed of two TDNNs connected in series. One of the TDNNs is composed of a first multilayer perceptron type neural network (hereinafter referred to as NN) 1, and the other TDNN is composed of a second NN2. First
The first NN1 and the second NN2 in the figure are simplified by omitting the input layer, the intermediate layer, the output layer, the units included in each layer, the coupling between the units, and the like.

Syllables consist of a chain of phonemes. Therefore, in the syllable recognition device of the present embodiment, the phonemes forming the syllable to be recognized are used as a means for knowing the progress of the syllable recognition operation. That is, the input data of the first NN1 is acoustic parameters, while the recognition category is phonemes. In addition, the input data of the second NN2 is the data representing the phoneme which is the identification category of the first NN1 (the output data of the first NN1 in this embodiment), while the identification category is the syllable. By doing this, the
By monitoring the output data of 1NN1, it is possible to know the progress of the syllable recognition operation for the acoustic parameters input to the first NN.

The learning sample input to the input layer of the first NN1 is a feature pattern extracted from a speech signal of a word whose utterance content is already known, and phonemes are labeled by visual inspection such as power. Here, one frame is about 8 ms to 10 ms. Further, as the characteristic pattern, for example, an output value from an m-channel band pass filter group, an m-th order autocorrelation coefficient, an m-th order cepstrum coefficient, or the like is used. That is, the order of the input data is m. As the first NN1 teacher data, data representing a phoneme based on the label of the learning sample created as described above is used.

The input layer (not shown) in the first NN1 is m × (A + 1)
(A: maximum number of delayed frames described in detail later) units. Each unit of the input layer is sequentially divided into m blocks of (A + 1) units from one unit, and the first unit of the i-th (1 ≦ i ≦ m) block has the i-th unit. The following acoustic parameters are input. In addition, an acoustic parameter obtained by delaying the i-th acoustic parameter by one frame by the delay element 3 that delays the input signal by a time corresponding to one frame is input to the next unit. Further, an acoustic parameter obtained by delaying the i-th acoustic parameter by two frames by two delay elements 3 is input to the next unit.
Hereinafter, in the same manner, the acoustic parameter obtained by delaying the i-th acoustic parameter by A frames by A delay elements 3 is input to the last unit. In this way, the input pattern of the nth frame × mth order as described above is sequentially delayed from 0th frame to Ath frame, and sequentially input to m × (A + 1) units of the input layer one frame at a time.

On the other hand, the number of output layers (not shown) in the first NN1 is p (p:
The number of phonemes to be identified by the first NN1). Usually, there are about 20 types of phonemes in Japanese. Each unit in the output layer is assigned to an individual phoneme to be identified (for a phoneme susceptible to articulatory coupling, a plurality of units may be assigned based on experience). That is, in FIG. 1, the unit at one end (first unit) is assigned to the phoneme / a /, the j-th unit is assigned to the phoneme / r /, and the p-th unit is the phoneme / b /.
Is assigned to. By doing so, when the first unit has the maximum output, the phoneme of the input acoustic parameter is recognized as / a /, and when the j-th unit has the maximum output, the input sound is recognized. The parameter phoneme is recognized as / r /.

The input layer (not shown) in the second NN2 is p × (B
+1) units (B: maximum delay frame number described in detail later). As in the case of the first NN1, each unit of the input layer is divided into p blocks of (B + 1) units sequentially from the one end unit, and the j-th (1 ≦ j ≦ p) block 1st NN1 for first unit of block
The output signal from the j-th unit of the output layer of is input.
A signal obtained by delaying the output signal from the j-th unit by one frame by the delay element 3 is input to the next unit. Further, a signal obtained by delaying the output signal from the j-th unit by two frames by the two delay elements 3 is input to the next unit. Hereinafter, in the same manner, a signal obtained by delaying the output signal from the j-th unit by B delay elements 3 by B frames is input to the last unit. In this way, the input pattern composed of the p output signal strings as described above is delayed from 0 frame to B frame in sequence, and p × (B +
1) The data is sequentially input to each unit.

On the other hand, the number of output layers (not shown) in the second NN2 is s (s:
The number of syllables to be identified by the second NN2). Generally, there are about 100 types of Japanese syllables. Each unit in the output layer is assigned to an individual syllable to identify. That is, in FIG. 1, the first unit is assigned to the syllable / a / and the second unit is assigned to the syllable / i /.
, And the sth unit is assigned to the syllable / syo /. By doing this, for example, the first
Recognize that the syllable for the input pattern input is / a / when the unit has the maximum output, and / ka / for the input pattern when the sixth unit has the maximum output. When the s-th unit exhibits the maximum output, the syllable for the input pattern inputted is recognized as / syo /.

The syllable recognition device having the above configuration is trained as follows.

FIG. 2 is an illustration of how to give teacher data. Hereinafter, a method of giving learning data will be described in detail with reference to FIG.

2 (a) is a power curve of a word / two people / whose voicing contents are known as a learning sample, and FIG. 2 (b)
Shows acoustic parameters (input data to the input layer of the first NN1) corresponding to the power curve of FIG. 2 (a), FIG. 2 (c) shows teacher data of the first NN1, and FIG. 2 (d). Is the
2NN2 shows teacher data. In FIG. 2B, the acoustic parameters are represented by a matrix of orders and frames, but concrete data are omitted.

As described above, the phoneme label is attached to the acoustic parameter for each frame by observing the power curve of the learning sample / two /. This added phoneme label is the second
The lower part of the figure (a) is associated with the power curve. The data representing this phoneme label becomes the first NN1 teacher data.

The teacher data of the first NN1 is created as follows.
That is, in the frame after the time corresponding to the A frame has elapsed from the input of one frame of the acoustic parameter with a certain phoneme label, the signal "1" is output to the unit of the output layer assigned to the above phoneme. The data that gives a signal "0" to the other units is used as the teacher data of the phoneme. For example, FIG. 2 (b)
In the frame f 2 in FIG. 2 (c) after the time corresponding to the A frame has elapsed from the frame f ₁ corresponding to the phoneme / h / in FIG. ₂ , “1” is assigned to the unit assigned to the phoneme / h /. The data that is given to the other units and "0" is given to the other units as the teacher data (called the phoneme / h / teacher data).

The first NN1 executes learning by the first NN1 alone as follows. That is, in the learning sample / two /
The primary acoustic parameters of the first frame (first frame) labeled / h / are input to input terminal 4,
The secondary acoustic parameter is input to the input terminal 5, the i-th acoustic parameter is input to the input terminal 6, and the m-th acoustic parameter is input to the input terminal 7. Hereinafter, similarly, the acoustic parameters of the second frame, the third frame, ... Are sequentially input to the input terminals 4 to 7.

On the other hand, as shown in FIG. 2 (c), "1" is given to the unit assigned to the phoneme / h / after the time of A frames has elapsed since the acoustic parameters of the first frame were input, Phoneme / h / teacher data that gives “0” to other units is input to each unit of the output layer for a time corresponding to three frames. Similarly, "0" teacher data (that is, teacher data that gives "0" to all units in the output layer) is 1 frame, phoneme / u / teacher data is 4 frames, and "0" Input 4 frames of teacher data in sequence.

FIG. 2 (c) shows the teacher data input as described above in a matrix in which phonemes are arranged in rows and frames are arranged in columns (where A = 3, and the data content is "1"). Fill in only "0" is omitted). Also, phonemes / h /
Between / and / u /, between phonemes / a / and / r /, between phonemes / i / and / n /, and between phonemes / n / and / o / “0” teacher data Is inserted to remove the large influence of the phoneme before or after.

Here, the acoustic parameters to be input to the input layer are sequentially delayed from 0 frame to A frame, and the input timing of the teacher data to the output layer is A frames for the input of the acoustic parameters to the input terminals 4 to 7. The reason for delaying is as follows. That is, even if the acoustic parameters are given the same phoneme label, there are various acoustic parameters due to articulation coupling or the like. Therefore, in order to absorb the influence of such articulatory coupling, an acoustic parameter having a certain phoneme label is first delayed from 0 frame to A frame and input to the input layer, and then the above phoneme level is added. The teacher data is input after A frames have elapsed from the input of the acoustic parameter of the first frame. By doing this, it is possible to input the teacher data corresponding to the above label with the same label and various acoustic parameters that are changed by the articulatory coupling being simultaneously input to the first NN1 (that is, with a large amount of information). It is possible to absorb the influence of articulation coupling.

The input terminals 4 to 4 of the first NN1 learned as described above
When the time series of the m-th order acoustic parameter of the unknown word is sequentially input to 7, the acoustic parameter is sequentially input from 0 frame.
The data delayed by the time corresponding to the frame is input to each unit in the input layer. Then, output data corresponding to the structure of the first NN1 after learning (that is, the weight of coupling between units in each layer) is output from each unit in the output layer.
At that time, the unit assigned to the phoneme corresponding to the input acoustic parameter outputs the maximum value signal.

In the output layer of the first NN1 on which learning is completed in this way, the first
As shown in the figure, the input layers of the second NN2 are connected and the learning of the second NN2 is executed. In this case, the content of the output signal from each unit of the output layer of the first NN1 is displayed on a display device (not shown) so that the determination result of the first NN1 (that is, the recognized phoneme) can be monitored.

The second NN2 teacher data is created as follows. That is, in the frame after the time corresponding to the B frame has elapsed since the data representing the earliest phoneme of the phoneme chain forming the syllable was input, it is assigned to the syllable composed of that phoneme and the next phoneme. The data that gives the signal "1" to the output layer unit and the signal "0" to the other units is used as the teacher data. For example, in frame f ₃ in FIG. 2 (d) after the time corresponding to B frame has elapsed from the previous frame f ₂ in the teacher data of phoneme / h / shown in FIG. 2 (c), syllables / h
The data that gives "1" to the unit assigned to u / and "0" to the other units is the teacher data for the syllable / hu /.

The learning of the second NN2 is executed as follows. That is, similarly to the case of learning the first NN1, learning samples / two acoustic parameter time series and delayed acoustic parameter time series are sequentially input to each unit of the learned first NN1 input layer. Then, since the first NN1 has already finished learning the boundary of the phonemes to be identified, each unit of the output layer of the first NN1 outputs output data that is substantially equal to the teacher data in FIG. 2 (c). Then, the data (0 or more and 1 or less) representing the phoneme / h / of the first frame (first frame) of the output data which is approximately equal to the teacher data of FIG. 2 (c) output from the first NN1 is stored in the input layer. Input to each unit. Similarly, in each unit, the data of the second frame (data representing phoneme / h /), the third frame (data representing phoneme / h /), the fourth frame (data of “0”) ... Are sequentially input. On the other hand, the phoneme of the first frame
After the time for B frames has passed since the data representing / h / was input, give "1" to the unit assigned to syllable / hu / and give "0" to the other units. Teacher data is input to each unit in the output layer for a time corresponding to one frame. Further, the teacher data of the syllable / hu / is input for a time corresponding to two frames, following the time corresponding to one frame.

In the same manner, 8 frames of teacher data of "0", 2 frames of teacher data of syllable / ta /, 4 frames of teacher data of "0" are sequentially input.

FIG. 2 (d) represents the teacher data input as described above in a matrix in which syllables are arranged in rows and frames are arranged in columns (where B = 5, and the data content is "1"). Fill in only "0" is omitted).

If the learning does not easily converge during the learning of the second NN2, the contents of the output signal from each unit of the output layer of the first NN1 are confirmed by the display device. As a result,
If the data represents the phonemes corresponding to the acoustic parameters input to 1NN1, it is assumed that the cause of non-convergence of learning is on the second NN2 side, and some measure is applied to the second NN2, such as changing the weight of the synaptic connection. . If the data does not represent the phoneme corresponding to the input acoustic parameter, it is assumed that the cause of the non-convergence of learning lies on the side of the first NN1 and the re-learning of the first NN1 is performed. In this way, by learning the operating state of the first NN1, it is possible to efficiently perform the learning of the syllable recognition device and shorten the learning time.

Here, data representing a phoneme input to the input layer is sequentially delayed from 0 frame to B frame, and the input timing of the teacher data to the output layer is B frame with respect to the input of the data representing the phoneme to the input layer. The reason for delaying the delay is as follows. That is, for example, syllable / hu / is phoneme / h / and phoneme /
Made up of u / chains. Therefore, the teacher data of the voice / hu / must be input with the data representing the phoneme / h / and the data representing the phoneme / u / input to the second NN2.
Therefore, the data representing the phoneme / h / is set to 1 by the delay element 3.
By delaying from frame to B frame in sequence, data representing phoneme / h / is held, and in this state, the next phoneme / h /
Input data representing u / and input data representing phoneme / h / and phoneme /
The data representing u / should be input at the same time. Then, in this state, the teacher data of the syllable / hu / is input. Therefore, the teacher data of the syllable / hu / need only be input for several frames in which the data representing the phoneme / h / and the data representing the phoneme / u / are held at the same time.

In addition, the boundary position between the phoneme / h / frame chain in the syllable / hu / and the phoneme / u / frame chain changes depending on the speaker, the speaking speed, and the like. Therefore, the phoneme / h input to the second NN2
By delaying the data representing / and the data representing phoneme / u /, teacher data of syllable / hu / is input to the second NN2 during three frames (see FIG. 2 (d)). The boundary position between the signal chain representing the phoneme / h / and the signal chain representing the phoneme / u / is changed (the boundary position moves to the front of the syllable / hu / over time). By doing so, it is possible to absorb the change in the boundary position between the phoneme / h / and the phoneme / u / due to the speaker or the vocalization speed.

In each unit of the input layer of the second NN2 learned as described above, when a time series of data representing phonemes of unknown words is input, output data corresponding to the structure of the second NN2 after learning is output in each of the output layers. Output from the unit. At that time, the unit assigned to the syllable corresponding to the input data outputs the maximum value signal.

The syllable recognition device composed of the first NN1 and the second NN2 learned as described above recognizes a syllable as follows.

In FIG. 1, when a time series of signals representing m-th order acoustic parameters of an unknown word are sequentially input to the input terminals 4 to 7 of the first NN1, the input m-th order acoustic parameters are set to 0 by the delay element 3. The signals are sequentially delayed from the frame to the time corresponding to the A frame and input to each unit in the input layer.
Then, the first NN1 converts the time series of the input acoustic parameters into the time series of the data representing the phonemes according to the structure after learning as described above, and outputs the time series of the units of the output layer. This output data is data that represents a phoneme corresponding to the input acoustic parameter.

In this way, the time series of the data representing the phonemes output from each unit of the output layer of the first NN1 is delayed by the delay element 3 of the second NN2 from 0 frame to the time corresponding to the B frame, and the second NN2 Input to each unit of the input layer of. Then, the second NN2 converts the time series of the data representing the input phonemes into the time series of the data representing the syllables and outputs them from each unit of the output layer according to the structure after learning as described above. This output data is data such that the output signal from the unit of the output layer assigned to the syllable corresponding to the data string representing the input phoneme has the maximum value.

That is, the output data from each unit in the output layer of the second NN2 becomes a syllable time series corresponding to the m-th order acoustic parameter time series of the unknown word input to the input terminals 4 to 7 of the first NN1.

At that time, as described above, the first NN1 and the second NN2 have a TDNN structure. Therefore, it is possible to absorb the influence of articulation and the like when the phoneme is recognized by the first NN1, and it is possible to absorb the influence of the speaker and the speaking speed to some extent when the syllable is recognized by the second NN2. Therefore, the syllable can be correctly recognized regardless of the speaker or the pronunciation speed.

In addition, the syllable recognition device is the first N
The output data from each unit of the N1 output layer can be monitored to know the progress of the syllable recognition operation. Therefore, when the recognition result of the syllable is incorrect, the cause of the misrecognition is either the first NN1 or the second NN2 by knowing the progress of the syllable recognition operation (that is, the recognition result of the phonemes that form the syllable). You can know That is, it is possible to take appropriate measures depending on the cause of the misrecognition,
It is possible to obtain a more accurate syllable recognition result.

As described above, the syllable recognition device of the present invention is composed of two TDNNs connected in series, and the first TDNN is a time series of the signals representing the acoustic parameters of the input unknown word and the time series of the signals representing the phonemes. While converting to a sequence and outputting,
The TDNN inputs the time series of the signal representing the phoneme output from the first TDNN, converts the time series of the signal representing the input phoneme into the time series of the signal representing the syllable, and outputs the time series. There is. Therefore, it is possible to know the progress of the syllable recognition process by monitoring the output data of the first TDNN. Therefore, for example, when the learning of the NN does not easily converge or is erroneously recognized, it is possible to know the cause of the unconverged learning and the erroneous recognition from the content of the output data of the first TDNN.

That is, according to the syllable recognition device of the present invention, it is possible to appropriately deal with unconverged learning and misrecognition.
The learning time can be shortened and more correct recognition result can be obtained.

In the above embodiment, the delay element is used as the time delay means, but the invention is not limited to this.

In the above embodiment, when learning the NN, the first NN1 is first learned, and the learned first NN1 and the unlearned second NN1 are learned.
The second NN2 is learned by connecting with N2. However, the present invention is not limited to this, and the first NN1 and the second NN2 are independently learned, and then the learned first NN1 and the second NN2 are connected to further reduce the learning time. Good.

In the above embodiment, the first NN1 classification category is phonemes. However, the present invention is not limited to this, and single sounds may be used as the classification category. By doing this, it is possible to introduce conventional knowledge about speech recognition and make allophones of a single phoneme representing a certain phoneme one of the recognition categories of the 1st NN1, which enables more accurate syllable recognition. is there.

In the above embodiment, the TDNN is composed of a multilayer perceptron type neural network. However, the present invention is not limited to this, and may be configured by a Kohonen type neural network.

<Effects of the Invention> As is clear from the above, the syllable recognition device of the present invention is
A first time-delay neural network having time delay means and a second time-delay neural network having time delay means,
The first time-delay neural network generates a phoneme or a single sound by combining a signal obtained by sequentially inputting a time series of signals representing acoustic parameters and a signal obtained by delaying the time series of signals representing the acoustic parameters by a predetermined time. Is output after being converted into a signal indicating
A signal representing a syllable is a combination of a time series of signals representing phonemes or single sounds sequentially input from the time-delayed neural network and a signal obtained by delaying the time series of signals representing the phonemes or single sounds by a predetermined time. Since the signal is converted and output, by monitoring the signal representing the phoneme or the single sound output from the first time delay neural network, the progress of the syllable recognition process (that is, the recognition result of the phoneme or the single sound) ) Is known.

Therefore, according to the syllable recognition device of the present invention, by knowing the progress of the syllable recognition process, the cause of the unconvergence of learning can be clarified and appropriately dealt with, so that the learning time can be shortened. Become. Further, by knowing the progress of the syllable recognition process, the cause of the erroneous recognition can be clarified and the correct action can be taken, so that a more correct recognition result can be obtained.

[Brief description of drawings]

FIG. 1 is a block diagram of an embodiment of the syllable recognition device of the present invention, and FIG. 2 is a diagram showing an example of acoustic parameters and teacher data for the syllable recognition device of FIG. 1 ... 1st NN, 2 ... 2nd NN, 3 ... delay element, 4,5,6,7
...... Input terminal.

Claims (1)

[Claims] 1. A combination of a time series of signals representing sequentially input acoustic parameters and a signal obtained by delaying the time series of signals representing the acoustic parameters by the time delay means by a time delay means. A first time-delay neural network for converting a signal into a time series of a signal representing a phoneme or a single tone and outputting the phoneme or a phoneme output from the first time-delay neural network. A time series of signals representing a single sound is sequentially input, and a time series of the signals sequentially representing the phonemes or the single sounds and a signal obtained by delaying the time series of the signals representing the phonemes or the single sounds for a predetermined time by the time delay means are described. A syllable recognition apparatus comprising a second time-delay neural network for converting a signal obtained by combining the signals into a time series of signals representing syllables and outputting the time-series signals.