JP3039408B2 - Sound classification method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention automatically classifies transient sounds (sounds that occur transiently in a short period of time when hitting an object, etc.) and sounds using a processing device. It is about the method.

[0002]

2. Description of the Related Art HMM (Hidden Markov Model) and NN (Neural Network) are methods for classifying the source of a transient sound and recognizing words spoken from speech.
Is widely known. Furthermore, HMM and N
In order to obtain a higher classification ability than using N alone,
A method of using both in combination has also been proposed.

FIG. 10 shows a configuration of a conventional sound classification system combining an HMM and a NN. To perform classification using this system, parameters included in the system must be determined in advance by learning. First, the learning process will be described below.

[0004] Sound data is converted into a feature vector sequence by appropriate preprocessing. Let I be the number of categories to be categorized. Hidden Markov model, HMM1, HM corresponding to each category using learning data belonging to each category
M2, HMM3,... HMMI are created. Next, each learning data is processed by all HMMs, and H indicating the maximum likelihood is obtained.
A feature vector sequence is created by using the MM to standardize the length of the feature vector sequence from which expansion and contraction has been removed. Finally, the NN is trained so as to output a desired classified output using the feature vector sequence obtained by standardizing the length obtained in this manner.

Next, the classification process will be described. First, the input sound is converted into a feature vector sequence, and the feature vector sequence is input to all HMMs. Next, a feature vector sequence whose length has been normalized using an HMM indicating the maximum likelihood is created. Finally, a length-normalized feature vector sequence is input to NN, and N
Classify sounds by N outputs.

FIG. 11 shows an NN of the conventional example. This is a three-layer perceptron, which receives a feature vector sequence whose length is standardized as input, and learns that only the output corresponding to the category to which the sound belongs becomes active.

However, in the conventional method, the HMM is used only for normalizing the length of the feature vector sequence.
The classification ability of MM is not being used effectively. Depending on the sound data, the HMM may show a more accurate classification result than the NN.
A system with higher categorization capability is desired.

[0008]

Problems with the conventional method are as follows.
Since the HMM is used only for normalization of the length of the feature vector sequence, and the classification capability of the HMM is used only for determining the HMM to be used for the standardization, the classification capability of the HMM is effective. It is not used for.

Depending on the type of sound, the HMM may show a more accurate classification result than the NN, and it is desired to effectively use the classification result by the HMM.

An object of the present invention is to provide a sound classification system having a higher classification capability by effectively utilizing the classification result by the HMM.

[0011]

According to the present invention, a hidden Markov model (HM) corresponding to a category to be classified is provided.
M) and a multi-layer perceptron having an output corresponding to the category to be categorized. The length (number of vectors) of the input feature vector sequence and the maximum likelihood of the hidden Markov model with respect to the input feature vector sequence are A feature vector sequence whose length has been standardized using the hidden Markov model shown,
A sound classification method is obtained in which a log likelihood of an input feature vector sequence of the Hidden Markov Model corresponding to each category is divided by the length of the input feature vector sequence as an input to the multilayer perceptron.

Further, according to the present invention, the length of the input feature vector sequence and the log likelihood of the Hidden Markov Model corresponding to each category with respect to the input feature vector sequence are divided by the length of the input feature vector sequence. The sound classification method is characterized in that the sound is directly input to the output layer of the multilayer perceptron without passing through the intermediate layer of the multilayer perceptron.

[0013]

According to the present invention, as shown in FIG. 1, in addition to the normalized feature vector sequence, the length of the feature vector sequence before normalization and the normalized log likelihood of each HMM are also input as NN inputs. In addition, the NN can automatically perform comprehensive classification in consideration of the classification result of the HMM. It should be noted that how to classify and determine the classifying ability of the NN and the HMM comprehensively is determined automatically and optimally by the learning of the NN.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of the first exemplary embodiment of the present invention.
First, any given input sound is converted into a feature vector sequence by a feature extractor. The feature vector sequence is
HMM1, HMM2, HMM3,... Corresponding to each category to be classified
The normalized log likelihood of the MM is determined. H that gives the likelihood having the maximum value (maximum likelihood) in the normalized log likelihood
Using the MM, the number of feature vectors included in the feature vector sequence, that is, the length of the feature vector sequence is standardized.
A sequence of feature vectors whose lengths are standardized,
The normalized log likelihood obtained from M and the length of the feature vector sequence before normalization are input to NN. The NN processes the input and outputs estimated classification results. Hereinafter, each part will be described in detail.

The feature extractor shown in FIG. 1 converts an input sound into a vector that expresses its features well, that is, a sequence of feature vectors. Although the present invention may use any feature vector, a typical feature vector is a power spectrum. FIG. 2 shows an example of feature extraction using a power spectrum. First, the input sound is sampled at intervals that do not impair its characteristics. The sampled input sound is divided into appropriate sections in the time direction. In the example of the figure, the section is divided into four sections from section 1 to section 4. The sections may overlap each other. Next, fast Fourier transform (F
FT), and the square of the absolute value of the discrete Fourier coefficient obtained as a result is obtained to obtain a discrete power spectrum corresponding to each section. A vector having the discrete power spectrum as a component is a feature vector x corresponding to the section i.
_i (i = 1 to 4), and the original input sound is converted into a sequence of feature vectors X = x ₁ x ₂ x ₃ x ₄ thus obtained.

The HMM is a model characterized by stochastic transition between states and stochastic output of a feature vector in each state. One HMM has a probability of being initially in state i ｛π _i
| I = 1 to N} and the transition probability {a from state i to state j}
_ij | i, j = 1 to N} and the probability density {b _i (x) | i = 1 to N} of outputting the feature vector x in the state i. Here, N is the number of states of the HMM.

Given feature vector sequence X = x ₁ x ₂ x
₃ ... Although probability i.e. the likelihood of x _T output by the HMM is a quantity proportional to the probability rather than the probability itself strictly being asked in obtained (where Viterbi algorithm shown in FIG. 3, the probability for simplicity I will call it.) In FIG. 3, a step V1 calculates the probability ξ ₁ (i) of outputting the feature vector x ₁ in the state i first, and initializing variables (formula 2) for holding the optimal state transition path. Is going. This step V1 is performed over all states (i = 1 to N). Step V2
Is a feature transition sequence that outputs a feature vector sequence x ₁ x ₂ x ₃ ... X _t and finds a state transition path that gives a maximum feature vector sequence output probability among the state transition paths, and ξ _t (i) is the output probability , Ψ _t (i) represent the state when x _t−1 is output on the path. Equation 1 shown below is the maximum value of the expression in [] when i is changed from 1 to N.

[0018]

(Equation 1) Equation 2 shown below is the value of i that gives the maximum value.

[0019]

(Equation 2) Step V2 is performed in all states (j = 1 to N), and is sequentially executed from the second feature vector to the last (Tth) feature vector (t = 2 to T). Step V3 is a feature vector sequence X = x ₁ x ₂ x
₃ ... output probability of the state transition path for outputting x _T is for claiming output probability P path or optimum state transition path which is the maximum. q _T is the final state of the optimal state transition paths. Step V4 is the first time the q _T, intended to follow the optimal state transition path in the reverse direction, whereby an optimal state transition path _{q 1 q 2 q 3 ... q} T is determined.

The HMM alone has a classification capability as follows. First, HMMs corresponding to each category to be classified are determined in advance by learning. Next, the probability (likelihood) that each HMM outputs an arbitrarily given feature vector sequence is determined by the Viterbi algorithm. Classification is performed by determining that the feature vector sequence belongs to the category corresponding to the HMM that gives the maximum likelihood among the likelihoods.

Next, the parameter {π _i |
i = 1 to N}, {a _ij | i, j = 1 to N}, {b
_The learning which is a method of determining _i (x) | i = 1 to N｝ will be described.

FIG. 4 is a schematic diagram showing an initial process of learning. First, a sufficient number of learning sound data corresponding to each category to be categorized such as “door knocking sound” and “shoe sound” (for example, even with the same knocking sound, tapping strength, door quality, temperature, humidity, etc.) , But these various data are used as knocking sounds and used as learning data of one category.) Next, each sound data is processed by the above-described feature extractor and converted into a learning feature vector sequence. These learning feature vector sequences are processed by the K-means algorithm shown in FIG. 5 to determine HMM parameters corresponding to each category. Hereinafter, the K-Means algorithm will be described.

First, feature vectors belonging to each category are classified into several clusters (clustering in FIG. 4). Here, the number of clusters in each category is set equal to the length of the longest learning feature vector sequence in that category (this generally differs for each category).
However, when the feature vector sequence is composed of T feature vectors, T is called the length of the feature vector sequence. Various methods can be applied to clustering of feature vectors,
A typical clustering method is described below.

First, N random feature vectors, each component of which is generated using random numbers, are created. Here, N is the length of the longest learning feature vector sequence. These vectors are called centroids, and one cluster is associated with each centroid. Next, the distance between the feature vector in the category and each centroid is calculated, and the feature vector is assumed to belong to the cluster corresponding to the shortest centroid. In this way, all feature vectors in a category can be classified into clusters. Here, the Euclidean distance is used as the distance between the vectors. Each cluster is numbered sequentially from 1 to N. In the K-means algorithm, the cluster is HM
It corresponds to the state of M on a one-to-one basis, and thus this number can be regarded as an index indicating the state of the HMM. After clustering in this manner, a state is uniquely associated with each feature vector, and therefore, one state transition path is associated with each feature vector sequence.

The above-described clustering in the initial stage of learning is the K-means algorithm step K1 in FIG. The K-means algorithm is applied independently for each category. FIG. 5 and the following description relate to one category. In FIG. 5, the number of clusters, that is, the number of states is N. Sum of Step K2, K3 means adding the feature vector x _t belonging to the cluster i. N _i is the number of feature vectors belonging to cluster i. Μ _{i in} step K2 is the average vector of cluster i. ^{T of} step K3 (
) Means transposing the vector in (), that is, converting a row vector to a column vector. Here, the feature vector _Xt is a row vector. V of step K3
_i is the variance matrix of cluster i. Step K4 is H
This gives a probability π _i that the initial state of the MM is i. The numerator on the right side of step K4 is the number of the feature vector sequence (for example, X ₁ X ₂ X ₃ ...) In the category in which the first feature vector (x ₁ ) belongs to the cluster i. Step K5 is for giving a transition probability a _ij from the state i to the state j. Right molecules, there is a feature vector x _t contained in cluster i, a feature vector x _{t + 1} to be subsequently included in the cluster j comes among one feature vector sequence (e.g., X ₁ X ₂ X ₃ ...) The number of such arrangements (t is arbitrary) is counted up, and the number thus obtained is added up for all the feature vector sequences. In this way, all states (i, j = 1 to
One HMM is specified by giving parameters μ _i , V _i , π _i , and a _ij regarding N). Note that the probability density of outputting the feature vector x in the state i is calculated by the following mathematical formula 3 assuming a Gaussian distribution.

[0026]

(Equation 3) Here, C is the dimension of the feature vector, and det (V _i ) is V
The determinant of _i , V _i ^-1 is the inverse matrix of V _i .

In step K6, the aforementioned Vite
The optimum state transition path of each learning feature vector sequence is obtained using the rbi algorithm. If this optimal state transition path is different from the state transition path given in step K1, re-clustering of the feature vectors is performed according to the optimal state transition path. That is, for any t, the feature vectors of each feature vector sequence are clustered such that the t-th feature vector x _t is classified into a cluster corresponding to the t-th state q _t of the optimal state transition path. Do it again. If re-clustering is not required in step K6, that is, the state transition path and Vite given in step K1 for all learning feature vector sequences
If the optimal state transition paths obtained by the rbi algorithm match, the K-means algorithm ends. If re-clustering has been performed in step K6, steps subsequent to step K1 in FIG. 5 are executed again. Next, when the process reaches step K6, if the optimum state transition path newly obtained therefrom is different from the previously obtained optimum state transition path, re-clustering is performed according to the newly obtained optimum state transition path, and step K1 in FIG.
Execute after If there is no re-clustering, the K-means algorithm ends. in this way,
The processing after step K1 is repeated until there is no re-clustering in step K6. When the K-means algorithm ends, the parameter μ that determines the HMM
_i , V _i , π _i , a _ij (i, j = 1 to N) are determined, and the HMM for the category is determined. When the HMMs corresponding to all the categories are obtained in this way, the HMM learning process ends.

Next, normalization of the length of the feature vector sequence by the HMM will be described. Excessive sounds and voices have varying lengths of sound from time to time. Since the NN (neural network) is vulnerable to such a change in pitch, the HMM
Is used to create a feature vector sequence in which the length from which the change in the sound length has been removed is standardized, and this is used as the input of the NN. The standardization of this length is performed as follows. First, an optimal state transition path of a given feature vector sequence is obtained by the Viterbi algorithm. Next, when there is a sequence in which the same state continues continuously in the optimum state transition path, the sequence and the corresponding feature vector sequence are replaced with an average vector of those vectors.

The HMM has the ability to classify to some extent even if the sound starts from the land of the original sound as a whole or is cut off halfway. In the case of such a missing feature vector sequence, comparing the optimal state transition path with the optimal state transition path of another feature vector sequence in the category, the missing part has a feature corresponding to the background sound. Complement the vector or feature vector where all components are zero. In this way, the lengths of all feature vector strings in the category are made uniform. Further, in order to make the lengths of the feature vector strings between different categories uniform, the lengths of the normalized feature vector strings of all the categories are adjusted to the length of the longest normalized feature vector string. Here, a feature vector corresponding to the background sound or a feature vector in which all components are 0 is supplemented at the beginning (or at the end) of the feature vector sequence for the insufficient length. In this way, the length of all feature vector sequences is made equal to the length of the longest standardized feature vector sequence for learning (hereinafter referred to as T _L ). The feature vector sequence thus formed is hereinafter referred to as a normalized feature vector sequence.

Next, the NN will be described. In the first embodiment of the present invention, a three-layer perceptron shown in FIG. 6 is used. The black circles in FIG. 6 indicate that the input is output as it is and distributed to each input in the next stage. 6 represent the neurons shown in FIG. FIG. 7 shows that the neuron j multiplies the input s _k (k = 1 to K) by weights of w _ik , respectively, and calculates the difference between the sum and the threshold value θ _j of the nonlinear function f (). As an argument, the nonlinear function f () at that time
Is used as the output _Sj . As this nonlinear function f (), a sigmoid function is usually used as shown in the following equation (4).

[0031]

(Equation 4) In the first embodiment, the normalized feature vector sequence, the normalized log likelihood of all HMMs, and the length of the feature vector sequence before normalization are input to NN. The normalized log likelihood is obtained by taking the logarithm of the likelihood P obtained by the Viterbi algorithm in FIG. 3 and dividing the logarithm by the length of the feature vector sequence before normalization. By taking the logarithm, N
The dynamic range of N inputs can be suppressed, and by dividing by the length of the feature vector sequence before standardization,
Equal likelihood comparison between feature vector sequences having different lengths becomes possible.

[0032] The dimension of the feature vector when is C, normalized feature vector sequence consists of components of total CT _L. Also,
Assuming that the number of classification categories is I, the total number K of inputs of the NN is CT _L + I + 1. The output layer of the NN is composed of a number I of neurons of a category, and is trained so that only the output corresponding to the category to which the feature vector sequence belongs becomes active (1), and the other outputs become inactive (0). You. The number J of the intermediate layer neurons is determined by gradually increasing the number of neurons while observing the classification result from a small number, and stopping when the desired classification ability is obtained.

Next, a back propagation algorithm (FIG. 8) which is a learning algorithm of the NN will be described. In FIG. 8, m is an index representing a learning sample (a serial number assigned to all the learning feature vector sequences by removing the category frame), k is an index representing a neuron of the input layer (input is output as it is), and j is an intermediate number. index representing the neuron layer, i is an index representing the neurons of the output layer, s _k ^m k-th component of the learning samples m, w _ij and theta _j is the weight and the threshold of the hidden neuron j, respectively, S _j ^m is learning samples m the corresponding hidden neuron j output, W _ij and theta _i is the weight and the threshold of each output layer neuron i, with L _i ^m is the i th component of the teacher output for learning samples m, and the learning samples Only the component corresponding to the category to which it belongs is 1 and the other components are 0. I is the number of output layer neurons (and thus the number of category categories), J is the number of hidden neurons, and K is the number of input layer neurons, CT _L + I + 1. The neuron parameters w _ik , θ _j , W _ij , and Θ _i give appropriate initial values by random numbers. In FIG. 8, steps B1 and B2
Is calculating the output of the hidden neuron j for the learning sample m, and this is calculated for all hidden neurons. In step B3, the argument of the nonlinear function f () in the output layer neuron i is calculated from the output of the intermediate layer neuron. Step B4 is to determine the parameters D _i ^m necessary for calculating the improved values of the parameters W _ij and theta _i of the output layer neurons. This parameter is calculated for all output layer neurons. Where f
() Is the derivative of f (). Step B5 is intended to determine the parameters d _j ^m needed to calculate the improvement of the parameters w _ik and theta _j of hidden neurons, calculate this for all the hidden neurons. The above processing is performed for all learning samples. Step B6 Step B7 weight parameters of the hidden neurons is intended to improve the threshold parameter of the hidden neurons respectively, the right-hand side of the w _ik, theta _j conventional value, is that the improved value of the left side. This improvement is made for all intermediate layer parameters. ε is a positive constant. Step B8 is to improve the weight parameter of the output layer neuron, and step B10 is to improve the threshold parameter of the output layer neuron.
W _ij and Θ _i on the right side are conventional values, and those on the left side are improved values. This improvement is made for all output layer parameters. If the difference between the conventional value and the improved value of all the parameters in the hidden layer and the output layer is sufficiently small, the learning is concluded as convergence, and if not, the algorithm is restarted from the beginning using the improved parameters. . The NN parameters are determined by the above back propagation algorithm. After the parameters of the HMM and the NN are determined by learning as described above, classification can be performed by the processing shown in FIG. Classification is performed according to the following procedure.

First, any given input sound is converted into a feature vector sequence by a feature extractor. Next, the feature vector sequence is processed by a Hidden Markov Model HMM1, HMM2, HMM3,... Corresponding to each category, and a normalized log likelihood is obtained. The length of the feature vector sequence is normalized using an HMM that gives the maximum value among the normalized log likelihoods. The normalized feature vector sequence, the normalized log likelihood obtained from each of the HMMs, and the length of the feature vector sequence before the length is normalized are input to NN. The NN processes the input and activates the output of the neuron corresponding to the category to which the sequence of feature vectors is estimated to belong. This is the classification result determined by the present system.

Next, the effect of the first embodiment of the present invention will be described. In general, the classification result depends on the type and length of the feature vector sequence, and the classification result of NN is correct or H
The classification result of MM is correct. In the first embodiment of the present invention, not only the normalized feature vector sequence but also the length of the feature vector sequence before normalization and the normalized log likelihood of each HMM are input to the NN, so that the NN is Judgments are made comprehensively in consideration of the HMM classification results. Regarding how to classify the NN and the HMM in total, the final classification is also determined by the learning of the NN. As a result, it is possible to provide a higher classification capability than in the conventional case where classification is performed only by NN or only by HMM.

Next, a second embodiment of the present invention will be described. In the second embodiment of the present invention, the NN shown in FIG. 9 is used as the NN in FIG. Ie HM
The normalized log likelihood of each HMM, which is the classification result of M, and the length of the feature vector sequence before normalization, which is a parameter having an important effect on how to consider the classification result of HMM and NN, are directly output to the output layer. To enter.

The learning algorithm is basically the same as the algorithm in the first embodiment shown in FIG.
The input of the normalized log likelihood of M and the length of the feature vector sequence before normalization is not included in the input layer neurons but in the hidden neurons. However, the neurons corresponding to the input of the normalized log likelihood of each HMM and the length of the feature vector sequence before the standardization output the same input as the input layer neuron as it is.

[0038] Accordingly, the number K of the input layer neurons Figure 8 is a second embodiment of the present invention is the total number of components CT _L of normalized feature vector sequence. Steps B3 and B
9 is replaced by J + I + 1, j = J + 1 to J
S _j ^{m at} + I + 1 is the normalized log likelihood of each HMM and the length of the feature vector sequence before normalization.

The second embodiment of the present invention has an effect that the learning time is shorter than that of the first embodiment, in addition to the effect of the first embodiment. This is because the length of the feature vector sequence before normalization and the normalized log likelihood of each HMM are directly connected to the output layer without passing through the hidden neurons, and therefore are smaller than in the first embodiment. This is because the number of hidden neurons is sufficient.

[0040]

The first effect is that a higher classification capability can be provided as compared with the conventional system.

The reason is that, in addition to the normalized feature vector sequence, the classification result by the HMM (normalized log likelihood of each HMM) and the length of the feature vector sequence before normalization are also input to the NN. This is because classification is performed by comprehensively considering the classification ability.

A second effect is that an optimum classification method can be automatically obtained by comprehensively considering the classification capabilities of the NN and the HMM.

The reason is that the optimal way of considering both is automatically acquired by learning of the NN.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a schematic diagram of feature extraction.

FIG. 3 is a flowchart of the Viterbi algorithm.

FIG. 4 is a conceptual diagram of an initial process of HMM learning.

FIG. 5 is a flowchart of a K-means algorithm.

FIG. 6 is a diagram showing a neural network according to the first embodiment of the present invention.

FIG. 7 is a conceptual diagram of the operation of a neuron.

FIG. 8 is a flowchart of a back propagation algorithm.

FIG. 9 is a diagram showing a neural network according to a second embodiment of the present invention.

FIG. 10 is a block diagram showing an embodiment of the related art.

FIG. 11 is a diagram showing a conventional neural network.

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification code FI G10L 15/28 G10L 3/00 531E 539 (56) References JP-A-9-198083 (JP, A) JP-A-8-54893 (JP, A) JP-A-8-254995 (JP, A) JP-A-8-44695 (JP, A) JP-A-7-219581 (JP, A) JP-A-2-298998 (JP, A) United Kingdom Patent application publication 2230370 (GB, A) French patent application publication 2645999 (FR, A1) (58) Fields investigated (Int. Cl. ⁷ , DB name) G10L 15/10 G10L 15/14 G10L 15/16 G10L 15/28 INSPEC (DIALOG) JICST file (JOIS) WPI (DIALOG) Practical file (PATOLIS) Patent file (PATOLIS)

Claims (2)

(57) [Claims] 1. A hidden Markov model (HMM) corresponding to a category to be categorized, and a multilayer perceptron having an output corresponding to the category to be categorized, the length of an input feature vector sequence (the number of vectors); A feature vector sequence whose length is normalized by using a hidden Markov model showing the maximum likelihood with respect to the input feature vector sequence among the hidden Markov models, and a logarithm of the hidden Markov model corresponding to each category with respect to the input feature vector sequence. A sound classification method wherein a value obtained by dividing likelihood by the length of an input feature vector sequence is used as an input of the multilayer perceptron. 2. A multi-layer perceptron, wherein the length of the input feature vector sequence and the log likelihood of the hidden Markov model corresponding to each of the categories for the input feature vector sequence divided by the length of the input feature vector sequence are: 2. The sound classification system according to claim 1, wherein the sound is directly input to the output layer of the multilayer perceptron without passing through the intermediate layer.