JPH0752438B2 - Dynamic programming neural network - Google Patents

Description

TECHNICAL FIELD The present invention relates to improvement of a neural network used in a pattern recognition system for speech or the like.

(Prior Art) A neural network, which is an information processing mechanism simulating a neural circuit of a living organism, is expected to be promising for solving various problems such as pattern recognition, and research and development are being actively conducted all over the world.

Regarding the neural network, an article published in Nikkei Electronics, No. 427, page 115 (published on August 10, 1987) entitled "Using Neural Networks for Pattern Recognition and Knowledge Processing" (see (1 ))) Is explained in an easy-to-understand manner. According to it, the basic constituent element of the neural network is an element called a neuron unit (hereinafter abbreviated as a unit), which calculates a weighted sum of a plurality of inputs and generates an output according to the weighted sum. The neural network has a multi-layer structure and typically has a three-layer structure of an input layer, an intermediate layer, and an output layer. The output of each unit of the input layer is transmitted to each unit of the intermediate layer and becomes an input to them. The output of each unit of the middle layer is transmitted to each unit of the output layer and becomes an input to them.

Taking speech recognition as an example, in the input layer, a paper published as "Pattern matching method in speech recognition" on page 99 of Measurement and Control Vol. 22 No. 1 (published No. 1 in 1983) (hereinafter A voice pattern as shown in FIG. 2 of reference (2)) is received as an input signal. Here, each element of the matrix of FIG. 2 is made to correspond to each unit of the input layer. Each unit of the output layer corresponds to each word of the recognition target vocabulary.
The word corresponding to the output layer unit showing the maximum output is the recognition result. In some cases, the output layer consists of one unit. Such a neural network as a whole,
The input pattern operates as a pattern matching means for calculating an amount (matching amount) indicating the certainty of being a certain word. The output of the output unit is the matching amount.

In order to effectively perform such recognition operation or pattern matching operation, it is necessary to previously learn the coefficient of the load given to the input of each unit. For this, the back propagation method described in detail in page 118 of the above-mentioned document (1) can be used.

(Problems to be Solved by the Invention) The neural network having the above-described configuration and function solves the pattern recognition problem that cannot be theoretically solved,
It is expected to exert a sophisticated function of learning and provide manageable liberation. However, the neural networks proposed up to now have a very general form in that completely free engagement is performed between units as shown in FIG. 2 of document (1). Is. It may therefore be versatile, but inefficient when dealing with a particular problem.

Considering an example of speech recognition, as described in Section 3.1 of Reference (2), there is a time-based expansion and contraction variation in the speech pattern. This variation has a complex non-linearity and produces a very large amount of distortion patterns. It is impossible in terms of calculation time to perform learning on all of these. However, if the learning is completed for a small amount of learning patterns, there is a risk of misrecognition when a distortion pattern of an unlearned type is input.

That is, the conventionally proposed neural network has a drawback that it is not suitable for processing expansion and contraction fluctuation, although it is general. The present invention proposes a neural network configuration having a dynamic programming function for the purpose of improving this point.

(Means for Solving Problems) A dynamic programming neural network according to the present invention comprises at least an input layer, an intermediate layer, and an output layer, and configures a unit array of the input layer and the intermediate layer and an interlayer connection in time series. , It has a function of associating the vector in the input pattern time series with each time on the input layer time series in a relationship that allows staying and overtaking, and has the function of performing the processing that maximizes the output of the output layer unit by dynamic programming. Characterize.

(Operation) The principle of the present invention will be described with reference to FIG. The neural network shown in FIG. 1 is composed of three layers: an input layer, an intermediate layer, and an output layer. The input layer has a time-series structure specified by time j, and the signal on its input line is a vector. Indicate. This time series The whole is input to the input layer. The units of the input layer are arranged in a matrix of K × J to receive them. Let these be {input unit (k, j)} (3). The input unit (j) means a set of input units (k, j) at time j.

The intermediate layer is arranged in a matrix of L × J.

Let these be {intermediate unit (l, j)} (4). The term “intermediate unit (j)” means a set of intermediate units (l, j) at time j. Between these units, one or more input units in the vicinity of the same time j are connected to the intermediate unit (l, j). In the figure, two columns of the j-th column and the j-th column are connected.

Such a time-series configuration is a feature of the present invention. By these connections, the following weighting is applied when input is made from the input unit (k, j) to the intermediate unit (l, j).

β _0j (k, l) · x _kj β _1j (k, l) · x _kj-1 (5) Therefore, the sum of inputs to the intermediate unit (l, j) is Becomes As a result, the output y (l, j) = f (net (l, j)) (7) is generated from the intermediate layer unit (l, j). As the function f, a sigmoid function f (net (l, j)) = 1 / (1 + exp (−net (l, j) + θ (l, j)) (8) as shown in the equation (3) of the literature (1) (8) ) Can be used.The above expressions (5) to (7) are abbreviated as follows.

here And the product “·” of the vector means the inner product.

The output layer generally consists of (1,2 ... n ... N) N units. The operation will be described below by focusing on the unit (n).
The output y (l, 1) of the intermediate unit (l, j) becomes the input. These are loaded with the coefficient ▲ α ⁿ _j ▼ (l) and their sum Is calculated, and the product of this and the sigmoid function is the output Z (n) of the output unit (n).

Z (n) = 1 / (1 + exp (−Net (n) + Θ _n ) (11) Here, the notation is rearranged in the same manner as the equation (9).

here, Is.

As described above, a neural network suitable for handling a time-series pattern as shown in FIG. 2 of Document (2) can be constructed by taking a time-series configuration of the input layer and the intermediate layer. In particular, the configuration in which the input layer and the intermediate layer are connected to the intermediate layer unit at time j and a plurality of input units at times close thereto is a kind of matched filter to extract local features such as phonemes. It becomes a mechanism and is suitable for voice recognition.

The greatest feature of the present invention is that it has a mechanism for optimally combining an input unit with a vector in an input pattern time series. Furthermore, it is characterized in that the processing for this is performed by dynamic programming.

The input pattern is now a K-dimensional vector Indicate. Between the time axis i and the time axis j of the input unit, the mapping i = i (j) Of the input unit (j) The time axis conversion of This mapping learns the speech pattern of the word (n) and calculates the weighting factor {▲ α
ⁿ _j ▼ (l), β _0j (k, l), β _1j (k, l)} are matched so that they best match what is abstracted and stored. That is, the output of the output unit (n) (1
2) It is done so that the formula becomes maximum.

Since the sigmoid function is monotonically increasing, in order to maximize Eq. (12), Net (n) in Eq. Should be the maximum. Substituting equation (9) into this And the following maximization problem results.

here Is a vector This vector (i, j) is input to the input unit (j) To the input unit (j-1) Is the output generated in the output unit (j).

Is a quantity that is determined by giving a set of (n, i (j), i (j-1), j). If you put Can be simplified.

As a result of the above analysis, the problem of time-axis distortion correction of the neural network having this time-series structure is that the sum of r (n, i (j), i (j-1), j) is expressed by the equation (17). I = i
With respect to (j), we have a problem and a formulation to maximize it. From this, it is known that dynamic programming (dynamic programming) can be applied as follows. An example of release is as follows.

(Initial condition) (Recurrence formula) (I = 1,2, ... I; j = 1,2 ... J) As a result, the output of the output unit (n) is determined as Z (n) = H (g (n, I, J)) (20).

The calculation of the equation (19) is carried out in the regular order in the lattice points shown in FIG. Between i (j-1) and i (j), three degrees of freedom with increments of 0, 1, 2, are allowed. Of these, an increase of 0 means staying (overlapping) on the i-axis, and an increase of 2 means passing by one point. Such correspondence of the degrees of freedom is called "correspondence that allows the input pattern vector to be stopped or overtaken".

(Embodiment) FIG. 3 shows an embodiment in which the present invention is applied to voice recognition.
This is an example in which the principle of the present invention described in FIGS. 1 and 2 is executed by a microprocessor.

The input pattern buffer 10 holds an input pattern as shown in equation (13), and the intermediate layer processing unit 20 Is accessible. This buffer is RA
Composed of M. In the intermediate layer coefficient storage unit 30, the _weighting factor groups {β _0j (k, l)}, {β _1j (k, l)} and the threshold value group {θ (l,
j)} is stored. The intermediate layer processing unit 20 including a microprocessor calculates the following three systems of intermediate layer output groups.

of Is the same as the input unit (j) Is the output of the intermediate unit (j) when is associated with the input unit (j-1). of To the input unit (j), Is the output of the intermediate layer (j) when is associated with the input unit (j-1). of To the input unit (j), Is the output of the intermediate layer (j) when is associated with the input unit (j-1). These are calculated for the required (i, j) set and output to the intermediate layer output buffer 40 for storage.

In the dynamic programming method processing unit 50 composed of a microprocessor, a group of coefficients stored in the output layer coefficient storage unit {▲ α ⁿ _j ▼
(L)}, the threshold value group {Θ (n)} and the intermediate layer output stored in the intermediate layer output buffer 40 (18), (1
Perform the dynamic programming operations in 9) and (20). R (n, i, i, j) and the like on the right side of the equation (19) are calculated and used as follows based on the intermediate layer outputs of the equations (21-a) to (21-c).

While calculating r (n, i, i, j) etc. by these, g (n, I, J) is obtained by advancing the recurrence formula calculation of the formula (19). Paying attention to the fact that the sigmoid function is monotonically increasing,
This is omitted, and Z (n) = g (n, I, J) −Θ (n) (23) is set as the output of the output unit (n).

The above-mentioned dynamic programming processing is an output unit (n = 1,2, ... N)
, And the output layer output z (n) is output for each.

The determination processing unit 70 mutually compares the output layer outputs Z (n) output from the dynamic programming processing 50, and outputs n = that maximizes Z (n) as a recognition result.

The principle and operation of the present invention have been described above based on the embodiments, but these descriptions do not limit the scope of rights of the present application. For example, the middle layer processing does not depend on the microprocessor,
A configuration in which the distributed processor is used as it is in FIG. 1 is also possible. Further, according to the present invention, it is possible to combine the neural network and the dynamic programming method. In response to this, various ideas in the field conventionally studied as the DP matching method in the field of the speech recognition can be applied to the dynamic neural network of the present invention. But I was able to adopt it. For example, the method of “matching window” shown in FIG. 5 of the reference (2), page 184 of Nikkei Electronics magazine, No. 329 (published on November 7, 1983), tilt limitation described in Table 1, etc. It is a technique. These incorporations are within the scope of the claimed invention.

(Effect) According to the configuration described above, since the time axis distortion can be corrected, good recognition performance can be obtained even when learning is performed with a small number of learning patterns. Moreover, the processing can be executed at high speed by adopting the dynamic programming.

[Brief description of drawings]

 1 and 2 are explanatory views of the principle, and FIG. 3 is an embodiment. In the figure, 10 ... input pattern buffer, 20 ... intermediate layer processing unit, 30 ... intermediate layer coefficient storage unit, 40 ... intermediate layer output buffer, 50 ... dynamic programming processing unit, 60 ... output layer Coefficient storage unit, 70 ... Judgment unit.

Claims (1)

[Claims] 1. A neural network comprising at least an input layer, an intermediate layer, and an output layer for recognizing an input time-series pattern such as voice, in which neural units of the input layer and the intermediate layer are arranged along a time axis, The connections between the layers are configured in time series, which is limited only to the neighborhood in terms of time, and the vectors in the time series of the input pattern are input to the neural units in the input layer in association with each other in a relationship that allows stopping and overtaking. A dynamic programming neural network having a function of searching a correspondence that maximizes the output of the output layer unit by using a dynamic programming method when obtaining the output of the output layer unit.