JPH0752439B2 - Neural network with dynamic programming function - Google Patents

Description

TECHNICAL FIELD The present invention relates to improvement of a neural network used in a pattern recognition system for speech or the like.

(Prior Art) A neural network, which is an information processing mechanism simulating a neural circuit of a living organism, is expected to be promising for solving various problems such as pattern recognition, and research and development are being actively conducted all over the world.

Regarding neural networks, an article was published in Nikkei Electronics magazine, No. 427, page 115 (published on August 10, 1987), entitled "Using neural networks for pattern recognition" knowledge processing (see reference (1) below). Call)
The explanation is easy to understand. According to it, the basic constituent element of the neural network is an element called a neuron unit (hereinafter abbreviated as a unit), which calculates a weighted sum of a plurality of inputs and generates an output corresponding to the weighted sum. The neural network has a multi-layer structure and typically has a three-layer structure of an input layer, an intermediate layer, and an output layer. The output of each unit of the input layer is transmitted to each unit of the intermediate layer and becomes an input to them. The output of each unit of the middle layer is transmitted to each unit of the output layer and becomes an input to them.

Taking speech recognition as an example, measurement and control in the input layer, a paper published as "Pattern matching method in speech recognition" on page 99 of Vol. 22 No. 1 (published 1st issue of 1983) ( A voice pattern as shown in FIG. 2 of reference (2)) is received as an input signal. Here, each element of the matrix of FIG. 2 is made to correspond to each unit of the input layer. Each unit of the output layer corresponds to each word of the recognition target vocabulary.
The word corresponding to the output layer unit showing the maximum output is the recognition result. In some cases, the output layer consists of one unit. Such a neural network as a whole,
The input pattern operates as a pattern matching means for calculating an amount (matching amount) indicating the certainty of being a certain word. The output of the output unit is the matching amount.

In order to effectively perform such recognition operation or pattern matching operation, it is necessary to previously learn the coefficient of the load given to the input of each unit. For this, the back propagation method described in detail in page 118 of the above-mentioned document (1) can be used.

(Problems to be Solved by the Invention) The neural network having the above-described configuration and function solves the pattern recognition problem that cannot be theoretically solved,
It is expected to exert a sophisticated function of learning and provide a solution that can be dealt with. However, the neural networks that have been proposed so far are of a very general form in that, as shown in FIG. 2 of document (1), completely free coupling is performed between units between layers. is there. It may therefore be versatile, but inefficient when dealing with a particular problem.

Considering an example of speech recognition, as described in Section 3.1 of Reference (2), there is a time-based expansion and contraction variation in the speech pattern. This variation has a complex non-linearity and produces a very large amount of distortion patterns. It is impossible in terms of calculation time to perform learning on all of these. However, if the learning is completed for a small amount of learning patterns, there is a risk of misrecognition when a distortion pattern of an unlearned type is input.

That is, the conventionally proposed neural network has a drawback that it is not suitable for processing expansion and contraction fluctuation, although it is general. The present invention proposes a neural network configuration having a dynamic programming function for the purpose of improving this point.

(Means for Solving Problems) A neural network according to the present invention comprises at least an input layer, an intermediate layer, and an output layer, and configures a unit array of the input layer and the intermediate layer and a connection between layers in time series, It has a function of associating the vector in the input pattern time series to each time on this time series in a relation that allows overlap and jump, and has a function to perform the processing to maximize the output of the output layer unit by dynamic programming. And

(Operation) The principle of the present invention will be described with reference to FIG. The neural network in FIG. 1 is composed of three layers: an input layer, an intermediate layer, and an output layer. The input layer has a time-series structure specified by time j, and the signal of the input cleaning is a vector. Indicate. This time series The whole is input to the input layer. The units of the input layer are arranged in a matrix of K × J to receive them. Let these be {input unit (k, j)} (3). The input unit (j) means a set of input units (k, j) at time j.

The intermediate layer is arranged in a matrix of L × J.

Let these be {intermediate unit (l, j)} (4). The term “intermediate unit (j)” means a set of intermediate units (l, j) at time j. Between these units, one or more input units in the vicinity of the same time j are connected to the intermediate unit (l, j). In the figure, two columns of the j-th column and the j-th column are connected.

Such a time-series configuration is a feature of the present invention. By these connections, the following weighting is applied when input is made from the input unit (k, j) to the intermediate unit (l, j).

β _0j (k, l) · x _kj β _1j (k, l) · x _kj-1 (5) Therefore, the sum of inputs to the intermediate unit (l, j) is Becomes As a result, the output y (l, j) = f (net (l, j)) (7) is generated from the intermediate layer unit (l, j). As the function f, a sigmoid function f (net (l, j)) = 1 / (1 + exp (−net (l, j) + θ (l, j)) (8) as shown in the equation (3) of the literature (1) (8) ) Can be used.The above expressions (5) to (7) are abbreviated as follows.

here And the product “·” of the vector means the inner product.

The output layer generally consists of (1,2 ... n ... N) N units. The operation will be described below by focusing on the unit (n).
The output y (l, l) of the intermediate unit (l, j) becomes the input. These are weighted by the coefficient ▲ α ⁿ _j ▼ (l) and the sum of them is added. Is calculated, and the product of this and the sigmoid function is the output Z (n) of the output unit (n).

Z (n) = 1 / (1 + exp (−Net (n) + Θ _n ) (11) Here, the notation is rearranged in the same manner as the equation (9).

here, Is.

As described above, a neural network suitable for handling a time-series pattern as shown in FIG. 2 of Document (2) can be constructed by taking a time-series configuration of the input layer and the intermediate layer. In particular, the configuration in which the input layer and the intermediate layer are connected to the intermediate layer unit at time j and a plurality of input units at times close thereto is a kind of matched filter to extract local features such as phonemes. It becomes a mechanism and is suitable for voice recognition.

The greatest feature of the present invention is that it has a mechanism when the combination of the intermediate unit and the input unit is optimally combined with the feature of the input pattern. Furthermore, it is characterized in that the processing for this purpose is performed by dynamic programming. The input pattern is now a K-dimensional vector Indicate. A mapping i = i between this time axis i and time axis j
(J) of the input pattern Of the input unit (j) Let us assume that the time axis conversion is performed.

This mapping learns the speech pattern of the word (n) and is abstracted and stored as _weighting factors {▲ α ⁿ _j ▼ (l), β _0j (k, l), β _1j (k, l)}. Try to best match what you have. That is, the equation (12), which is the output of the output unit (n), is maximized.

Since the Sigmoid function is monotonically increasing, in order to maximize Eq. (12), Net (n) in Eq. Should be the maximum. Substituting equation (9) into this And the following maximization problem results.

here Is a vector this Is input unit (j) To the input unit (j-1) Is the output generated in the output unit (j).

Is a quantity that is determined by giving a set of (n, i, j), If you put Can be simplified.

As a result of the above analysis, the problem of removing the time-axis distortion of the neural network having this time-series structure is to maximize the total sum of r (n, i, j) for i = i (j) as shown in equation (17). It was a problem and could be simplified. From this, it is known that the dynamic programming (dynamic programming) can be applied as in the case of the reference (2). An example of release is as follows.

(Initial condition) g (n, i, j) = r (n, 1,1) (18) (Recursion formula) (I = 1,2, ... I; j = 1,2 ... J) As a result, the output of the output unit (n) is determined as Z (n) = H (g (n, I, J)) (20).

The calculation of equation (19) is performed in the normal order on the grid points in FIG. Between i (j-1) and i (j), three degrees of freedom with increments of 0, 1, 2 are allowed. Of these, an increase of 0 means staying on the i-axis, and an increase of 2 means passing by one point. Such correspondence of the degrees of freedom is called "correspondence that allows the input pattern vector to be stopped or overtaken".

(Embodiment) FIG. 3 shows an embodiment in which the present invention is applied to voice recognition.
The present embodiment is an example in which the operation of the present invention described in FIGS. 1 and 2 is executed by a microprocessor.

The input pattern buffer 10 holds an input pattern as shown in equation (13), and the intermediate layer processing unit 20 Is accessible. This part is normal
Composed of RAM. The intermediate layer coefficient storage unit 30 stores a group of _weighting factors {β _0j (k, l), β _1j (k, l)} and a threshold group {θ (l,
j)} is stored.

The intermediate layer processing unit 20 including a microprocessor performs the following processing.

These are (14) and (15) are described in relation to the input unit (j). To the input unit (j-1) Is the output that occurs in the intermediate unit (j) when they are associated with each other. this Are calculated for the required (i, j) pairs and stored in the hidden layer output buffer 40.

In the dynamic programming processing unit 50 including a microprocessor, a group of coefficients stored in the output layer coefficient storage unit {▲ α ⁿ _j ▼
(L)}, the threshold value group {Θ (n)} and the intermediate layer output stored in the intermediate layer output buffer 40 (18), (1
Perform the dynamic programming operation of Eqs. 9) and 20). R (n, i, j) in the equation (19) is calculated by the following equation.

Thus, g (n, I, J) is obtained by advancing the recurrence formula calculation of the formula (19) while calculating r (n, i, j). s
Paying attention to the fact that the igmoid function is monotonically increasing, sigmoid
The function is omitted and Z (n) = g (n, I, J) −Θ (n) (23) is output.

This dynamic programming process is performed for each of the output units (n = 1, 2, ... N), and the output layer output z (n) is output to each of them.

The determination processing unit 70 compares the output layers z (n) with each other and outputs n = which maximizes z (n) as a recognition result.

The principle and operation of the present invention have been described above based on the embodiments, but these descriptions do not limit the scope of rights of the present application. For example, a configuration is conceivable in which the intermediate layer processing is performed not by the microprocessor but by the distributed processor as it is in FIG. Further, according to the present invention, it is possible to combine the neural network and the dynamic programming method. In response to this, various ideas in the field conventionally studied as the DP matching method in the field of the speech recognition can be applied to the dynamic neural network of the present invention. But I was able to adopt it.
For example, the method of "matching window" shown in FIG.
"Nikkei Electronics Magazine, No. 329 (November 7, 1983)
Tilt limitation as described in "Table 1", page 184
"Pattern Understanding" edited by Yoshiaki Shirai (Ohmsha, 1987, 8)
Issued on the 25th of the month) ”, page 210, continuous DP method, etc. These incorporations are within the scope of the claimed invention.

(Effect) According to the configuration described above, the ability to correct the time axis distortion is provided, so that good recognition performance can be obtained even when learning is performed with a small number of learning patterns. Moreover, the processing can be executed at high speed by adopting the dynamic programming.

[Brief description of drawings]

 1 and 2 are explanatory views of the principle, and FIG. 3 is an embodiment. In the figure, 10 ... input pattern buffer, 20 ... intermediate layer processing unit, 30 ... intermediate layer coefficient storage unit, 40 ... intermediate layer output buffer, 50 ... dynamic programming processing unit, 60 ... output layer Coefficient storage unit, 70 ... Judgment unit.

Claims (1)

[Claims] 1. A neural network comprising at least an input layer, an intermediate layer, and an output layer for recognizing an input time-series pattern such as voice, in which neural units of the input layer and the intermediate layer are arranged along a time axis, The connection between the layers is configured in time series, which is limited to only the neighborhood in terms of time, and the vectors in the input pattern time series are input in correspondence with each time on this time series in a relationship that allows stopping and overtaking. When obtaining the output of the output layer unit, a neural network having a dynamic programming function, which has a function of searching for a correspondence that maximizes the output of the output layer unit by using a dynamic programming method.