JPH0583920B2 - - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to improvement of a neural network used in a pattern recognition system for speech, etc.

(Prior Art) Neural networks, which are information processing mechanisms that simulate biological neural circuits, are expected to be promising for solving various problems such as pattern recognition, and research and development are being actively conducted around the world.

Regarding neural networks, see Nikkei Electronics Magazine, No. 427, page 115 (August 1988).
This is explained in an easy-to-understand manner in an article (hereinafter referred to as Reference (1)) published in ``Using Neural Networks for Pattern Recognition and Knowledge Processing'' (published on the 10th). According to this, the basic component of a neural network is an element called a Neuron unit (hereinafter abbreviated as the unit), which calculates the weighted sum of multiple inputs and generates an output accordingly. A neural network has a multilayer structure, typically consisting of three layers: an input layer, an intermediate layer, and an output layer. The output of each unit in the input layer is transmitted to each unit in the intermediate layer and becomes an input to them. The output of each unit in the intermediate layer is transmitted to each unit in the output layer and becomes an input to them.

Taking speech recognition as an example, in the input layer, there is a
A speech pattern as shown in FIG. 2 of a paper published on page 99 entitled "Pattern matching method in speech recognition" (hereinafter referred to as document (2)) is received as an input signal. Here, each element of the matrix shown in FIG. 2 is made to correspond to each unit of the input layer. Each unit of the output layer corresponds to each word of the recognition target vocabulary. The word corresponding to the output layer unit showing the maximum output becomes the recognition result. There is also a case where the output layer consists of one unit. As a whole, such a neural network has an amount that indicates the probability that an input pattern is a certain word (matching amount).
It operates as a pattern matching means to calculate. The output of the output unit becomes the matching amount.

In order for such a recognition operation or pattern matching operation to function effectively, it is necessary to learn in advance the weight coefficients to be applied to the inputs of each unit. For this purpose, the back propagation method detailed on page 118 of the above-mentioned document (1) can be used.

(Problem to be solved by the invention) A neural network with the above configuration and functions can solve pattern recognition problems that are theoretically impossible to solve by exhibiting an advanced function called learning. It is expected that this method will provide a solution. However, as shown in Figure 2 of Reference (1), the neural networks that have been proposed to date are extremely general in that there is completely free coupling between units between layers. It is something. Therefore, although it may be versatile, it is inefficient when dealing with specific problems.

Considering the example of speech recognition, as described in Section 3.1 of Reference (2), there are variations in the expansion and contraction of the time axis in speech patterns. This variation has complex nonlinearity and generates an extremely large amount of distortion patterns. Learning for all of these objects is impossible in terms of computational time. However, if the learning is completed for a small number of learning patterns, there is a risk that erroneous recognition will occur when a distortion pattern of a type that has not been learned is input.

That is, although the neural networks proposed in the past are general, they have the disadvantage that they are not suitable for processing expansion/contraction fluctuations. The present invention proposes a configuration called a dynamic neural network in order to improve this point.

(Means for Solving the Problems) The dynamic neural network according to the present invention consists of at least an input layer, a middle layer, and an output layer, and the arrangement of the units of the input layer and the middle layer and the connections between the layers are arranged in a chronological order. It is characterized by having a means for switching the temporal correspondence between these and the input pattern so that the output of the output unit is maximized.

(Operation) The operation and principle of the present invention will be explained with reference to FIG. The neural network in Figure 1 has an input layer,
It consists of three layers: an intermediate layer and an output layer. The input layer is at time j
The signal on the input line is expressed by the vector x _j = (x _1j , ..., x _kj ...x _kj ) (1). This time series X = (x ₁ x ₂ x _j ...x _J ) (2) The entire time series becomes the input to the input layer. The units in the input layer are arranged in K×J rows and columns to accommodate these. Let these be {input unit (k, j)} (3). The set of input units at time j is denoted as input unit (j).

The intermediate layer is arranged in L×J rows and columns. Let these be {intermediate unit (l, j)} (4). The set of intermediate units at time j is denoted as intermediate unit (j). Among these units, one or more columns of input units near the same time j are connected to the intermediate unit (l, k).
In the figure, two columns, the j-th column and the j-1th column, are connected.

Such a chronological structure is a feature of the present invention. These connections form the input unit (k,
When an input is made from unit (l, j) to intermediate unit (l, j), the following loading is performed.

β _pj (k, l) x _kj β _1j (k, l) x _kj-1 (5) Therefore, the sum of inputs to the middle layer unit (l, j) is net (l, j) = 〓 ^k β _0j (k, l) x _kj + 〓 ^k β _1j (k, l) x _kj-1 (6). This allows the middle layer unit (l,j)
The output y(l,j)=f(net(l,j)) (7) is generated. The function f is a sigmoid function as shown in equation (3) of literature (1) f(net(l,j))=1/(1+exp(−net(l,j))
j)+θ(l,j)) (8) can be used. The above equations (5) to (7) will be abbreviated as follows.

y(l,j)=h _lj (〓 _0j (l)・x _j +〓 _lj (l)・
x _jl ) (9) Here〓 _0j (l) = (β _0j (1, l),…, β(K,
l), 〓 _1j (1)=(β _1j (1, l), ..., β _1j (K, l
), and the product “・” of a vector and a vector means an inner product.

The output layer generally consists of N units (1, 2...n...N). The operation will be explained below, focusing on unit (n). The output y(l,1) of the intermediate unit (l,j) becomes the input. These have coefficients α ⁿ _j
(1) is loaded, their sum Net(n) = 〓 ^j α ⁿ _j (1)・y(l,j) (10) is calculated, and this is multiplied by the sigmoid function to obtain the output unit (n ) is the output Z(n).

Z(n)=1/(1+exp(−Net(n)+Θ _o ) (11) Here, the notation will be rearranged in the same manner as in equation (9).

Z(n)=H _o ( 〓 ^j 〓 _j・y(j)) (12) Here, 〓 ⁿ _j = [α ⁿ _j (1)…α ⁿ _j (L), y(j)=
(y
(1, j)...y(L, j))].

By configuring the input layer and the intermediate layer in a time-series manner as described above, it is possible to construct a neural network suitable for handling a time-series pattern as shown in FIG. 2 of Reference (2). In particular, the connection between the input layer and the middle layer is such that a middle layer unit at time j is connected to multiple input units at nearby times, forming a kind of matched filter, which extracts local waits such as phonemes. This mechanism is suitable for voice recognition.

The most important feature of the present invention is that it has a mechanism for optimally combining the combination of intermediate unit and input unit with the characteristics of the input pattern. Furthermore, the present invention is characterized in that processing for this purpose is performed by dynamic programming. The input pattern is now expressed as a K-dimensional vector 〓 _i = (a _1i ... a _ki ... a _Ki ) time series A = a ₁ a ₂ ... a _i ... a _I (13). Mapping j= between this time axis i and time axis j
In j(i), a time axis conversion is performed in which the input pattern vector a _i is set as the input x _j of the input unit (j). This mapping is performed so as to best match what is learned as the speech pattern of word (n) and stored as the value of the weighting coefficient. In other words, this is done so that the output of output unit (n), expressed by equation (12), is maximized. (12) in formula
Since the sigmoid function is word increasing, equation (10) is
It is known that it is sufficient to maximize the value in brackets [ ] in equation (12), which is Net(n). Substituting equation (9) into this, x _j
If we perform the permutation =a _i (j=j(i)), we will end up with the following maximum problem.

max j=i(x) 〓 〓 ⁱ 〓 _j・(h _1j (〓 _0j (1)・a _i +〓 _1j (1)・a _i-1 ),…
,h _Lj (〓 _0j (L)・a _i +〓 _1j (L)・
a _i-1 )) (14) In this equation (14), the individual values in Σ on the right side are determined by giving the combination of (n, i, j), so r(n, i, j) It will be written as.

r (n, i, j) = 〓 ⁿ _j・(h _1j (〓 _0j (1)・a _i +〓 _1j
(1)・a _i-1 ),…, h _1j (〓 _0j (1)・a _i +〓 _1j (1)・a _i ),
…
…h _Lj (〓 _0j (L)・a _i +〓 _1j (L)・a _i )) (15) Accordingly, equation (14) becomes max j=i(x) _i=1 〓 ^I r(n , i, j) (16).

As a result of the above analysis, the problem of removing time axis distortion in a neural network with this time series structure is as shown in equation (16), where j = j
Regarding (i), the problem can be simplified to the maximization problem.
From this, it is known that dynamic programming can be applied, similar to literature (2).

An example of the procedure for calculating equation (16) using dynamic programming is as follows.

(Initial condition) g (n, 1, 1) = r (n, 1, 1) (17) (Recurrence formula) g (n, i, j) = r (n, i, j) + max
g(n,i-1,j) g(n,i-1,j-1) g(n,i-1,j-2) (i=1,2...I;j=1,2...J ) (18) As a result, the output of output unit (n) is determined as Z(n)=H(g(n, I, J)) (19). The calculation of equation (18) is performed on the grid points in FIG. 2 in positive order. Three degrees of freedom with increments of 0, 1, and 2 are allowed between j(i-1) and j(x). Among these, an increase of 0 means that the point stays (overlaps) on the j-axis, and an increase of 2 means that only one point is overtaken. This correspondence of degrees of freedom is expressed as a correspondence that allows the input unit to stop and jump.

(Embodiment) FIG. 3 shows an embodiment in which the present invention is applied to speech recognition. This embodiment is an example in which the operation of the present invention explained in FIGS. 1 and 2 is executed by a microprocessor.

The input pattern buffer 10 holds an input pattern such as equation (13), and the vector a _i can be accessed by the intermediate layer processing section 20. This part consists of regular RAM.

The intermediate layer coefficient storage unit 30 stores a group of weight coefficients {β _0j
(k, l), β _1j (k, l)} are stored. With reference to this, the following processing is performed in the intermediate layer processing section 20 consisting of a microprocessor.

y^(i,j,l)=h _lj (〓 _0j (l)・a _i +〓 _lj
(l)・a _i-1 ) (20) These appear on the right side of equation (14), and if a _i is input to input unit (j) and input unit (j-1) is input,
This is the output that occurs in the intermediate unit (j) when a _i-1 is associated. This output y^(i, j, l) is calculated for the necessary pair of (i, j) and stored in the intermediate layer output buffer 40.

The dynamic programming processing section 50 consisting of a microprocessor stores a group of coefficients {α ⁿ _j (l)} stored in the output layer coefficient storage section 60 and the intermediate layer output buffer 40.
While referring to the output y^(i, j, l) stored in
Perform dynamic programming operations of equations (17), (18), and (19). r(n, i, j) in equation (18) is calculated by the following equation.

r(n,j,l)=α ⁿ _j・(y(i,j)(21) where y(i,j)=(y^(i,j,l),...,y^
(i, j, L)). As a result, g(n, I, J) can be obtained by proceeding with the recurrence formula calculation of equation (18) while calculating r(n, i, j). Taking note of the fact that the sigmoid function is monotonically increasing, this embodiment directly outputs Z(n)=g(n, I, J)-Θ(n) (22). This dynamic programming process is performed for each output unit (n=1, 2, . . . N), and an output layer output Z(n) is output for each output unit.

The determination processing section 70 mutually compares the output layer outputs Z(n) output from the dynamic programming processing section, and determines that Z
n=n^, where (n) is the maximum, is output as the recognition result.

Although the principle and operation of the present invention have been explained above based on examples, these descriptions do not limit the scope of the rights of the present application. For example, a configuration in which the intermediate layer processing is performed not by a microprocessor but by a distributed processor as shown in FIG. 1 is also conceivable. Furthermore, the present invention has made it possible to combine neural networks and dynamic programming, and in response, the present invention has developed various devices that have been studied as DP matching methods in the field of speech recognition. It is now possible to use this technology in dynamic neural networks. For example, the "matching window" method shown in Figure 5 of Reference (2), "Nikkei Electronics Magazine, No. 329"
No. (issued November 7, 1981), page 184, Table 1.”
This is a technique such as the slope restriction described in . These incorporations are within the scope of the present invention.

(Effects) According to the configuration described above, since it has the ability to correct time axis distortion, good recognition performance can be obtained even when learning is performed using a small number of learning patterns. Moreover, the process can be executed at high speed by using dynamic programming.

[Brief explanation of the drawing]

1 and 2 are principle explanatory diagrams, and FIG. 3 is an embodiment. In the figure, 10...input pattern buffer,
20... Middle layer processing section, 30... Middle layer coefficient storage section, 40... Middle layer output buffer, 50... Dynamic programming processing section, 60... Output layer coefficient storage section, 70
... Judgment department.

Claims (1)

[Claims] 1. In a neural network consisting of at least an input layer, a middle layer, and an output layer, the arrangement of the neural units of the input layer and the middle layer and the connections between the layers are configured in a time-series manner, and A dynamic neural network characterized by having means for switching the temporal correspondence with an input pattern so that the output of an output layer unit is maximized. 2. A dynamic neural system according to claim 1, characterized in that the means for determining the temporal correspondence includes a mechanism for calculating by dynamic programming using the sum of inputs to the output layer unit as an objective function. network. 3 If the above temporal correspondence is j = j (i), j is the time on the neural network side, and i is the time of the input pattern, then j for j (i-1)
The dynamic neural network according to claim 1, characterized in that (i) allows a stopping and overtaking relationship.