JPH0583919B2 - - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to improvements in neural networks used in time-series pattern recognition systems for speech and the like.

(Conventional technology) Neural networks, which are information processing mechanisms that simulate biological neural circuits, are considered promising for solving various problems such as pattern recognition, and have been actively researched and developed around the world in recent years. .

Regarding neural networks, Nikkei Electronics Magazine, No. 427, (published August 10, 1986)
This is explained in an easy-to-understand manner in the commentary published on pp. 115-124 entitled ``Using neural networks for pattern recognition and knowledge processing'' (hereinafter referred to as Document (1)). According to this, the basic component of a neural network is an element called a neuron unit, which calculates the weighted sum of multiple inputs and generates an output accordingly. A neural network has a multilayer structure, typically consisting of three layers: an input layer, a middle layer, and an output layer.

The output of each neural unit in the input layer is transmitted to each neural unit in the intermediate layer and becomes an input to them. The output of each neural unit in the intermediate layer is transmitted to each neural unit in the output layer,
It becomes an input to them.

Taking speech recognition as an example, in the input layer, there is a method described in "Measurement and Control" magazine, Volume 22, No.
A speech pattern as shown in FIG. 2 of the paper entitled "Pattern Matching Method in Speech Recognition" published on pp.99-105 (hereinafter referred to as Document (2)) is received as an input signal. Here, each element of the matrix shown in FIG. 2 is made to correspond to each neural unit of the input layer. For each neural unit in the output layer, the recognition result is a word corresponding to the neural unit from which the maximum output corresponding to each word of the recognition target vocabulary is obtained.

In order for such a recognition operation to function effectively, it is necessary to learn in advance the coefficients of the weights to be applied to the inputs of the neural unit. For this purpose, the back propagation method detailed on page 118 of the above-mentioned document (1) can be used.

(Problems to be Solved by the Invention) A neural network with the above configuration and functions can exhibit advanced learning functions for pattern recognition problems that cannot be formulated theoretically. It is hoped that this will provide a manageable solution. However, as shown in Figure 2 of literature (1),
Neural networks that have been proposed to date are of extremely general form in that neural units between layers are freely coupled. Therefore, although it may be versatile, it is inefficient when dealing with special problems.

Consider the example of speech recognition as a representative example of handling time-series patterns. As described in Section 3.1 of Reference (2), there is expansion and contraction of the time axis in speech patterns. This expansion and contraction has complex nonlinearity and produces an extremely large amount of distortion patterns. Learning for all of these objects is impossible in terms of computational time.

From the above examples, it can be seen that pattern learning requires a large number of patterns. However, it is clear that the recognition rate decreases as the number of learning patterns decreases.

That is, although the neural networks proposed in the past are general, they have the disadvantage that they are not suitable for processing variations in expansion and contraction distortion.
The present invention aims to improve this by realizing a dynamic neural network configuration.

(Means for Solving the Problems) The dynamic neural network according to the present invention configures lower layers close to the input pattern in chronological order, and adjusts the connection relationship between the layers so that the output of the upper layer is maximized. It has the ability to dynamically switch using dynamic programming methods.

(Function) According to such a configuration, the lower layer can process the time-series pattern that is the input pattern, and dynamically determines the optimal connection relationship at the part that transmits the output of the lower layer to the upper layer. As a result, it is possible to remove the expansion/contraction distortion of the time axis that existed in the lower layer.

(Embodiment) FIG. 1 shows an example of the configuration of a speech recognition system based on the principle of the present invention.

Reference numerals 11 to 14 indicate input layers, which each consist of 16 units (4 units are omitted in the figure) and have a time series structure corresponding to time i. Figure 2 of the above document (2)
A time series of 16-dimensional spectrum vectors {x _in } (i in the time direction and m in the frequency direction) is input and sent to the intermediate layer (reference numbers 21 to 24). As shown in the figure, the input layer and the intermediate layer are connected to the i-th example unit of the intermediate layer to the i-1st, i-th, and i+1th units of the input layer. Each unit in column i of the intermediate layer is designated by a row number of k=1, 2, 3. Any combination of m and k is allowed between these two layers.

The coupling from the (i, m) unit in the input layer to the (i, k) unit in the hidden layer is loaded with the coefficient β _i,i,n,k, and the connection from the (i-1, m) unit in the input layer to the (i-1, m) unit in the The coupling to the (i,k) unit of the intermediate layer is loaded with the coefficients β _i-1,i,n,k . Similarly, from (i+1, m) unit in the input layer to (i, k) unit in the middle layer
The units are loaded with coefficients β _i+1,i,n,k . Therefore, the sum of input signals to the (i, k) unit in the middle layer is net(i, k) = 〓 ^m β _i-1,i,n,k・x _i-1,n + 〓 ^m β _{i,i ,n,k}・x _i,n + 〓 ^m β _i+1,i,n,k・x _i,n −θ (1) (θ is the bias factor generated inside the unit). In response to this, the (i,k) unit in the intermediate layer generates the output y _ik =1/1+e ^-net(i,k) (2). The signals in equation (1) are β _{i-1,i,n,k ,} β _i,i,n,k , β _i+1,i,n for _input x ik near time i
_,k is applied as a filtering process, and by appropriately learning the coefficient groups, it becomes a quantity that expresses the audio characteristics corresponding to the spectrum shape and change tendency. Equation (2) is obtained by converting it using the sigmoid function and normalizing it to the range of (0, 1). As a result, the output y _i = (y _i1 , y _i2 , y _i3 ) of units (i, 1) to (i, 3) at time i in the hidden layer is the detection of the audio features of the input in the vicinity of this time i. result.

Reference numbers 51-54 are output layers. A unit is provided corresponding to each word 1, 2, . . . n . . . N of the vocabulary to be recognized. (In a special case, the output layer may have only one unit. In that case, the entire neural network functions only as a detector for a specific word.) In the configuration according to the prior art,
Each unit n is connected to each unit (i, k) in the hidden layer, and the input sum net(n) = 〓 ⁱ 〓 ^k y _ik・α ⁿ _ik −θ (3) is weighted by a coefficient α ⁿ _ik . In response to this, an output O(n)=1/1+e ^-net(n) (4) is generated. When the unit n for which O(n) is maximum is n^, n^ is the recognition result.

A feature of the present invention is that there are switching layers 30 and 40 between the intermediate layer and the output layer. This switching layer has the function of making an optimal connection between time i in the intermediate layer and time j in the output layer.

As mentioned above in connection with Section 3.1 of Document (2), the voice pattern expands nonlinearly in the time direction. For this purpose, a mapping between time i of the input pattern, that is, time i of the intermediate layer, by a nonlinear function j=j(i) is provided between time axes i and j. To optimize this, it is sufficient to determine the mapping that maximizes the output of equation (4), but it is important to note that the sigmoid function in equation (4) is an increasing function and that θ on the right side of equation (3) is j( Since it is unrelated to i), it is sufficient to calculate the linear combination maximization problem f(n)=max _I 〓 ⁱ⁼¹ α ⁿ _jk ·y _ik (5).

Such problems can be calculated by dynamic programming, as is also done in literature (2). That is, under the initial condition g(1,1)=r(1,1) (6), g(i,j)=r(i,j)+maxg(i-
1, j) g (i-1, j-1) g (i-1, j-2) (7) By calculating each (i, j) grid point in positive order, )=g(I,J) (8) and maximization is achieved. Here, in equations (5) and (6), r(i, j)= 〓 ^k α ⁿ _jk ·y _ik (9).

The processing of equations (5) to (9) above is as shown in Figure 2 (i,
j) Clearly explained on a plane. The outputs y _ik from the intermediate layer are lined up on the horizontal axis (i). Load coefficients α ⁿ _jk are lined up on the vertical axis (j). At each (i, j) grid point, r(i, j) in equation (9) is calculated. Based on this, first, the initial condition setting of equation (6) is (1,
1) Made to a point. Thereafter, the recurrence formula of equation (7) is calculated while changing (i, j) in the direction of increasing (i, j), and the value of g(i, j) is filled in the grid points.
The process ends when g(i,j) is finally obtained.

The switching layer and output layer that operate in this manner are constructed by a microcomputer as shown in FIG. In the figure, a buffer 35 holds the output group {y _ik } from the intermediate layer. The load storage unit 45 stores the coefficient group {α ⁿ _jk }. The microprocessor unit 50 uses the table g (i, j) on the work memory 55 to calculate equations (6) to (9) for each output unit n, and calculates the output 0 for each output unit n. Calculate (n). Therefore, the processing program executed by the microprocessor section is as shown in FIG. During this process, when the coordinate of g on the right side in the calculation of equation (7) is negative or 0, it is assumed that g=-∞.

In addition, as a recognition system, 0(n) sequentially calculated by the process of FIG. 4 is the maximum, n=
The microprocessor section 50 also executes the process of determining n^ and producing a recognition result.

Equivalent processing functions in Figures 3 and 4 can be found, for example, in the Proceedings of the IEEE International Conference on Speech and Signal Processing.
Conference on Acoustics, Speech and Signal
"A High Speed Array Computer Dynamic Time Warping (A High Speed Array Computer Dynamic Time Warping)
Array Computer for Dynamic Time
This can also be done using parallel circuits, as shown in the paper entitled "Warping").

Although the principle and operation of the present invention have been explained above based on examples, these descriptions are not intended to limit the scope of the present invention. For example, in the above embodiment, the intermediate layer is one layer, but it may be two or three layers if necessary. A switching section is provided between either of these layers. Conversely, a configuration with no intermediate layer and an input layer and an output layer is also conceivable. Further, although the number of neuron units in the output layer is generally N, it may be N=1 in special cases. In this case, it operates not as a recognition system but as a pattern matching device or a pattern detection device.

(Effects) By configuring the lower layer in a time-series manner, it is possible to process time-series patterns such as audio, and by dynamically determining the optimal connection relationship in the part that transmits the output of the lower layer to the upper layer, Time axis distortion can be normalized. Furthermore, the optimization of the coupling relationship is carried out extremely efficiently by dynamic programming. Therefore, according to the dynamic network of the present invention, a time-series pattern recognition system (for speech, etc.) that operates with high accuracy and high speed can be realized at low cost using a small number of learning patterns.

[Brief explanation of drawings]

FIG. 1 is a diagram showing an embodiment of the present invention. FIG. 2 is an explanatory diagram of the operation. FIG. 3 shows a configuration example of the main part of the embodiment shown in FIG. 1, and FIG. 4 is an explanatory diagram of its operation. In the figure, 11, 12, 13, 14...input layer neural unit, 21, 22, 23, 24
...Middle layer neural unit, 30...Middle layer output, 40...Output layer input, 51, 52, 53,
54...Output layer neural unit, 31...Buffer, 45...Load storage section, 50...Microprocessor, 55...Work memory.

Claims (1)

[Claims] 1 In a neural network in which the layers are configured in a multilayered manner by a group of neural units that generate output according to the weighted sum of inputs, and the output is propagated from the lower layer of the adjacent layer to the upper layer, the lower layer is A dynamic neural network configured in a time-series manner and having a function of dynamically switching the coupling relationships between layers by dynamic programming means so that the output of the upper layer is maximized.