JPH0638196B2 - Dynamic neural network for time series patterns - Google Patents

Description

TECHNICAL FIELD The present invention relates to improvement of a neural network used in a time-series pattern recognition system for speech or the like.

(Prior Art) A neural network, which is an information processing mechanism simulating a neural circuit of a living organism, is considered to be promising for solving various problems such as pattern recognition and has been actively researched and developed all over the world in recent years.

Regarding neural networks, "Comment published in Nikkei Electronics magazine, no.427, pp.115-124, published on August 10, 1987, entitled" Using neural networks for pattern recognition and knowledge processing. " (Hereafter referred to as reference (1))
It is explained in plain text. According to it, the basic component of the neural network is an element called a neuron unit, which calculates a weighted sum of a plurality of inputs and generates an output corresponding to the weighted sum. The neural network has a multi-layer structure, and typically has a three-layer structure of an input layer, an intermediate layer, and an output layer.

The output of each neural unit in the input layer is transmitted to each neural unit in the intermediate layer and becomes an input to them. The output of each neural unit in the middle layer is transmitted to each neural unit in the output layer and becomes an input to them.

Taking voice recognition as an example, in the input layer, “Measurement and control magazine,
Figure of "A paper published as" Pattern matching method in speech recognition "in pp.99-pp.105 of Vol. A voice pattern as shown in 2 is received as an input signal. Here, each element of the matrix of FIG. 2 is assumed to correspond to each neural unit of the input layer. For each neural unit in the output layer, the word corresponding to the neural unit that gives the maximum output corresponding to each word in the recognition target vocabulary is the recognition result.

In order to effectively function such a recognition operation, it is necessary to previously learn the coefficient of the weight given to the input of the neural unit. For this, the back propagation method described in detail in page 118 of the above-mentioned document (1) can be used.

(Problems to be Solved by the Invention) The neural network having the above-described configuration and function can cope with a pattern recognition problem that cannot be theoretically formulated by exhibiting an advanced function of learning. It is expected to give a solution. However, Figure 2 of reference (1)
As shown in (1), the neural networks proposed to date are of a very general form in that completely free coupling is performed between the neural units between layers. It may therefore be versatile, but inefficient when dealing with special problems.

Consider an example of speech recognition as a typical example of handling time series patterns. As described in Section 3.1 of Reference (2), there is expansion and contraction of the time axis in the voice pattern. This stretching has a complex non-linearity and results in a very large amount of strain patterns. It is impossible to perform learning for all of these in terms of calculation time.

From the above example, it can be seen that a large number of patterns are necessary for pattern learning. However, it is clear that the recognition rate decreases as the learning pattern decreases.

That is, the conventionally proposed neural network has a drawback that it is not suitable for processing the fluctuation of expansion / contraction strain, although it is general. The present invention aims to realize a dynamic neural network configuration in order to improve this.

(Means for Solving Problems) A dynamic neural network according to the present invention configures an input layer and an intermediate layer in time series, and transmits an output of an input layer unit at a near time to an intermediate layer unit at each time. And a switching layer for dynamically switching the coupling between the output of the intermediate layer and the input of the output layer so that the output of each unit of the output layer is maximized.

(Operation) According to such a configuration, the input layer can receive the time-series pattern, and can output to the middle while keeping the time structure. In the middle layer, local features can be extracted as a time series from the input pattern near each time. By optimizing the coupling of the middle and output layers,
The time series distortion existing in the input time series pattern can be normalized.

(Embodiment) FIG. 1 is a structural sequence of a voice recognition system based on the principle of the present invention.

Reference numerals 11 to 14 are input layers, each of which is composed of 16 units (not shown in the figure, 4) and has a time-series configuration corresponding to time i. 16-dimensional spectrum vector time series {x _im } (i in the time direction, m in the frequency direction) as shown in FIG.
And send it to the middle tier (reference numerals 21-24). As shown in the drawing, the input layer and the intermediate layer are connected to the unit of the i-th example of the intermediate layer by the three units of the i-th, i-th, and i + 1-th units of the input layer. Each unit in the example i of the middle layer has k = 1,
Specified with a few line numbers. Any combination of m and k is allowed between these two layers.

Coupling from the (i, m) unit in the input layer to the (i, k) unit in the intermediate layer is weighted by a coefficient β _{i, i, m, k} ,
β from the (m) unit to the (i, k) unit in the middle layer
_The coefficients _{i-1, i, m, k} are weighted. Similarly, (i + 1,
β _{i + 1, i, m, k} from the (m) unit to the (i, k) unit in the middle layer
Is loaded. Therefore, the sum of input signals to the (i, k) unit in the middle layer is (θ is the bias factor generated inside the unit). In response to this, the (i, k) unit in the middle layer Will produce output. The signal in Eq. (1) is the input near time i.
x _ik is filtered by applying a coefficient group of β _{i-1, i, m, k} , β _{i, i, m, k} , β _{i + 1, i, m, k} . , By appropriately learning the coefficient group, it becomes the amount that expresses the voice feature corresponding to the spectrum shape and the change tendency. Equation (2) is converted by the sigmoid function and
It is normalized to the range of (0,1). As a result, the outputs y _i = of the units (i, 1) to (i, 3) at time i in the middle layer
(y _i1 y _i2 y _i3 ) is the detection result of the input speech feature in the vicinity of this time i.

Reference numerals 51 to 54 are output layers. Words in the recognition target vocabulary 1,
Units are provided corresponding to 2, 2, ..., N. (As a special case, the number of units in the output layer may be one. In that case, the entire neural network functions only as a detector for a specific word.) In the conventional configuration, each unit n corresponds to each unit in the intermediate layer ( i, k), and the input sum weighted by the coefficient α ▲ ⁿ _ik ▼ Received and output accordingly Will produce output. The unit n that maximizes 0 (n) is A feature of the present invention is the switching layers 30, 40 between the intermediate layer and the output layer. This switching layer has the function of providing an optimum coupling between time i in the intermediate layer and time j in the output layer.

As described above with reference to Section 3.1 of Reference (2), the speech pattern grows nonlinearly in the time direction. For this reason, a non-linear function between the time i of the input pattern, that is, the time i of the hidden layer
The mapping by j = j (i) is provided between the time axis i and the time axis j. To optimize this, the mapping that maximizes the output of Eq. (4) may be defined, but the sigmoid function of Eq. (4) is an increasing function, and θ on the right side of Eq. (3) is j ( i) is irrelevant It is sufficient to calculate the maximization problem of

As such a problem is also done in the literature (2),
It can be calculated by dynamic programming. That is, under the initial condition g (1,1) ＝ r (1,1) (6) By calculating the following recurrence formula at each (i, j) grid point in the positive order, f (n) ＝ g (I, J) (8) and maximization are achieved. Where, in equations (5) and (6) Is.

The processing of the above equations (5) to (9) is clearly explained on the (i, j) plane as shown in FIG. The output y _ik from the intermediate layer is lined up on the horizontal axis (i). The vertical axis (j) has the load factors α ▲ ⁿ _jk ▼ lined up.
At the (i, j) lattice point, r (i, j) in equation (9) is calculated. Based on this, first, the initial condition setting of Eq. (6) is made to the (1,1) point. After that, while changing (i, j) in the direction of (i, j) increasing
Calculate the recurrence formula of Eq. (7) and fill the grid points with the values of g (i, j). The process ends when g (i, j) is finally obtained.

The configuration of the switching layer and output layer that operate in this way is
It is composed of a microcomputer as shown in Fig. 3. In the figure, the buffer 35 has an output group from the middle layer.
{y _ik } is retained. The load storage unit 45 stores the coefficient group (α
▲ ⁿ _jk ▼} is stored. The microprocessor unit 50 uses the table g (i, j) on the work memory 55 to perform calculations of equations (6) to (9) for each output unit n, and output 0 (n ) Is calculated. Therefore, the processing program executed by the microprocessor unit is as shown in FIG. When negative or 0 occurs as the coordinate of g on the right side in the calculation of Eq. (7) during this process, g = −∞ is assumed.

As the recognition system, 0 (n), which is sequentially calculated by the process of FIG. 4, is the maximum. The microprocessor unit 50 also executes a process of determining and determining as a recognition result.

The equivalence of the processing functions of Figs. 3 and 4 is, for example, the Proceedings of IEE International Conference on Acoustics Speech and Signal Processing.
EEE International Conference on Acoustics, Speech a
nd Signal Processing) (pp.471-474 issued in March 1981, "A High Speed Array Computer for Dynamic Time Warping (A High Speed Array Comp
uter for Dynamic Time Warping) ”), as shown in a paper published in parallel.

The principle and operation of the present invention have been described above based on the embodiments, but these descriptions do not limit the scope of the present invention. For example, although the intermediate layer is one layer in the above embodiment, it may be two or three layers if necessary. The switching part is provided between the last of these layers and the output layer. Also, the number of neuron units in the output layer is generally N,
N = 1 may be a special case. In this case, it operates not as a recognition system but as a pattern matching device or a pattern detection device.

(Effect) By arranging the input layer and the intermediate layer in time series, it is possible to effectively process time-series patterns such as speech, and since each unit of the intermediate layer is connected to a unit near the input layer. It is possible to effectively extract local features (phonological properties) such as voice. Moreover, the time distortion can be efficiently normalized by optimizing the coupling between the hidden layer and the output layer by the dynamic programming method. Therefore, according to the dynamic neural network of the present invention, it is possible to inexpensively realize a time-series pattern recognition system that operates with high accuracy and high speed with a small number of learning patterns (such as voice).

[Brief description of drawings]

FIG. 1 is a diagram showing an embodiment of the present invention. FIG. 2 is an explanatory diagram of the operation. FIG. 3 is an example of the configuration of the main part of the embodiment shown in FIG. 1, and FIG. 4 is an explanatory diagram of its operation. In the figure, 11, 12, 13, 14 ... Input layer neural unit, 21, 22, 23,
24 ... Middle layer neural unit, 30 ... Middle layer output,
40 ... Output layer input, 51, 52, 53, 54 ... Output layer neural unit, 31 ... Buffer, 45 ... Load storage section, 50 ...
Microprocessor, 55 ... Work memory.

Claims (2)

[Claims] Claim: What is claimed is: 1. A neural network in which a layer is constructed in a multi-layered manner by a group of neural units which generate an output in accordance with a weighted sum of inputs, and the output is propagated from a lower layer of an adjacent layer to an upper layer of the input layer. And the middle layer are configured in time series, and the output of the input layer unit at the near time is transmitted to the middle layer unit at each time, and the middle layer output is set so that the output of each unit of the output layer is maximized. A dynamic neural network having a switching layer for dynamically switching a connection with an output layer input. 2. The dynamic neural network according to claim 1, wherein the output of the intermediate layer and the input of the output layer are connected by a dynamic programming method.