JPH0638200B2 - Dynamic neural network - Google Patents

Description

The present invention relates to an improvement of a neural network used in a pattern recognition system for speech or the like.

(Prior Art) A neural network, which is an information processing mechanism simulating a neural circuit of a living organism, is considered promising for solving various problems such as pattern recognition, and in recent years, research and development have been actively carried out all over the world.

Regarding the neural network, pp.115-12 of "Nikkei Electronics" magazine, No. 427 (issued on August 10, 1987).
Section 4 is a commentary entitled "Using neural networks for pattern recognition and knowledge processing."
(1) ”) is easy to understand. According to it, the basic component of the neural network is an element called a neuron unit, which calculates a weighted sum of a plurality of inputs and generates an output corresponding to the weighted sum. The neural network has a multi-layered structure and typically has three layers: an input layer, an intermediate layer, and an output layer.
It has a hierarchical structure. The output of each neural unit in the input layer is transmitted to each neural unit in the intermediate layer and becomes an input to them. The output of each neural unit in the middle layer is transmitted to each neural unit in the output layer and becomes an input to them.

Taking speech recognition as an example, in the input layer, "Measurement and Control" magazine,
Figure of the paper (hereinafter referred to as "Reference (2)") published in Volume 22, No. 1 (published in January 1983), pp.99-pp.105, entitled "Pattern matching method in speech recognition" A voice pattern as shown in 2 is received as an input signal. Here, each element of the matrix of FIG. 2 is made to correspond to each neural unit of the input layer. For each neural unit in the output layer, the word corresponding to each word in the recognition target vocabulary and corresponding to the neural unit that gives the maximum output is the recognition result.

As described above, in order for the recognition operation to function effectively, it is necessary to learn the coefficient of the load applied to the input of the neural unit in advance. For this, the back propagation method described in detail in page 118 of the above-mentioned document (1) can be used.

(Problems to be solved by the invention) The neural network having the above-mentioned configuration and functions can cope with the pattern recognition problem that cannot be theoretically formalized by demonstrating an advanced function of learning. It is expected to give a simple solution. However, as shown in FIG. 2 of document (1), the neural networks proposed to date have a very general form in that completely free coupling is performed between the neural units between layers. It is a thing. It may therefore be versatile, but inefficient when dealing with special problems. For example, the position of a print character, which is a relatively simple recognition target, varies depending on printing conditions. It is extremely difficult to give a learning pattern and learn the weighting factor by assuming all position variations. The same thing can be said for fluctuations in the enlargement / reduction of characters.

Consider another example of speech recognition. As described in Section 3.1 of Reference (2), there is expansion and contraction of the time axis in the voice pattern. This stretching has a complex non-linearity and results in a very large amount of strain patterns. It is impossible to perform learning for all of these in terms of calculation time.

It can be seen from the above two examples that a large amount of learning patterns are required for learning patterns. However, it is clear that the recognition rate decreases as the number of learning patterns decreases.

That is, the conventionally proposed neural network has a drawback that it is not suitable for processing expansion and contraction and position variation, although it is general. The present invention aims to realize a dynamic neural network configuration in order to improve this.

(Means for Solving the Problems) The dynamic neural network according to the present invention is characterized by having a switching layer that optimally switches the connection relationship of the inputs to any of the layers.

(Operation) At the part where the connection relation of the above input is switched, any connection relation is not given in advance. Moreover, it is assumed that each connection line at both ends of the switching unit has some correspondence with the position of the input pattern. Thus, when the input pattern is given, by changing the above connection so that the output of each neural unit in the output layer is maximized, the position, shape (size, distortion) of the input pattern in the lower layer changes. Can be normalized and transmitted to the upper layer, and a stable recognition operation can be achieved.

(Embodiment) FIG. 1 is a diagram showing a configuration of a voice recognition system based on the principle of the present invention. Reference numerals 11 to 14 are input layers, each consisting of 16 units (a part of which is omitted in the figure). The time series of the 16-dimensional spectrum vector as shown in FIG.
Input {x _im } (i in time direction, m in frequency direction) and send it to the middle layer (reference numerals 21 to 24). As shown in the drawing, the input layer and the intermediate layer are connected to the unit of the i-th column of the intermediate layer by the three units of the input layer i-1, i, and i + 1. Each unit in the i-th column of the intermediate layer is designated by a row number of k = 1, 2, 3. Any combination of m and k is allowed between these two layers.

Coupling from the (i, m) unit in the input layer to the (i, k) unit in the intermediate layer is weighted by a coefficient β _{i, i, m, k} ,
β from the (m) unit to the (i, k) unit in the middle layer
It is loaded with the coefficients _{i-1, i, m, k} . Similarly, (i + 1,
β _{i + 1, i, m, k} from the (m) unit to the (i, k) unit in the middle layer
Is loaded. Therefore, the sum of input signals to the (i, k) unit in the middle layer is (θ is the bias factor generated inside the unit). In response to this, the (i, k) unit in the middle layer Will produce output. The signal of equation (1) is β _{i-1, i, m, k} , β _{i, i, m, k} , β _{i + 1, i, m, k} for input x _{i, k} near time i.
The weighting is applied to the following coefficient group, and by appropriately learning the coefficient group, it becomes an amount that expresses a voice feature corresponding to the spectrum shape and the changing tendency. Equation (2) is converted by the sigmoid function and normalized to the range (0,1). As a result, the outputs y ₁ = (y _i1 , y _i2 , y _i3 ) of the units (i, 1) to (i, 3) at the time i in the hidden layer are detected as the input speech features in the vicinity of the time i. Will result.

Reference numerals 51 to 54 are output layers. Words in the recognition target vocabulary 1,
Units are provided corresponding to 2, 2, ..., N. (In a special case, the number of units in the output layer may be one. In that case, the entire neural network functions only as a detector for a specific word.) In the conventional configuration, each unit n corresponds to each unit in the intermediate layer ( i, k) and input weighted with a coefficient of ▲ α ⁿ _{i, k} ▼ Received and output accordingly Will produce output. The unit n that maximizes 0 (n) is When Is the recognition result.

A feature of the present invention is the switching layers 30, 40 between the intermediate layer and the output layer. This switching layer is the time i of the middle layer.
And the function of performing an optimal coupling between the output layer time j and the output layer time j.

As described above with reference to Section 3.1 of Reference (2), the speech pattern grows nonlinearly in the time direction. For this purpose, a mapping by a non-linear function j = j (i) between the time i of the input pattern, that is, the time i of the intermediate layer is provided between the time axis i and the time axis j. To optimize this, the mapping that maximizes the output of Eq. (4) may be defined, but the sigmoid function of Eq. (4) is an increasing function, and θ on the right side of Eq. (3) is j ( i) is irrelevant It is sufficient to calculate the maximization problem of

As such a problem is also done in the literature (2),
It can be calculated by dynamic programming. That is, under the initial condition g (1,1) ＝ r (1,1) (6) By maximizing the recurrence formula for each (i, j) lattice point in the positive order, the maximization of f (n) ＝ g (I, J) (8) is achieved. Where, in equations (5) and (6) Is.

The processing of the above equations (5) to (9) is clearly explained on the (i, j) plane as shown in FIG. The output y _ik from the intermediate layer is arranged on the _dummy axis (i). The load factor ▲ α ⁿ _jk ▼ is lined up on the vertical axis (j). At each (i, j) grid point, r (i, j) in equation (9) is calculated. Based on this, first, the initial condition setting of Eq. (6) is made to the (1,1) point. After that, while changing (i, j) in the direction of increasing (i, j), the recurrence formula of Eq. (7) is calculated and the values of g (i, j) are packed into the grid points. The process ends when g (i, j) is finally obtained.

The switching layer and the output layer which operate as described above are constituted by a microcomputer as shown in FIG. In the figure, the buffer 35 has an output group from the middle layer.
{y _ik } is retained. The load storage unit 45 stores the coefficient group {▲
α ⁿ _jk } is stored. The microprocessor unit 50 uses the table g (i, j) on the work memory 55 to perform calculations of equations (6) to (9) for each output unit n, and output 0 (n ) Is calculated. Therefore, the processing program executed by the microprocessor unit is as shown in FIG. When negative or 0 occurs as the coordinate of g on the right side in the calculation of the equation (7) during this processing, it is assumed that g = -∞.

In the recognition system, 0 (n), which is sequentially calculated by the process of FIG. 4, is the maximum. The microprocessor unit 50 also executes a process of determining and determining as a recognition result.

Equivalent to the processing functions shown in FIGS. 3 and 4, for example, is Proceedings of the EE International Conference on Acoustics Speech and Signal Processing.
EE International Conference on Acoustics, Speech an
d Signal Processing) pp.471-474 published in March 1981, "A High Speed Array Computer for Dynamic Time Warping (A High Speed Ar
rey Computer for Dynamic
This can be done using parallel circuits, as shown in the paper published under the title "Time Warping".

The principle operation of the present invention has been described above based on the embodiment,
These descriptions do not limit the scope of the invention.
In particular, although the present embodiment deals with an example of time distortion of a voice pattern, it is obvious that pattern variations such as position and size can also be dealt with. For example, when dealing with position fluctuations, the coupling of the switching layers in FIG. 1 can be simply shifted between i and j to maximize equation (5). Middle tier output 3
If 0 and the output layer input 40 are two-dimensionally configured to perform enlargement / reduction, parallel movement, etc., it is effective for recognizing characters and the like. Although the switching layer is provided between the intermediate layer and the output layer in the above-mentioned embodiment, it may be provided between the intermediate layer and the input layer or before the input layer. Further, although the intermediate layer is one layer in this embodiment, it is also possible to set a plurality of layers such as two layers and three layers as required. On the contrary, the principle of the present invention can be applied to a neural network including an input layer and an output layer without providing an intermediate layer.

The number of neuron units in the output layer is generally N
Although the number is N, it may be N = 1. In this case, it operates not as a recognition function but as an acceptor.

(Effect) According to the dynamic neuron network of the present invention having the function of optimally switching the input to any of the layers, a stable output can be output with respect to pattern variations such as position and expansion / contraction, so a small amount of learning is required. With the data, it is possible to realize a highly accurate recognition operation.

[Brief description of drawings]

FIG. 1 is a diagram showing an embodiment of the present invention. FIG. 2 is an explanatory diagram of the operation. FIG. 3 is a structural example of the main part of the embodiment shown in FIG. 1, and FIG. 4 is an operation explanatory diagram thereof. In the figure, 11, 12, 13, 14 ... Input layer neural unit, 21, 2
2,23,24 ... Middle layer neural unit, 30 ... Middle layer output, 40 ... Output layer input, 51,52,53,54 ... Output layer neural unit, 31 ... Buffer, 45 ... Weight memory Department,
50: Microprocessor, 55: Work memory.

Claims (1)

[Claims] 1. A neural network in which a layer is constituted by a group of neural units which generate an output in accordance with a weighted sum of inputs, and the output is propagated between the neural units of adjacent layers. A dynamic neural network having a switching layer that optimally switches the connection relations of each other.