JP2756170B2 - Neural network learning circuit - Google Patents

Description

The present invention relates to a neural network learning circuit.
In particular, the present invention relates to a neural network learning circuit that introduces a learning technique for adapting internal parameters according to input / output conditions.

Conventionally, it is composed of many processing elements that perform addition, subtraction, and nonlinear processing on input signals, and is used in applications similar to human intelligence activities, such as pattern recognition, inference, and optimization problems. In a network, a process of adapting internal parameters in accordance with input / output conditions is required to provide desired input / output characteristics. This is generally called neural network learning. However, the conventional neural network learning is suitable only for simulators and emulators using digital circuits and software, and a neural network learning circuit realized by an analog circuit is desired.

[Conventional technology]

In general, a neural network weights signals input from a plurality of network input ports 31, a plurality of or a single network output port 32, and a plurality of input ports as shown in FIG. It comprises a plurality of processing elements 33 for nonlinearly processing the sum signal and outputting the processed signals to an output port, and a plurality of connection elements 34 for connecting the processing elements to each other.

FIG. 4 shows a conceptual diagram of the processing element 33 described above. Processing element 33, as shown in this figure, the non-linear after performing weighting signal x ₁ ~x ₅ input from a plurality of input ports 330, an addition unit 332 performs addition processing, the sum signal Non-linear processing unit that processes and outputs to output port
332. In addition processing section 331, input port 330
Since the weighting amount is assigned for each, the degree of coupling of the input signal from the input port 330 changes according to the weighting amount.

The characteristics of the nonlinear processing of the nonlinear processing unit 332 are shown in FIG.
It is often represented by the sigmoid function shown by 33. This sigmoid function is expressed by the following equation.

Z = 1 / [1 + exp {k (by)}] where k is a constant representing a nonlinear characteristic, and b is a threshold value of the nonlinear characteristic.

As can be seen from FIG. 4, when the level of the input signal is extremely large, the output signal level becomes, for example, "1".
, And when the level of the input signal is extremely small, the output signal level is close to, for example, “0”.
Also, when the level of the input signal is halfway between them,
The output signal level is determined according to a characteristic curve that changes monotonically (when k: finite) or discontinuously (when k: infinity).

Since the neural network is a network having such a configuration, it has a function of nonlinearly mapping between input and output patterns, and can be applied to pattern recognition, inference optimization problems, and the like.

In order to give a desired input / output characteristic to a neural network having the above concept, a process of changing internal parameters is necessary. This process is called neural network learning, and comprises the following processes.

(1) A predetermined input pattern is input to the neural network.

(2) An error amount is obtained from a difference between the output pattern at that time and a desired output pattern known in advance.

(3) Then, the parameters of the neural network are sequentially changed so as to reduce the error amount.

Conventionally proposed learning methods are, for example, DERumelhar
t, JLMcClelland's `` Parallel Distributed Process
ing: Explorations in the Microstructures of Cogniti
on.Vol.1 ", MIT Press (1968) [hereinafter referred to as Reference 1]
It was expressed as an algorithm, as in the backpropagation method proposed in. That is, the procedure for calculating the amount of correction of each internal parameter from the desired output required of the neural network and the actual output of the neural network has been expressed in a complicated mathematical form.

(Problems to be Solved by the Invention) However, such a conventional learning method is suitable for a neural network realized by a digital circuit or a simulator or emulator using software, There is a problem that it cannot be directly applied to a neural network realized by a circuit. In order to apply a conventional learning method to a neural network realized by an analog circuit, an A / D and D / A converter were required.

In addition, when trying to use a conventional learning method for a neural network realized by an analog circuit, calculations have to be performed sequentially for each internal parameter, which requires an enormous amount of time for learning. On the other hand, if these learnings are to be performed in parallel, the amount of wires connecting the calculation processing unit for learning and the elements inside each of the neural networks becomes enormous, and it is extremely difficult to realize them with hardware. there were.

An object of the present invention is to provide a learning method suitable for a neural network realized by an analog circuit so that learning can be performed at high speed. The purpose of the present invention is to simplify hardware necessary for the above.

[Configuration of the invention]

(Means for Solving the Problems) FIG. 1 shows a principle configuration of a neural network learning circuit of the present invention which achieves the above object. As shown in FIG. 1, the neural network learning circuit of the present invention comprises:
A neural network 1 including a plurality of network input ports, a plurality or a single network output port, a plurality of processing elements for performing weighted addition and non-linear processing on an input signal, and a plurality of connection elements for connecting the processing elements to each other And an error signal generating means for generating an error signal by comparing the output signal of the neural network 1 with the teacher signal during learning of the neural network 1. Means 3 for synchronously detecting a predetermined frequency component from the error signal received from the error signal generating means 3 at the time of learning, sequentially accumulating the detected signal output over time, and obtaining the obtained accumulated signal and the predetermined signal. By mixing the frequency component with the neural network 1 and applying it to the neural network 1, the neural network Composed of parameter control means 4 for controlling the parameters of the network 1.

(Operation) According to the neural network learning circuit of the present invention,
At the time of learning of the neural network, the output signal of the neural network and the teacher signal are compared in the error detection unit, and an error signal is generated. A predetermined frequency component is synchronously detected from the error signal in the parameter control unit, and the output of the detected signal is sequentially accumulated to form an accumulated signal. The accumulated signal is mixed with the detected frequency component. And applied to the neural network. As a result, a neural network learning method based on hardware is provided, and further, by adopting a signal processing technique in the frequency domain, learning can be speeded up and hardware can be simplified.

(Examples) Hereinafter, the present invention will be described in detail by examples with reference to the drawings.

FIG. 2 shows the configuration of an embodiment of the present invention. Reference numeral 21 denotes a neural network having a plurality of network input ports 211, a plurality or a single network output port 212,
A plurality of processing elements 213 for adding weighted signals input from a plurality of input ports, adding the weighted signals, nonlinearly processing the sum signal, and outputting the resultant signal to an output port, and a plurality of interconnecting processing elements Are connected. Then, the neural network 21 processes the input pattern for learning according to the initially set internal parameters, and sends out an output pattern corresponding to the processing from the network output port 212.

An input pattern generator 25, an error detection unit 23, and a plurality of parameter control units 24 are connected to the neural network 21 having such a configuration. Furthermore, an error detection unit
The teacher pattern generator 22 that generates a desired output pattern corresponding to the input pattern is connected to the reference numeral 23.

The error detection unit 23 is an analog circuit to which an output pattern of the neural network 21 and a teacher pattern from the teacher pattern generator 22 are input from an input port 231. The error detection unit 23, and a neural network output pattern and the teacher pattern is the signal from the input port 231, to generate an error signal S _1. This error signal S ₁
Is a square sum error signal, for example, and FIG. 2 shows a case where a square sum error signal is generated. Therefore, the error detector 23 shown in FIG.
And the adding circuit 234 is provided, the differential amplifier outputs of the teacher pattern and the output pattern is squared by a square characteristic circuit 233, the sum is taken by the sum of squares error signals S ₁ produced by the addition circuit 234 . This error signal S ₁ is transmitted from the output port 235.

The parameter control unit 24 controls each internal parameter (weights W ₁ to
One is provided for each of W _n and threshold values θ ₁ to θ _m ). The parameter control unit 24 has two functions. One is to send a control signal for each internal parameter of the neural network 21, and the other function is
Is to perform synchronous detection on the error signal S ₁ generated by the error detecting section 23.

First, to explain the first function, in an actual neural network realized by an analog circuit,
Internal parameters (weights W _{1 to} W _n and thresholds θ ₁ to
Since the hardware corresponding to θ _m ) is realized by an analog multiplication circuit, the control of the parameter value is performed by an electric signal (in some cases, light is interposed). The control signal at this time is composed of a component that is constant in time and a component that changes minutely periodically. That is, if this is expressed electrically, the control signal is composed of a DC signal (direct current signal) S ₄ and a minute level RF signal (high frequency signal) S ₅ . These are the synthesis circuit 2
It is synthesized at 415 and applied to the neural network 21 via the output port 2417.

An RF signal oscillator 2416 is provided inside the parameter control unit 24 shown in FIG. Here, the number of frequencies (f _{1 to} f ₁₎ of the RF signal S ₅ emitted from the plurality of parameter control units 24
_{n + m} ) are set differently. here,
Each parameter of the neural network 21 is controlled by the RF signal S ₅ of different frequencies. These contributions are all derived from the neural network 21 and the error detector.
Synthesized by 23, will appear in the error signal S _1.

Will now be described synchronous detection on the error signals S ₁ generated by the error detecting unit 23 which is another function of the parameter control section 24. The input ports 2410 of the plurality of parameter controllers 24 are connected to the output port 235 of the error detector 23. The same frequency signal component and the RF signal S ₅ to the parameter control unit 241~24 _{n + m} is emitted is detected by synchronous detection.
Therefore, from the RF signal oscillator 2416, the RF signal S ₆ of another of the same frequency is taken out and used as the local oscillation signal of the synchronous detection. FIG. 2 shows a circuit including a multiplying circuit 2411 and a low-pass filter 2412 as an example of the synchronous detection circuit. Frequency separation is automatically performed by synchronous detection, and only frequency components unique to each parameter control unit are detected. During learning, but we must be sequentially modify the values of the internal parameters, the detection output S ₂ here corresponds to the correction signal for modifying the parameter values. The reason is described below.

The basic concept of learning a neural network is that, as described in, for example, the above-mentioned document 1, the amount of correction required for the internal parameter value of the neural network 21 is such that a small change in the parameter causes an error. may be set to be proportional to the degree of contribution to the signal S _1. This can be expressed as follows using mathematical expressions. That is, the correction amount c _i of the i-th internal parameter values, ∂w minimal change parameter _i, when the level of change of the error signal associated therewith and _{∂E i, c i = -∂E i} / ∂w i ...

Level of the RF signal S ₅ described above can be regarded as minimal change ∂w _i internal parameters. The synchronous level of the detection output S ₂ of the can be regarded as the change level ∂E _i of the error signal S _1. Therefore, if all the parameter control units 24
If allowed to match the level of the RF signal S ₅ emitted from (241~24 _{n + m),} the synchronous detection output S ₂ levels above can be considered as the correction amount c _i of each internal parameter values Becomes

Incidentally, the above-described modification of the internal parameters is repeated with the continuation of the learning. Therefore, the internal parameter value at a certain point in time is a temporal accumulation of the correction amount up to that point. That is, the parameter value W _i is represented by W _i = (c _i ) t ₁ + (c _i ) t ₂ +... + (C _i ) t _n . It can be expressed as. Here, (c _i ) t _j is the parameter value correction amount at the timing t = t _j at which learning is performed.
T is the length of time before learning is performed.

In the parameter control unit 24, an integrating circuit 2413 is used in order to temporally accumulate such a correction amount. Also,
The output S ₃ of the integration circuit 2413 corresponds to the aforementioned W, and is used as the DC signal component S ₄ of the control signal. The output S ₃ of the integration circuit 2413 is temporarily stored in the memory circuit 2414 as necessary, in some cases the output from the memory circuit 2414 is used as the DC signal component S _4.

As already mentioned in the parameter control section 24, by the synthesis circuit 2415 synthesizes the RF signal S ₅ and the DC signal S ₄ described above,
The signal is used for controlling each internal parameter of the neural network 21 via the output port 2417.

There are a plurality of parameter control units 24 corresponding to the number of internal parameters of the neural network 21. RF signal S in control signal
_Since the frequency of ₅ is set to be different for each parameter control unit 24, the control for these internal parameters is performed independently of each other.

The error detector 23 and the parameter controller 24 shown in FIG.
Can be realized by an analog circuit by using today's LSI technology. The circuit elements required at this time are a differential amplifier, a multiplication circuit, an addition circuit, a low-pass filter, an integration circuit, a memory circuit, and the like. As a matter of course, the error detection unit 23 and the parameter control unit 24 can also be realized by digital circuits. In particular, the memory circuit 2414 is suitable for being realized by a digital circuit.

As described above, the configuration shown in FIG. 2 realizes the basic concept when learning the neural network 21 by hardware. The key point in the configuration of FIG. 2, it is assumed that the frequency of the RF signal S ₅ in the control signals different from each other, using a signal processing technique in the frequency domain, by detecting a parameter correction signal for each frequency That is, each parameter is controlled independently. Thereby, the internal parameters of the neural network 21 can be controlled in parallel, so that the learning can be speeded up. Also,
In the parameter control unit 24, since it is not necessary to incorporate information on the internal parameters of the neural network 21 for each parameter, the connection for that becomes unnecessary,
The circuit configuration can be simplified. Further, in the configuration of FIG. 2, the correction of the internal parameters is determined not based on the individual parameter values but based only on the comprehensive output characteristics of the neural network 21.
The effect of errors generated during learning is reduced.

In the description of FIG. 2, the error detector 23 generates a square sum error signal for easy understanding.
Of course, another error signal (for example, an absolute value error signal) may be used. In addition, although the parameter control unit 24 is described to be provided with an RF signal oscillator individually, it is not always necessary, and the harmonic signal from one RF signal oscillator is divided for each frequency, and each of them is different. There is also a method of supplying to the parameter control unit.

Furthermore, the level of the RF signal S ₅ supplied to the neural network 21 from the parameter control unit 24, but need not temporally constant. For example, the initial learning to increase the level of the RF signal S _5, the progresses of learning continue to reduce its level, local minimum is that the minimum value of error well known in the learning of the neural network ) Local Minimum) can be avoided and learning speed can be increased. That is,
In learning, which can be said to be a process of searching for a condition in which the error is "minimum", it is possible to improve the learning speed by eliminating the problem that the learning sometimes stops locally at a local minimum value where the error is small.

In FIG. 2, the neural network is 1
Although the direction of signal flow has been described as an example, the application of the present invention is not limited to such a type of neural network.

〔The invention's effect〕

As described above, according to the neural network learning circuit of the present invention, the learning of the neural network realized by the analog circuit can be performed at high speed, and the learning circuit can be simplified and downsized. Can be done. This makes it easy to realize a neural network with a high-speed learning function, and has an effect of expanding the application field of the neural network.

[Brief description of the drawings]

FIG. 1 is a diagram showing the principle of a neural network learning circuit according to the present invention, FIG. 2 is a diagram showing the configuration of a neural network learning circuit according to an embodiment of the present invention, FIG. FIG. 4 is a conceptual diagram of a processing element of the neural network. 31 network input port, 32 network output port, 33 processing element, 331 addition processing section, 332 nonlinear processing section, 333 sigmoid function, 34
…… Connection elements, 21 …… Neural networks,
211: Network input port, 212: Network output port, 213: Processing element, 214: Connection element, 22: Teacher pattern generator, 23: Error detection unit, 231: Error detection unit input port 232 ... Differential amplifier, 233 ... Squaring circuit, 234 ... Addition circuit, 235 ... Error detection unit output port 24 ... Parameter control unit, 2410 ... Parameter control unit input port, 2411 ... Multiplication circuit, 2412 …… Low-pass filter, 2413 …… Integration circuit, 2414 …… Memory circuit, 2415…
... Synthesis circuit, 2416 ... RF signal oscillator, 2417 ... Parameter control unit output port, 25 ... Input pattern generator,

Claims (1)

(57) [Claims] A plurality of network input ports, a plurality of or a single network output port, a plurality of processing elements for performing weighted addition and non-linear processing on an input signal, and a plurality of connection elements for connecting the processing elements to each other. A learning circuit connected to a neural network (1) comprising: a teacher signal generating means (2) for generating a teacher signal; and an output signal of the neural network (1) and a teacher when learning the neural network (1). An error signal generating means (3) for comparing the signal with an error signal to generate an error signal; and at the time of learning, synchronously detecting a predetermined frequency component from the error signal received from the error signal generating means (3), and outputting the detected signal output. While accumulating sequentially in time, the obtained accumulated signal and the predetermined frequency component are mixed to form a neural network. By applying the chromatography click (1), a neural network learning circuit, characterized in that it comprises a parameter control means for controlling the parameters of the neural network (1) (4), a.