JP3126762B2 - neural network - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network, and more particularly, to a parallel distributed information processing apparatus (neurocomputer) for artificially realizing an information processing function of a neural network. It can be applied to image recognition, voice recognition, robot control, image processing, natural language processing, associative memory, optimization problem processing, and the like.

[0002]

2. Description of the Related Art A neural network is an information processing device designed to imitate a biological neural network. A neural network is a unit in which constituent units called neuron units are connected so that the neuron units can send and receive signals to and from each other by providing means for transmitting signals between the neuron units. A neuron unit corresponds to a nerve cell in an organism's neural network. Some neuron units in a neural network receive signals from outside the neural network. Some neuron units in the neural network output signals outside the neural network. A neural network is a parallel information processing apparatus that receives a signal from the outside, processes the signal simultaneously and in parallel by a number of neuron units inside the network, converts the signal, and outputs the converted signal to the outside. In some types of neural networks, each time a signal is input to the neural network and the signal is output from the neural network, the desired value of the output signal is given to the neural network, so that this operation is repeated several times. Then, the value of the output signal can be made closer to its desired value. This is called neural network learning. A signal that gives a desired value of the output signal to the neural network is called a teacher signal. Because of such a learning ability, a program necessary for an existing computer is not required, and the development cost of the program is not required. In addition, there is no need to adjust the program according to a use environment.

Each neuron unit in a neural network receives a signal from some other neuron unit in the neural network or from outside the neural network, and sends a signal to some other neuron unit in the neural network or outside the neural network. Output a signal. When a neuron unit receives a signal from a neuron unit connected to the neuron unit, the neuron unit receives the signal with a different receiving efficiency for each connection. For example, in a neuron unit that receives a signal from the neuron unit 1 through the connection 1 and receives a signal from the neuron unit 2 through the connection 2, the reception efficiency of the connection 1 is 0.8 and the connection 2
Is 0.5, the input signal value received by the neuron unit from the neuron unit 1 through the connection 1 is 0 when the input signals having the strength of 1.0 enter the connection 1 and the connection 2 respectively. .8, and the value of the input signal received from the neuron unit 2 through the connection 2 is 0.5. The neuron unit outputs a signal based on the input signal thus received. Hereinafter, the accepting efficiency of this coupling will be referred to as a coupling coefficient. There are various ways in which a neuron unit outputs a signal based on an input signal, and one of the methods widely used is described below.

FIG. 7 is a diagram for explaining the operation of a neuron unit. As shown in the figure, consider a neuron unit j that receives signals from k neuron units. The input signal coming from the i-th neuron unit of k is written as y _i . The coupling coefficient between the i-th neuron unit and neuron unit j is written as _Tji . First, the input signal y _i (i = 1, 1) which is input to the neuron unit j weighted by the coupling coefficient T _ji
2,..., K). Expressing the sum with the symbol net _j ,

[0005]

(Equation 1)

[0006] Next, this net _j is converted by a monotonically increasing saturation function (written as f) to convert the neuron unit j
Output y _j

[0007]

(Equation 2)

[0008] The following function called a logistic function or a sigmoid function is often used as the function f.

[0009]

(Equation 3)

FIG. 8 shows this function.

There are various methods for creating a neural network. At its simplest, it can be created as software that runs on a regular computer. In this case, it is relatively easy to operate a neural network having various configurations under various conditions. However, there is a drawback that the operation speed is slow and it is difficult to incorporate it into another device because it cannot be made small.
Therefore, it is suitable for use in research on neural networks, but is often not suitable for practical use.
As a method for overcoming the drawbacks of the neural network as software on a computer, there have been proposed a number of methods for creating a neural network using hardware.

Among them, a neural network "pulse density logical operation type neural network" made by using a digital circuit by the present applicant (Japanese Patent Application No. 1-1).
79629) will be described below. However, only the part related to the present invention, that is, the input / output processing function of the neuron unit will be described. FIG. 9 shows a method of processing an input signal in each neuron unit in the pulse density logical operation type neural network.
In the pulse density logical operation type neural network, an input signal (21 in FIG. 9) and an output signal (2 in FIG. 9) are used.
2) consists of synchronized bit strings. Each bit in the bit string takes a value of either 0 or 1, and the value of the signal is represented by the pulse density of the bit string, for example, by the number of 1s within a certain fixed interval. If the i-th input signal is y _i ,

[0013]

(Equation 4)

## EQU1 ## This indicates a signal having a value of 4/6, and among the six synchronization pulses, the input signal y _i has four 1s and zeros.
Are two. At this time, the arrangement of 1 and 0 is desirably random. On the other hand, a coupling coefficient T _ji indicating the degree of coupling between neuron units.
(23 in FIG. 9) is also prepared in advance as a bit string of 0 and 1 on the memory. Also, the value of the coupling coefficient is expressed by the pulse density in the same manner as the value of the input / output signal as follows.

[0015]

(Equation 5)

This represents "101010" = 3/6, and it is desirable that the arrangement of 1s and 0s at this time is also random. First, each neuron unit sequentially reads out the bit string of the coupling coefficient from the memory according to the synchronization clock, and ANDs the bit string of the input signal with the AND circuit (24 in FIG. 9). To explain using the examples so far, when the input signal is input as “101101”, a bit string is called from the memory in synchronization with the input signal,
By taking the AND sequentially,

[0017]

(Equation 6)

"101000" as shown in the following is obtained.
This indicates that the input y _i was weighted by T _ji and the pulse density became 2/6. The pulse density of the output of the AND circuit 24 is approximately the product of the pulse density of the input signal 21 and the pulse density of the coupling coefficient 23, and has the same function as the analog coupling coefficient. This is a function closer to a product as the signal sequence is longer and the arrangement of 1s and 0s is more random. Note that, when the bit string of the coupling coefficient is shorter than the input bit string and there is no more data to be read, return to the beginning of the data again,
Reading may be repeated. Since one neuron unit has multiple inputs, the input signal and the coupling coefficient AN
There are many results of taking D. Next, the OR of these multiple results is taken (25 in FIG. 9). Since the inputs are synchronized, for example, the first data is “101000”, 2
When the third data is “010000”, the OR of both is “111000”. When this operation is performed for L inputs,

[0019]

(Equation 7)

## EQU1 ## When the pulse density is low, this OR
, The pulse density approximately matches the sum of the respective pulse densities. In addition, the pulse density obtained by taking the OR is a monotonically increasing function of the sum of the pulse densities. Since the pulse density is never greater than 1, the output of the OR does not match the sum of the pulse densities and becomes increasingly saturated as the pulse density increases. Therefore, the operation of ORing corresponds to the above-described operation of converting the sum of the input signals by the monotone increasing saturation function f.

By the way, the case where the coupling coefficient is positive has been described above. However, in order to make the function of the neural network practical, it is necessary that the coupling coefficient is not only a positive value but also a negative value. Therefore, it is desirable that the output can be suppressed by the negative input. A connection having a positive coupling coefficient is called an excitatory connection, and a connection having a negative coupling coefficient is called an inhibitory connection. In an analog circuit, in the case of suppressive coupling, an output is inverted using an amplifier and coupled to another neuron unit with a resistance value corresponding to a coupling coefficient. On the other hand, the pulse density is always positive, but if the output signal of the neuron unit is generated by any one of the following three methods (1) to (3), the value of the coupling coefficient can be expressed by the pulse density. Even if it represents, it is possible to cope with the excitability and the inhibitory property of the connection.

(1): See FIG. 10 For each connection, whether it is excitatory or inhibitory is set in advance, and a group of excitatory connections (30 in FIG. 10) and a group of inhibitory connections (FIG. 10) 10 and 31), the OR of a pulse train obtained by ANDing the input signal and the coupling coefficient is separately obtained (this OR is 25 in the OR of the group 30 in FIG. 10 and 25 in the OR of the group 31 in FIG. 10). To 26
). Alternatively, for each input, whether it is excitatory or inhibitory is set in advance, and the above-mentioned OR is separately performed for the group 30 of excitatory inputs and the group 31 of suppressive inputs. Next, OR2 of the inhibitory group 31
6 is NOT (negated) by the NOT circuit 27, and the result is ANDed with the result of the OR25 of the excitatory group 30.
D is taken by an AND circuit 28, and the result is used as an output signal 22 from the neuron.

(2): See FIG. 11 This example has a memory (32 in FIG. 11) for each connection indicating whether the connection is excitatory or inhibitory. , The suppression property can be set arbitrarily (33 in FIG. 11). The above-mentioned OR is separately performed for the group of the excitatory connection and the group of the inhibitory connection determined by the contents of the memory (25 and 26 in FIG. 11).
Next, NOT (negation) 27 of the result of OR26 of the inhibitory group is taken (27 in FIG. 11), and the result is ORed with the result of OR25 of the excitatory group (2 in FIG. 11).
8), and let the result be the output signal from the neuron (22 in FIG. 11).

(3): See FIG. 12 For each connection, both an excitatory coupling coefficient and an inhibitory coupling coefficient are provided, and both are stored in the memory (FIG. 12, 23a,
23b). This is equivalent to expressing the coupling coefficient in the form of a sum of a positive amount and a negative amount. Then, the OR 25 of the results obtained by taking the AND 24a of the input signal 21 and the excitatory coupling coefficient 23a is taken. Also, the OR 26 of the results obtained by taking the AND 24b of the input signal 21 and the suppressing coupling coefficient 23b is taken. Next, OR26 of the inhibitory group
The NOT (negation) 27 of the result is taken, the AND 28 of the result and the result of the OR 25 of the excitatory group is taken, and the result is set as the output signal 22 from the neuron.

In these methods, the output signal value becomes 0 when the input signal value is 0, and the input signal from the inhibitory group is more output than the input signal from the excitatory group. It has characteristics such as having a strong influence. For these schemes, the applicant has filed an application to increase the capacity of the neuron unit by allowing the output level to be adjusted. In that method, the above associations correspond to excitability and inhibition (1), (2),
(3) becomes the following (4), (5), and (6), respectively.

(4): See FIG. 13 For each connection, whether it is excitatory or inhibitory is set in advance, and the above-mentioned OR is separately set for the excitatory connection group and the inhibitory connection group. Take (25 and 2 in FIG. 13)
6). Alternatively, for each input, whether it is excitatory or inhibitory is set in advance, and an excitatory input group (FIG. 1)
3-30) and the group of suppressive inputs (31 in FIG. 13)
And the above-mentioned OR is separately performed. Next, the result of OR25 of the excitatory group and the result of OR26 of the inhibitory group obtained in this way are combined as follows (3 in FIG. 13).
4) The output value 22 from the neuron unit is
Is calculated. First, if the result of OR of the two does not match, the result of OR25 of the excitatory group is set as the value of the output bit from the neuron (22 in FIG. 13). That is,
If the result of OR25 in the excitatory group is 0 and the result of OR26 in the suppressive group is 1, 0 is output 22 and the result of OR25 in the excitatory group is 1 and the result of OR26 in the suppressive group is 0. If so, 1 is output 22. When the result of the OR25 of the excitatory group matches the result of the OR26 of the suppressive group, an input signal prepared separately (hereinafter, this signal is referred to as an output shift input signal).
(35 in FIG. 13) is output as it is.

(5): See FIG. 14 Each connection has a memory (32 in FIG. 14) that indicates whether the connection is excitatory or inhibitory.
The excitability and the suppressiveness of the connection can be arbitrarily set (33 in FIG. 14). The above OR is separately performed for the group of excitatory connections and the group of inhibitory connections determined by the contents of the memory (25 and 26 in FIG. 14). next,
By combining the result of OR25 of the excitatory group and the result of OR26 of the inhibitory group obtained in this manner in the same manner as in the above (4) (34 in FIG. 14), the output value 22 from the neuron unit is obtained. calculate.

(6): See FIG. 15 An excitatory coupling coefficient and an inhibitory coupling coefficient are provided for each coupling, and both are stored in the memory (23a and 23 in FIG. 15).
b). This is equivalent to expressing the coupling coefficient in the form of a sum of a positive amount, a negative amount, and a sum. Then, the OR 25 of the results obtained by taking the AND 24a of the input signal 21 and the excitatory coupling coefficient 23a is taken. Also, the result of AND 24b of the input signal 21 and the suppressing coupling coefficient 23b is an O
Take R26. Next, the result of OR25 of the excitatory group and the result of OR26 of the inhibitory group obtained in this way are combined in the same manner as in the above (4) (34 in FIG. 15).
Thus, the output value 22 from the neuron unit is calculated.

In the above (4), (5) and (6), the result of OR25 of the excitatory group and the result of OR2 of the inhibitory group
When the value of the output shift input signal (35 in FIGS. 13, 14, and 15) output when the result of No. 6 matches is always 0, that is, when the pulse density of the output shift input signal is 0
, The output in (4) becomes the output in (1), the output in (5) becomes the output in (2),
The output at (6) is equal to the output at (3). In the above (4), (5), and (6), when the probability that the output shift input signal 35 takes the value 1 is 0.5 (therefore, the probability that it takes 0) is 0.5, that is, the output shift When the pulse density of the input signal 35 is 0.5, the input signal from the excitatory group and the input signal from the inhibitory group have the same influence on the output signal, and the output function of the neuron unit is excited. And the suppression function are balanced, and the form of the output function is close to the form of the sigmoid function (Equation 3), which is the output function widely used in ordinary hierarchical neural networks. Therefore, it is preferable.

As described above, the output signal from the neuron unit is sent to another neuron unit, where it is weighted by the coupling coefficient and taken in by the neuron unit to determine the signal output amount from the neuron unit. used. Thus, a neuron unit can influence the signal output from another neuron unit by sending a signal to the other neuron unit. However, the neuron unit
By directly controlling the transmission efficiency of signals transmitted between other neuron units, it is not possible to influence the signal transmission between other neuron units. The prior art did not have such a function. Therefore, if a signal transmission controller having the function of controlling the signal transmission efficiency is interposed on the signal transmission path between neuron units, and the signal transmission controller operates in conjunction with the state of another neuron unit, In addition, a neural unit can directly control signal transmission between other neuron units, and a neural network can have a more complicated signal propagation pattern than a case without such a mechanism. Can be applied to a wider range of applications.

[0031]

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and in particular, in order to realize a more functional neural network, a mechanism by which a neuron unit can directly control signal transmission between other neuron units. It was made for the purpose of making.

[0032]

To achieve the above object, the present invention provides a first neuron unit, a second neuron unit,
Is this is found first neuron unit and disposed on the signal transmission path between the second neuron unit, the second news on the signal transduction <br/> path from said first neuron unit
A signal transmission control means to control the amount of signal transmitted to over Ron unit, third New controlling the signal transmission control unit
It consists Ron unit, the signal transmission control means the third
Is controlled in conjunction with the degree of excitement of the neuron unit. Hereinafter, a description will be given based on an example of the present invention.

In the following, when the present invention is applied to a neural network composed of a sequence of bits in which signals between neuron units take a value of either 0 or 1, as in the above-described pulse density logical operation type neural network. An example will be described. A bit having a value of 1 is regarded as a pulse, and the signal strength is represented by a pulse density, that is, the number of bits having a value of 1 within a predetermined time.

FIG. 1 shows two neuron units 1 and 2 transmitting a signal, signal lines 4 and 5 which are routes of signals between the neuron units, and a signal transmission control interposed on the signal line. FIG. 1 shows a means 7, a neuron unit 3 for controlling the signal transmission control means, and a signal line 6 for sending a signal from the neuron unit 3 to the signal transmission control means 7. Signal transmission control means 7
Is composed of a NOT circuit 9 and an AND circuit 8. The result of performing a logical NOT operation on the bit input from the signal line 6 and the signal line 4
And outputs the signal on the signal line 5 by taking the logical product with the bit coming in. If the neuron unit 3 is in the rest state, that is, if only the bit of the value 0 is output from the neuron unit 3, the output bit of the NOT circuit 9 is always the value 1, so The signal that has entered the signal transmission control means 7 passes through the signal transmission control means 7 as it is and enters the signal line 5. On the other hand, when the neuron unit 3 is outputting an output signal, as shown below, the pulse input from the signal line 4 to the signal transmission control means 7 is thinned out to the signal line 5 according to the strength of the signal. Sent.
Now, let us look at six clocks of the bit string sent on the signal line 4. Assume that the bit string sent from the neuron unit 1 to the signal line 4 has a pulse density of 4/6 as follows.

[0035]

(Equation 8)

The signal line 6 from the neuron unit 3
Is transmitted at a pulse density of 3/6 as follows.

[0037]

(Equation 9)

When the logical NOT of this signal is taken and the logical product of the signal and the signal on the signal line 4 is taken, as is easily understood,
The result is the following bit train with a pulse density of 2/6.

[0039]

[Equation 10]

If the pulse density of the signal output from the neuron unit 1 is a and the pulse density of the signal output from the neuron unit 3 is b, if the pulse positions of both signals are statistically independent, signal transmission The pulse density of the signal output from the control means 7 is a × (1-
b). That is, the intensity of the signal transmitted to the neuron unit 2 is multiplied by (1-b) by the signal transmission control means. Therefore, as the signal output from the neuron unit 3 becomes stronger, the transmission of the signal transmitted from the neuron unit 1 to the neuron unit 2 is suppressed by the signal transmission control means 7, and as shown in FIG. (Indicates the strength of the signal transmitted from the unit 1 to the neuron unit 2), and the original signal strength a at b = 0 changes to 0 at b = 1.

FIG. 2 shows another example of the signal transmission control means 7. In this example, unlike the signal transmission control means 7 shown in FIG. 1, only one OR circuit 10 is provided. In this case, the signal transmission control means 7
A bit string that has entered the signal transmission control means 7 through
The logical sum of the bit string input to the signal transmission control means 7 through the signal line 6 is calculated, and the result is sent to the signal line 5. In this case, assuming that the pulse density of the signal output from the neuron unit 1 is a and the pulse density of the signal output from the neuron unit 3 is b, if the pulse positions of both signals are statistically independent, signal transmission is performed. The pulse density of the signal output from the control means 7 is a + b ×
(1-a) is obtained. Therefore, neuron unit 3
The stronger the signal output from is, the greater the intensity of the signal transmitted from the neuron unit 1 to the neuron unit 2 by the signal transmission control means, and as shown in FIG. 5, the original signal at b = 0 Strength a, b = 1
In the case of, the intensity changes to 1.

FIG. 3 shows still another example of the signal transmission control means 7. In this case, the signal transmission control means 7 includes a shift register 11, an AND circuit 12, and an OR circuit 13. The shift register 11 stores the bit input from the signal line 4 and stores the bit A after one clock.
Send it to the ND circuit 12. The AND circuit 12 calculates the logical product of the bit taken out from the shift register 11 and the bit sent out by the neuron unit 3 to the signal line 6, and
The result is sent to the OR circuit 13. Therefore, both the bit output by neuron unit 3 and the bit output by neuron unit 1 one clock before it are the value 1
, The AND circuit 12 outputs a bit having a value of 1. The OR circuit 13 includes the neuron unit 1
Is ORed with the bit output by the AND circuit 12 and sent to the signal line 5. In this case, assuming that the pulse density of the signal output from the neuron unit 1 is a and the pulse density of the signal output from the neuron unit 3 is b, if the pulse positions of both signals are statistically independent, an AND circuit The pulse density of the signal output from 12 is a × b on average. Therefore, O
The pulse density of the signal output from the R circuit 13 is 1- (1-a) × (1-a × b) = a × ｛1+ (1
−a) × b}. Therefore, the stronger the signal output from the neuron unit 3, the stronger the signal transmitted from the neuron unit 1 to the neuron unit 2 by the signal transmission control means, as shown in FIG.
The original strength a at the time of b = 0 changes to the strength a × (2-a) at the time of b = 1. In this case, unlike the case of the circuit of FIG. 2, the amount of increase of the signal is obtained by further multiplying the amount of increase (1-a) × b of the signal of the circuit of FIG. 2 by a, that is, a × ( 1-a) × b. That is, in this case, the signal is increased so that the amount of increase of the signal in the case of the circuit in FIG. 2 further increases in proportion to the original signal strength a.

As described above, on the signal transmission path from the first neuron unit to the second neuron unit,
A single, that is, an example in which a signal transmission control unit that operates in conjunction with the state of the third neuron unit is described, but the present invention is not limited to the above-described embodiment. Further, the second signal transmission control means operating in conjunction with the state of the fourth neuron unit may be interposed in series or in parallel with the signal transmission path, for example, in conjunction with the state of each neuron unit as necessary. It is easily considered that a signal transmitted from the first neuron unit to the second neuron unit may be converted by a desired function and transmitted by interposing a plurality of operating signal transmission control means. Let's do it.

[0044]

As is apparent from the above description, according to the present invention, in a neural network, a neuron unit can directly control signal transmission on a signal transmission path between other neuron units. In particular, it is possible to individually control signal transmission paths. (In the prior art, by controlling the output of the neuron unit, all the signal transmission paths from one neuron unit could be controlled together.)
Therefore, a more complicated signal propagation pattern can be provided to the neural network as compared with a case without such a mechanism, and the application range of the neural network can be expanded.

[Brief description of the drawings]

FIG. 1 is a configuration diagram for explaining an embodiment of the present invention.

FIG. 2 is a configuration diagram for explaining another embodiment of the present invention.

FIG. 3 is a configuration diagram for explaining another embodiment of the present invention.

FIG. 4 is a diagram illustrating transfer characteristics of the circuit of FIG. 1;

FIG. 5 is a diagram illustrating transfer characteristics of the circuit of FIG. 2;

FIG. 6 is a diagram illustrating a transfer characteristic of the circuit of FIG. 3;

FIG. 7 is a diagram for explaining an outline of a neuron unit.

FIG. 8 is a diagram showing a sigmoid function.

FIG. 9 is a diagram illustrating an example of a pulse density logical operation type neural network.

FIG. 10 is a diagram for explaining an example of a conventional neural network.

FIG. 11 is a diagram for explaining another example of a conventional neural network.

FIG. 12 is a diagram for explaining another example of a conventional neural network.

FIG. 13 is a diagram for explaining another example of a conventional neural network.

FIG. 14 is a diagram for explaining another example of a conventional neural network.

FIG. 15 is a diagram for explaining another example of a conventional neural network.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... 1st neuron unit, 2 ... 2nd neuron unit, 3 ... 3rd neuron unit, 4, 5 ... Signal transmission path, 6 ... Signal line, 7 ... Signal transmission control means.

────────────────────────────────────────────────── ─── Continuation of front page (56) References JP-A-4-549 (JP, A) JP-A-1-193,982 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06G 7/60 G06F 15/18

Claims (1)

(57) [Claims] A first neuron unit and a second neuron unit;
And-menu Ron unit, the signal transmission path between the Collector et first neuron unit and the second neuron unit
And the first neuron unit is disposed on the signal transmission path.
A signal transmission control means to control the amount of signal transmitted to the second neuron unit from the knitting, the signal transmission control unit
Consists third neuron unit for controlling said neural network signaling control means being controlled in conjunction with the excitement level of the third neuron units.