JPH07101415B2 - Information processing apparatus and learning method thereof - Google Patents

Description

TECHNICAL FIELD The present invention relates to an information processing apparatus capable of learning a neural network used for voice recognition, image processing, control, etc. at high speed. In particular, wafer scale integrated circuits (WSI)
The present invention relates to an information processing apparatus which is convenient to be configured above and which provides a large-scale neural network with a high-speed learning function in a compact size and at a low price, and a learning method thereof.

[Prior Art] Neural networks are applied to information processing such as recognition and knowledge processing. Neural networks can self-organize by learning if they are given inputs and expected output values. Therefore, the program is not required and its application is expected.

One of the learning algorithms is Rumelhar
t) et al. Parallel distributed processing (Chapter 8 3)
Pages 18-362 and Nikkei Electronics 1987.8.10 (No.42)
7) Introduced on pages 115-121. "Backpropagation is known. This backpropagation will be described below.

FIG. 2 shows a model of a neuron that constitutes a neural network. A neuron inputs input x _i from another neuron. Then, each input x _i is weighted with the weight value w _ji , and the total internal energy (inner product) u _j is calculated. This value is converted by, for example, a sigmoid function f, and the output x _j is output. Where the weight w
_ji is a weight value for the input neuron i in the neuron j.

w _j = f (u _j ) (1) u _j = Σ _i w _ji · x _i (2) A neural network can be constructed by connecting a plurality of neurons. Then, various information processing can be performed by changing the weight value w _ji .

FIG. 3 shows a hierarchical neural network. Although this figure is composed of three layers of an input layer, an intermediate layer, and an output layer, it may be configured to have a plurality of intermediate layers. For the input, the mode that operates in the forward direction to perform information processing and output and the expected output value for the given input are given from the outside, and the weight value w _ji is set from the output layer side to the input layer side. It has a mode that works in the opposite direction to correct. Weight value w at that time
a correction value [Delta] w _ji of _ji and the neuron ^{_{j Δw ji n + 1 = ηδ}} j x i + αΔw ji n (3). Here, n is the number of times of learning, and α and η are constants. Further, the function δ is determined as follows.

In the neuron k in the output layer, if the expected output value is t _k , then δ _k = (t _k −x _k ) f ′ (u _k ) (4), and in the neuron j in the intermediate layer, δ _j = f ′ (U _j ) Σ _k δ _k w _ji (5). Here, the function f'is a derivative of the function f.

The correction value Δw is determined as described above, and the weight value w is corrected. Then, the above learning is repeated until the output of the output layer is as expected.

In general, research on the elucidation of a neural network and its application is carried out by software simulation on a sequential computer. However, in order to learn a large-scale neural network composed of a large number of neurons, for example, thousands or tens of thousands of neurons, enormous calculation time is required. Also, in general, learning often requires a large number of iterations. Therefore, at present, the capabilities of large-scale neural networks are not well understood.

On the other hand, there is an attempt to calculate a neural network at high speed with dedicated hardware. One of them, “Basic Design of Completely Digital Neuro-WSI”, Proceedings of the 1989 IEICE Spring National Convention, 7 volumes SD-1-5,29
Figure 4 shows the method announced on pages 1-292. here
WSI stands for Wafer Scale Integration (Wafer
Scale Integration).

In FIG. 4, 201 is a neuron calculation unit and 210 is a bus. Each neuron calculation unit 201 is connected by a bus. One of the neuron calculation units 201 is selected, and its output value is sequentially output to the bus 210. Each neuron calculation unit 201 holds the weight value for the output neuron in the memory,
The value successively input from the bus 210 is weighted by reading the weight value corresponding to the value from the memory, and the product is cumulatively added. When outputting to the bus 210, the value is converted by the sigmoid function of Expression (1) and output. The weighting circuit operates in a time division manner. Each neuron calculation unit 201
Can be output to the bus 210, all can perform the calculation of the equation (2). Hereinafter, this is referred to as a time division bus coupling method.

By performing the above operation, the neuron model of equations (1) and (2) can be calculated at high speed. Another feature is to realize Fig. 4 on WSI. According to the time division bus coupling method, the weighting circuit can be used for time division, and the area of one neuron calculation unit 201 can be significantly reduced. In this way, the WSI has been successfully achieved by reducing the trouble of one calculation unit as much as possible, improving the yield per one calculation unit, and reducing the ratio of the neuron calculation unit 201 which becomes inoperable due to a defect. By adopting WSI, it becomes possible to realize a large-scale neural network faster and more compactly than before.

In a neural network in which neurons are completely connected, assuming that the number of neurons is N, one equation (1),
The number of times of weighting required to calculate equation (2) is about N ²
It becomes an individual. In the time division bus combination method, N weighting circuits operate simultaneously, so the calculation time becomes O (N),
A large-scale neural network can be calculated at high speed.

[Problems to be Solved by the Invention] In the time-division bus coupling method described in the related art, when the above-mentioned backpropagation is attempted, in the calculation of the backward expressions (3) to (5), the function To calculate δ, the weight value at the output destination of the neuron is necessary.
For example, to calculate δ _j of neuron j in the hidden layer,
The weight value w _kj for the neuron j in the neuron k in the output layer is required. However, as described above, the calculation unit 201 that calculates the neuron j holds only the weight value w _ji for the neuron i in the input layer. Therefore, it will be calculated (5) can not be the computing in parallel formula, sequential and [delta] _k w _kj one by one. Therefore, although the time required to perform the forward direction is O (N), the time required to calculate the backward direction is O (N ² ), and there is a problem that learning cannot be performed at high speed.

An object of the present invention is to realize an information processing apparatus configured by a neural network capable of high-speed learning, and particularly to realize an information processing apparatus and a learning method thereof that combine a time division bus coupling method and back propagation. It is in.

Other objects of the invention will be apparent from the detailed description below.

[Means for Solving the Problems] The present invention is provided with a first network that performs forward calculation and a second network that calculates δ in the reverse direction, and means for mutual communication.

Further, each network holds a weight value so that it is convenient to perform each calculation in parallel. for that reason,
The same weight value is held twice, but the correction is performed in each network, and the corrected result has the same value.

[Operation] The present invention is suitable for a dedicated computer such as a back-propagation learning algorithm that performs learning of a neural network that performs forward and backward calculations at high speed. In the present invention, the calculation of equations (1) and (2) is performed in the forward network,
The calculation of equations (4) and (5) is performed in the reverse network.
The weight value between the neurons is held at the same value in the two networks. Further, both the values calculated in each network are necessary as the values necessary for calculating the corrected value of the equation (3). In the present invention, by providing the communication means between the two networks, it is possible to obtain the value necessary for the unit constituting each respective network to correct the weight value, and each of the weight values has the same value. It is possible to modify. In the forward network, the equations (1) and (2) are conveniently configured to be calculated in parallel, and in the backward network, (4) and (5).
It is conveniently constructed for computing expressions in parallel. Also,
Equation (3) is also calculated in parallel in each unit of each network. Therefore, according to the present invention, the learning described above can be executed in O (N) calculation time.

In addition, since the forward and backward calculations are performed in different networks, each computing unit can be implemented in a small area. Therefore, in the case of WSI, the yield per calculation unit can be increased.

Further, the calculation unit that performs the forward calculation and the calculation unit that performs the backward calculation can have the same circuit configuration, and the number of design steps can be reduced. In that case, which of the calculation units operates will be determined by the control signal. Therefore, even if WSI is used,
It is only necessary to design and manufacture one type of WSI, so the cost can be reduced. If multiple WSIs are facilitated and they are connected, a larger-scale neural network can be easily constructed.

[Examples] Hereinafter, the details of the present invention will be described using examples.

FIG. 1 is a diagram showing an embodiment of the present invention. In the figure, 120F, 121F, 120B and 121B are calculation units, and 123 and 124 are control circuits. Each calculation unit is connected to the control circuit 123 or the control circuit 124. A control signal generation circuit 130 outputs control signals to the control signals 142 to 145. Each of the control circuits 123 and 124 inputs a control signal from the control signal line 142.
140 is an output bus. Reference numeral 128 denotes a buffer that outputs the output value of each calculation unit to the output bus 140, separates it from the output bus 140, or controls it by the control circuits 123 and 124. 131
Is a function conversion circuit, which converts the value input from the output bus 140 by the sigmoid function f of the equation (1) and outputs it.
Reference numeral 132 is a function conversion circuit, which converts the value input from the output bus 140 according to the derivative f ′ of the sigmoid function f,
Output. Reference numeral 125 denotes a selector which outputs one of the output of the output bus 140 or the function conversion circuit 131 or the output of the function conversion circuit 132 according to the control signal of the control signal line 145. 160
Is a multiplier, which multiplies the output of the selector 125 by a constant η and outputs the result. A selector 127 outputs the output of the selector 125 or the output of the multiplier 160 according to the control signal of the control signal line 144. Reference numeral 150 is an external input terminal, and 141 is an input bus. Reference numeral 126 denotes a selector which outputs the output of the selector 127 or a signal input from the external input terminal 150 to the input bus 141 according to the control signal of the control signal line 143. Each calculation unit is connected to the input bus 141 and can input a signal output to the input bus 141. 151 is an external output terminal,
The value output to the input bus 141 goes out.

FIG. 1 shows the configuration of the present invention for performing the calculations of equations (1) to (5) at high speed. Details of each calculation unit will be described later with reference to FIGS. 7, 8, and 9. FIG. 5 shows the operation of the configuration of FIG. 1 in detail. First,
The operation of the present invention will be described below with reference to FIG. 5, focusing on the operation of the neuron j. Note that the input layer may simply be configured as a layer for temporarily storing input data,
Further, as shown in FIG. 5, it may be configured as a neuromodel calculation unit. Each calculation unit in FIG. 5 corresponds to each calculation unit in FIG. 1, and the calculation units 120F, 121F, 120B, 121B in FIG. 1 correspond to each in FIG. The calculation units 120F and 121F form a forward network that performs a forward operation, and the calculation units 120B and 121B form a reverse network that performs a backward operation. Here, the forward network is (1)
The network for calculating the equation (2) is a network for performing the calculation for the equations (4) and (5). The equation (3) is calculated in each network.

The calculation unit in the middle layer of the forward network holds the weight value w _ji for the neuron i in the input layer, and the calculation unit in the output layer holds the weight value w _kj for the neuron j in the middle layer. Calculation unit 120 for the middle tier
F holds the weight value w _ji . Further, in the calculation unit of the middle layer of the backward network, the calculation unit of the output layer of the forward network is the neuron j of the middle layer.
Holds the weight value w _kj have contrast, in the calculation unit of the input layer of the backward network, a weight value w _ji the calculation unit of the intermediate layer in the forward network has on neurons of the input layer keeping. The calculation unit 120B in the middle layer holds the weight value w _kj .
Weight value w _ji forward network held, and w _kj, weights w _ji of backward network holds, w _kj corresponds, the same value.

In the forward network, the input layer to the intermediate layer, the output layer, and the outputs x _i , x _j , and x _{k of} each calculation unit are determined. The calculation unit 120F inputs the output x _{i of the} input layer, weights it with the weight value wx _ji , calculates the internal energy u _j , further performs nonlinear conversion, and outputs the output value x _j . Next, the calculation unit of the output layer of the forward network inputs the output x _j of the _hidden layer, weights it with the weight value w _{k j} , calculates the internal energy u _k , further performs the nonlinear transformation, and outputs the output value. Output x _k . Each computing unit of the forward network sends a respective output value x _i , x _j , x _k to the corresponding computing unit of the reverse network. The calculation unit 120F sends its output value x _j to the calculation unit 120B. Then, each calculation unit of the middle layer and the output layer of the forward network outputs its output value x _j , x _k to the corresponding calculation unit of the backward network.
The partial differential x _j ′, x _k ′ of is transmitted. The calculation unit 120F calculates the derivative x _j ′ (= f ′ (u _j )) of the output value x _j.
Output to 0B.

Then, if the output value x _k of the output layer of the forward network is different from the expected value t _k , the weight value needs to be modified. The operation will be described below.

In the output layer of the backward network, δ _k is calculated according to the equation (4), and is output to each calculation unit in the intermediate layer of the backward network. Required to calculate equation (4)
You have already entered x _k ′. When the calculation unit in the middle layer of the backward network inputs δ _k , the correction value Δw _kj is calculated according to the equation (3) using that value, and the weight value w _kj is changed. In addition, each calculation unit of the output layer of the backward network sends the respective output value δ _k to the corresponding calculation unit of the output layer of the forward network. As a result, the calculation unit in the output layer of the forward network can change the weight value w _kj by calculating the correction value Δw _kj by the equation (3).

Next, the output δ _k of each calculation unit in the output layer of the reverse direction network is output, and the calculation unit in the middle layer of the reverse direction network which inputs it outputs δ _k and the weight value w _kj, and (5 ), Δ _j is determined and output.
X _j ′ required to calculate equation (5) is the calculation unit 120F
Already entered. In the backward network, when each calculation unit in the input layer inputs δ _j , the value is used to calculate a modified value Δw _kj of and the weight value w _kj
To change. Also, each computation unit in the middle layer of the reverse network sends a respective δ _j to the corresponding computation unit in the middle layer of the forward network. The calculation unit in the middle layer of the forward network can change the weight value w _kj by calculating the correction value Δw _{kj according} to the equation (3).

As described above, the present invention provides (1) (2) (4)
Compute x, x '(= f' (u)) in equation (5) in a forward network convenient for calculating it at high speed, and compute δ in equation (4) (5) at high speed. Compute in a reverse network that is convenient to do.
Then, Δw in the equation (3) is the weight value w held by each calculation unit of each network.
Calculate against. In the calculation unit of the forward network, δ is needed to calculate the correction value Δw, and in the calculation unit of the reverse network ε and Δ
We need x, x 'to compute W. for that reason,
Data communication is required between the calculation units supported by each network, but the amount of data to be communicated corresponds to the number of neurons, and the transfer time required for this is O (N), which is a short time. Can be done at.

Although the intermediate layer has been described as one layer in this figure, learning can be similarly performed for a neural network including a plurality of intermediate layers.

Next, a method of realizing the above operation on hardware will be described with reference to FIG. FIG. 6 is a calculation flow chart showing the flow of the above operation, and shows the calculation procedure of the calculation units of the forward network and the backward network. The calculation unit j of the forward network is the calculation unit 120F of FIGS. 1 and 5, and the calculation unit j of the reverse network is the calculation unit 12 of FIGS. 1 and 5.
It is 0B. Hereinafter, FIG. 1 will be described with reference to FIG.

Each calculation unit holds the weight value for the input neuron in the memory. First, the initial value of the weight value w is written in each calculation unit. The weight value w _ji for the neuron i in the input layer is written in the calculation unit 120F. A weight value w _ji that is the same as the weight value for the neuron j in the intermediate layer held by the calculation unit k in the output layer of the seasonal network is written in the calculation unit 120B. -Operation 1 Next, the selector 126 is connected to the external input terminal 150 side by the input bus 141.
Output from the external input terminal 150 to write the input value to each calculation unit i in the input layer of the forward network. -Operation 2 The control signal generation circuit 130 sets the selector 125 to output the output bus 140, the selector 127 to output the output of the selector 125, and the selector 16 to the selector 127.
The control signal is output to the control signal lines 143 to 145 so as to output the output. Next, the control signal generation circuit 130 outputs the address of the calculation unit i in the input layer of the forward network to the control signal line 142. The control circuit 123 decodes the address, and if the address matches the address of the calculation unit, it enables the buffer 128 and outputs the output value x _i of the calculation unit i of the input layer of the forward network.
Is output to the output bus 140. The calculation unit i in the input layer of the forward network only outputs the input value. The computation unit j in the middle layer of the forward network is the input bus
The input x _i of the input layer of the forward network is sequentially input from 141, weighted by the weight value w _ji , and the products are cumulatively added. When the control signal generation circuit 130 specifies all addresses of the calculation unit i of the input layer of the forward network, each calculation unit of the intermediate layer can perform the calculation of the equation (2), and the output value u _j thereof is determined.

Next, the control signal generation circuit 130 sets the selector 125 to output the output of the function conversion circuit 131, and then outputs the address of the calculation unit in the middle layer of the forward network to the control signal line 142. The addressed computing unit outputs its output value u _j to the output bus 140. The non-linear conversion circuit 131 performs the conversion of the equation (1) and outputs x _j (= f (u _j )). x _j is broadcast by the input bus 141 to the computing unit k in the output layer of the forward network. The calculation unit k of the output layer successively inputs the output x _{j of the} intermediate layer from the input bus 141, weights it with the weight value x _kj , and cumulatively adds the weighted values. When the control signal generation circuit 130 specifies all the addresses of the calculation unit j in the middle layer of the forward network, each calculation unit k in the output layer can perform the calculation of the equation (2) and the output value thereof.
u _k is determined. -Operation 3 Next, the control signal generation circuit 130 outputs the address of the calculation unit k in the output layer of the forward network to the control signal line 142. The addressed computing unit outputs its output value u _k to the output bus 140. After that, it is converted into x _k (= f (u _k )) by the non-linear conversion circuit 131, and the selector 1
25, selector 127, selector 126, input bus 141
Is output to. The output result can be read from the external output terminal 151. Then, if the output value x _k of the output layer matches the expected output value t _k , the learning ends. If they do not match, the following operation is performed. -Operation 4 The control signal generation circuit 130 outputs the output of the selector 125 to the output bus.
It is set to the 140 side and the output of the selector 127 is set to the multiplier 160 side. The selector 127 is set on the multiplier 160 side, and the selector 126 outputs the output of the selector 127. Next, the control signal generation circuit 130 outputs the address of the calculation unit i in the input layer of the forward network to the control signal line 142. The output x _i of the address-specified calculation unit i passes through the selector 125, is output to the multiplier 160, is multiplied by the constant η, and is converted into η · x _i . Then, it outputs η · x _i to the input bus 141 through the selector 127 and the selector 126. The calculation unit i of the input layer of the backward network takes in the value of the corresponding η · x _i when it is output to the input bus 141. Next, the control signal generation circuit 130 operates the selector 125.
Is set to the function conversion circuit 131 side, and the output of the selector 127 is set to the multiplier 160 side. The control signal generation circuit 130 outputs the address of the calculation unit j in the middle layer of the forward network as the control signal 142. The output u _j of the address-specified calculation unit _j is the function conversion circuit 131.
_Is converted to x _j by and passes through selector 125 and multiplier 160
Is output to and is multiplied by a constant η and converted into η · x _j . Then, it outputs η · x _j to the input bus 141 through the selector 127 and the selector 126. The calculation unit j in the middle layer of the backward network inputs the corresponding η · x _j . For example, the output value u _j of the calculation unit 120F is the function conversion circuit 131
Is converted to η · x _j by the
Is output to. At that time, the calculation unit 120B is
Take in x _j . -Operation 5 Next, the control signal generation circuit 130 sets the output of the selector 125 to the function conversion circuit 132 side and the output of the selector 127 to the selector 125 side. Control signal generation circuit 130
Outputs the address of the calculation unit j of the middle layer and the calculation unit k of the output layer of the forward network to the control signal 142. The output of the addressed calculation unit u _j , u _k
Is converted into u _j ′, u _k ′ by the function conversion circuit 132, passes through the selector 125, the selector 127, and the selector 126, and the input bus 141
Output to. The calculation units in the reverse network middle layer and the output layer can input the corresponding x _j ′, x _k ′. For example, the output value u _j of the calculation unit 120F is converted into the function conversion circuit 132x _j ′ and output to the input bus 141. At that time, the calculation unit 120B takes in x _j ′. -Operation 6 In the next operation, the difference (t _k −) between the output value t _k expected from the calculation unit in the output layer of the reverse direction network and the output value x _{k obtained} earlier
x _k ) from outside the network. It may be performed by the host computer, or a circuit for calculating (t _k −x _k ) may be provided. The calculation unit k in the output layer of the backward network calculates δ _{k in} the equation (4).
Operation 7 Next, the control signal generation circuit 130 sets the selectors 125 to 127 so that the output of the output bus 140 is multiplied by η by the multiplier 160 and output to the input bus 141. Then, the address of the calculation unit k in the output layer of the reverse network is output to the control signal 142. The output δ _k of each addressed calculation unit is multiplied by η by a multiplier 160 to obtain the input bus 1
Η · δ _k is output to 41. At this time, the output network calculation unit k of the forward network outputs the output η of the reverse network calculation unit corresponding to itself to the input bus 141.
・ If δ _k is output, take it in. -Operation 8 Next, the control signal generation circuit 130 causes the value of the output bus 140 to be output to the input bus 141 through the function conversion circuit 131.
After setting the selectors 125 to 127, the address of the calculation unit j in the middle layer of the forward network is output to the control signal 142. The addressed calculation unit j outputs the already calculated internal energy value u _j . As a result, the function-converted value x _j is output to the input bus 141. In each calculation unit of the output layer of the forward network, the correction value Δw _kj of the weight value w _kj for the input value x _{j is} calculated according to the equation (3), and the weight value w _kj is corrected. -Operation 9 Next, the control signal generation circuit 130 causes the selectors 125 to 127 to output the value of the output bus 140 to the input bus 141 as it is.
, And outputs the address of the calculation unit k of the output layer of the reverse direction network to the control signal 142. The addressed calculation unit outputs the already calculated δ _k . As a result, δ _k passes through the output bus 140, the selector 125, the selector 127, and the selector 126 and is output to the input bus 141. In each calculation unit of the intermediate layer of the backward network, the correction value Δw _kj of the weight value w _kj for the input value δ _{k is} calculated according to the equation (3), and the weight value w _kj is corrected. −
Operation 10 Next, the control signal generation circuit 130 specifies the address of the calculation unit k in the output layer of the reverse network. The addressed computing unit outputs the already computed δ _k . As a result, δ _k passes through the output bus 140, the selector 125, the selector 127, and the selector 126 and is output to the input bus 141. In each computation unit of the middle layer of the reverse network, in operation 10, the modified weight value w _kj is used,
Δ _j is calculated according to the equation (5). -Operation 11 Next, the control signal generation circuit 130 sets the selectors 125 to 127 so that the output of the output bus 140 is multiplied by η by the multiplier 160 and output to the input bus 141. And the control signal to the address of the calculation unit in the middle layer of the reverse network.
Output to 142. The output δ _j of each addressed calculation unit is multiplied by η by the multiplier 160 and the selector 12
7, η · δ _j is output to the input bus 141 through the selector 126. At this time, the calculation unit j of the middle layer of the forward network takes in the output η · δ _j of the calculation unit j of the backward network corresponding to itself, if it is output to the input bus 141. -Operation 12 Next, the control signal generating circuit 130 sets the selectors 125 to 127 so that the value of the output bus 140 is output to the input bus 141, and then the calculation unit i of the input layer of the forward network.
The address of is output to the control signal 142. The addressed computing unit outputs x _i . As a result, x _i is output to the input bus 141. In each calculation unit in the middle layer of the forward network, the correction value Δw _ki of the weight value w _ji for the input value x _{i is} calculated according to the equation (3), and the weight value w _kj is corrected. -Operation 13 Next, the control signal generation circuit 130 sets the selectors 125 to 127 so that the value of the output bus 140 is output to the input bus 14 as it is, and then sets the address of the calculation j of the intermediate layer of the backward network. Output to the control signal 142. The addressed computing unit outputs the already computed δ _j . As a result, δ _j is output bus 140, selector 125, selector 12
7, output to the input bus 141 through the selector 126. In each calculation unit of the input layer i of the backward network, the correction value Δw _ji of the weight value w _ji for the input value δ _j is calculated by (3)
Calculate and correct the weight value w _kj according to the formula. -Operation 14 Then, for the input, the value t _k that the output value x _k of the output layer expects
The above learning is repeated until.

According to the present invention, forward and backward calculations of x and δ operate in parallel in each layer, and the amount of data communication required therefor is proportional to N. It is possible to learn.

Next, the details of the calculation units constituting the forward network and the backward network will be described with reference to FIGS. 7 and 8.

FIG. 7 is a diagram showing a calculation unit forming a forward network. In FIG. 7, 501 is a memory for storing the weight value w _ji and αΔw _ji ⁿ in the equation (3). Reference numerals 503 and 504 are adders, and 502 is a multiplier. 511 and 512 are latches. 520 is the previous correction value Δw in equation (3)
the _ji ⁿ is multiplied by the multiplier α. 531 to 534 are selectors. Reference numeral 540 is a connection terminal to the input bus 141. Reference numerals 550 to 557 are control signal lines for control signals output from the control circuit 123. Each of them is a selector 534, a memory 501, a latch 511, and a selector.
531, selector 533, selector 532, latch 512, buffer
Control 128. 541 is a connection terminal to the output bus 140. The connection relationship will be described below.

The latch 511 holds a signal input from the input bus connection terminal 540. The selector 531 outputs the weight value w output from the memory 501.
The output of _ji and the latch 511 is input. The multiplier 502 inputs the output of the selector 531 and the input bus connection terminal 540.
The adder 503 inputs the αΔw _ji ⁿ which is the output of the output memory 501 of the multiplier 502. The selector 532 inputs the output of the multiplier 502 and the output of the adder 503. Adder 504 is selector 532
Input the output of and the output of the selector 533. The latch 512 holds the output of the adder 504. The buffer 128 inputs the output of the latch 512 and outputs it to the output bus connection terminal 541. The selector 533 inputs the output of the latch 512 and the weight value w _ji that is the output of the memory 501. The multiplier 520 inputs the output of the adder 503. The selector 534 inputs the signal input from the input bus connection terminal 540 and the output of the adder 504. The memory 501 outputs the output of the selector 534 to the weight value w _ji side and the output of the multiplier 520 to αΔw
Input as write data on _ji ⁿ side. Also, the memory 501
Inputs the address at the time of access from the control circuit 123.

When the initial value of the weight value w _ji of operation 1 is written, the initial value is given to the input bus connection terminal 540, and the output of the selector 534 is on the input bus connection terminal 540 side, which can be written in the memory 501.

The calculation of equation (2) is performed as follows.
The output of the selector 531 is the weight value w _ji output side of the memory 501. Input the input value x _i input to the input bus connection terminal 540. The product w _ji x _i is performed by the multiplier 502. Selector 53
2 outputs on the multiplier 502 side. The selector 533 outputs the latch 512 side. The adder 504 adds the output of the latch 512, which has been the cumulative addition result, and the product w _ji x _i , and writes the result in the latch 512. If the above operation is performed one after another by changing the weight value w _ji for the input value x _i input to the input bus connection terminal 540,
Equation (2) can be calculated.

The ηδ _j input from the reverse network is input from the input bus connection terminal 540 and written in the latch 511.

The calculation of formula (3) is performed as follows.
The selector 531 outputs the latch 511 side. Input the input value x _i input to the input bus connection terminal 540. The multiplier 502 calculates ηδ _j x _i . The memory 50 by the adder 503
The output αΔw _ji ^{n of} 1 and the output ηδ _j x _i of the multiplier 502 are added to calculate the correction value Δw _ji ⁿ ₎ ¹ . The selector 532 outputs the adder 503 side. The selector 533 is the weight value w of the memory 501.
_ji Output side is output. The adder 504 adds the previous weight w _ji ⁿ of the output of the selector 533 and the output Δw _ji ⁿ ₎ ¹ of the selector 532 and outputs a corrected weight value w _ji ⁿ ₎ ¹ . The selector 534 outputs the adder 504 side. Also, the output of the adder 503 Δw _ji ⁿ
₎ ¹ is multiplied by α by the multiplier 520. Then, the corrected weight value w _ji ⁿ ₎ ¹ of the output of the selector 534 and the multiplier 520 are stored in the memory 501.
Write the output of α Δw _ji ⁿ ₎ ¹ . All the weight values w _ji ⁿ ₎ ¹ can be corrected by successively performing the above operation for the input value x _i input to the input bus connection terminal 540.

FIG. 8 is a diagram showing a calculation unit forming a reverse network. In FIG. 8, 601 is a memory for storing αΔw _kj ⁿ in the weight w _kj (3). 60
Reference numerals 3 and 604 are adders, and 602 is a multiplier. 611 to 615 are latches. 620 is the previous correction value Δw _kj ^{n in} equation (3)
Is a multiplier for multiplying by. 631 to 634,636 are selectors. 640 is an input bus connection terminal. 650 ~ 659, 661, 66
Reference numeral 2 is a control signal line for a control signal output from the control circuit 124.
632, selector 634, selector 633, latch 614, latch 61
5, control the buffer 128. 641 is an output bus connection terminal. The connection relationship will be described below. The latches 611 to 613 input the signal input from the input bus connection terminal 640. The selector 631 inputs the weight value w _kj output of the memory 601, the output of the latch 611, and the output of the latch 612. The selector 632 inputs the signal input from the input bus connection terminal 640, the output of the latch 613, and the output of the latch 614. Multiplier 602 is selector 631
Input and the output of selector 632 are input. The adder 603 inputs the αΔw _kj ⁿ output of the memory 601 and the output of the multiplier 602.
The selector 633 inputs the output of the adder 603 and the output of the multiplier 602. The adder 604 is the output of the selector 634 and the selector 633.
Input the output of. The latch 614 inputs the output of the adder 604. The selector 634 inputs the output of the latch 614 and the weight value w _kj output of the memory 601. The multiplier 620 is input to the adder 603. The selector 636 inputs the signal input from the input bus connection terminal 640 and the output of the adder 604. The memory 601 outputs the output of the selector 636 to the weight value w _kj side and outputs the output of the multiplier 620 to α
Input as write data on the Δw _kj ⁿ side. Also memory
601 inputs the address at the time of access from the control circuit 124. The latch 615 inputs the output of the multiplier 602. buffer
128 inputs the output of latch 615, and output bus connection terminal 641
Output to.

When writing the initial value of the weight value w _kj , the initial value can be given to the input bus connection terminal 640 and the output of the selector 636 can be written on the input bus connection terminal 640 side.

Η x _j input from the reverse direction network is input from the input bus connection terminal 640 and wound into the latch 611. x _j ′ (= f ′
(U _j )) is also input from the input bus connection terminal 640 and latched 612
Write in.

The difference (t _k −x _k ) between the output value required when operating as the calculation unit of the output layer and the expected output value is input from the input bus connection terminal 640 and written in the latch 613. The output layer calculation unit performs the following operations to calculate δ _k .

The selector 631 outputs the latch 612 side. The selector 632 outputs the latch 613 side. In the multiplier 602, the output value x _k ′ (= f ′ (u _k )) of the latch 612 and the output value (t _k −x) of the latch 613
_k ) is multiplied and δ _k in equation (4) is output. The latch 615 captures the output δ _k of the multiplier 602 and outputs it to the buffer 128.
Output to. Δ _k is output to the output bus connection terminal 641.

The calculation unit of the output layer performs the following operation to calculate δ _k in the equation (5).

The selector 631 outputs the weight value w _kj side of the memory 601. The selector 632 outputs the input bus connection terminal 640 side. The output δ _{k of the} output layer is input to the input bus connection terminal 640. Multiplier 6
The product δ _k w _kj is output by 02. Selector 633 is multiplier 6
The 02 side is output. The selector 634 outputs the latch 614 side. The adder 604 is the latch 6 which is the cumulative addition effect so far.
The output of 14 and the product δ _k w _kj are added and written to the latch 614.
One after another above operation, if performed for a value [delta] _k applied to the input bus connection terminal 640 can calculate the Sigma] [Delta] _k w _kj (2) below. Next, the selector 631 outputs the latch 612 side. The selector 632 outputs the latch 614 side. Multiplier 60
2 is the output x _j ′ (= f ′ (u _j )) of the latch 612 and the latch 614.
The output Σδ _k w _kj is multiplied, and δ _j is output. The latch 615 captures the output δ _j of the multiplier 602 and outputs it to the buffer 128.
Output to. Δ _j is output to the output bus connection terminal 641.

When the equation (3) is calculated, it is performed as follows in the middle layer of the backward network. Selector 631 has latch 6
Output 11 side. The input value δ _k input to the input bus connection terminal 640 is input. The multiplier 602 calculates η x _j δ _k . The output of the memory 601 α Δw _kj ^{n is} calculated by the adder 603.
And the output η x _j δ _k of the multiplier 602 are added, and the corrected value Δw
_ji ⁿ ₎ ¹ is calculated. The selector 633 outputs the adder 603 side. The selector 634 outputs the weight value w _kj output side of the memory 601. The adder 604 uses the previous weight value w of the output of the selector 634.
_kj ⁿ and the output w _ji ⁿ ₎ ¹ of the selector 633 are added and the corrected weight value w _ji ⁿ ₎ ¹ is output. The selector 636 outputs the adder 604 side. The output Δw _ji ⁿ ₎ ¹ of the adder 603 is multiplied by α by the multiplier 620. Then, the corrected weight value w _kj ⁿ ₎ ¹ of the output of the selector 626 and the output αΔw _kj ⁿ ₎ ^{1 of the} multiplier 620 are stored in the memory 601.
Write. The above operation is repeated one after another with the input bus connection terminal 64
All weight values w _kj ⁿ ₎ ¹ can be modified if the input value δ _k is input to 0. The weight value w _ji can be similarly corrected in the input layer of the backward network.

As described above, according to the present invention, the calculation unit of the output layer of the forward network and the calculation unit of the middle layer of the reverse network have the same weight value w _kj ⁿ , but the modification is performed by the calculation unit having the weight value. Do each one. The modified weight value w _kj ⁿ ₎ ¹ becomes the same value. Similarly, the calculation unit in the middle layer of the forward network and the calculation unit in the input layer of the reverse network have the same weight value w _ji ⁿ , but the correction is performed by each calculation unit having that weight. The modified weight value w _ji ⁿ ₎ ¹ becomes the same value.

Next, FIG. 9 will be described. FIG. 9 shows the calculation unit of FIG. 8 with a selector 635 and a control line 660 for controlling it. 122 is a control circuit. Selector 635 has latch 61
Input the output of 4 and the output of the multiplier 602. The latch 615 inputs the output of the selector 635. Others are the same as in FIG. If the output of the selector 635 is on the multiplier 602 side,
The same operation as in the figure is possible and it can be a computing unit of the reverse network. Further, the output of the selector 635 is set to the latch 614 side, and the latch 5 shown in FIG.
11, selector 531, multiplier 502, adder 503, selector 53
2, adder 504, latch 512, selector 533, multiplier 520,
The selector 534, the memory 501, the latch 611, the selector 631, the multiplier 602, the adder 603, the selector 633, and the adder 60 shown in FIG.
4, latch 614, selector 634, multiplier 620, selector 63
6. If it corresponds to the memory 601, it can be used as a forward network computing unit. That is, in order to implement the present invention, it is not necessary to design the two calculation units shown in FIGS. 7 and 8, but the calculation unit shown in FIG. 9 may be designed. In such a case, the number of design units can be significantly reduced.

As described above, according to one embodiment of the present invention, since it can be realized on an integrated circuit, it is possible to realize a neural network that learns at high speed.

Next, FIG. 10 will be described. In FIG. 10, 1 and 2 are wafers and 700 is a functional block. 710 to 712 are selectors. Reference numeral 701 denotes a control signal generation circuit, which has a function of controlling the selectors 710 to 712 added to the function of the control signal generation circuit 130 of FIG. 720 is an input terminal and 730 is an output terminal. 721 to 725 are control signal lines. The functional block 700 can be configured in FIG. The network in wafers 1 and 2 corresponds to FIG. The selector 711 inputs the signals input from the external output terminal 151 and the input terminal 720 of the wafer 1 in FIG. 1 and outputs the signals to the external input terminal 150 of the wafer 2 in FIG. The selector 710 is a signal input from the input terminal 720 and the external output terminal of the wafer 2 in FIG.
Input 151, and enter the external input terminal 1 in FIG.
Output to 50. The selector 712 is the external output terminal 1 of the wafer 1.
51 and the external output terminal 151 of the wafer 2 are input and output to the external outgoing terminal 730. The control signal generation circuit 701 uses the control signal line 72
Control signals are output to selectors 710-712 through 3-725. Further, the control signal lines 721 and 722 allow the wafers 1 and 2 to be connected.
To control. The signal on the input bus 141 of the wafer 1 is the selector 7
Via 11 to the input bus 141 of the wafer 2. Similarly, the wafer 2 can be sent to the wafer 1.
In addition, the signals input to the external input terminal 720 are input to the selectors 710, 7
Through 11, it can be delivered to the input gas 141 of the wafers 1, 2.

As described above, the wafers can be easily connected, and a very large-scale neural network can be learned at high speed.

Further, by configuring the forward network with the wafer 1 and the backward network with the wafer 2, (3)
Since the weight value in the equation can be corrected in each network at the same time, learning can be performed faster.

Further, in this figure, the connection of two wafers is shown, but the connection of three or more wafers can also be made by a simple correction.

Further, the configuration of FIG. 1 may be directly configured on the wafer. Further, each network can be configured by a plurality of wafers.

Next, FIG. 11 will be described. In FIG. 1, 880 and 890 are functional blocks. 801 to 803 and 812 to 814 are selectors. The function block 880 can be configured in FIG. 7 and the function block 890 can be configured in FIG. Also, the function block 88
0,890 can be configured in FIG. 850 and 860 are output buses, and 851 and 861 are input buses. 160a and 160b are multipliers. In the figure, the network on the left corresponds to the forward network, and the network on the right corresponds to the reverse network.

Although a control signal generation circuit and a control signal are not shown in this figure, each functional block and each selector can be controlled by using the same configuration as in FIGS. 1 and 10.

In the forward network, the output of each functional block 880 is output on output bus 850. The function conversion circuits 131 and 132 input the output bus 850, perform function conversion in each of them, and output. The selector 801 inputs the output of the output bus 850 and the outputs of the function conversion circuits 131 and 132, and outputs one of them.
The selector 802 inputs the output of the selector 801 and the output bus 860 of the reverse network. The multiplier 160a receives the output of the selector 802, multiplies it by a constant η, and outputs it. The selector 803 inputs the output of the selector 801 and the output of the multiplier 160 and outputs them to the input bus 851. Each function block 880 can input the value output to the input bus 851. In the reverse network, the output of each functional block 890 is output to output bus 860. The selector 812 inputs the output bus 860 and the output of the forward network selector 801. The multiplier 160b receives the output of the selector 812, multiplies it by a constant η, and outputs it. The selector 814 inputs the output of the selector 812 and the output of the multiplier 160b. The selector 813 inputs the output of the output bus 860 and the output of the selector 814 and outputs it to the input bus 861.

In the configuration of the figure, the functional block 880 forming the forward network can take in the value of the output bus 860 of the backward network multiplied by η by the multiplier 160a.
Similarly, functional block 89 that constitutes the reverse network.
0 is forward bus output bus 8 by multiplier 160b
The value of 50 can be captured by multiplying it by η. Therefore, the operation of communicating η x _j from the forward network to the reverse network of operation 5 in FIG. 6 can be performed in operation 3. In operation 3, when each functional block broadcasts the output x _j to other functional blocks,
At the same time, it can be multiplied by η by the multiplier 160b and output to the input bus 861 of the reverse network. Similarly, the act of communicating ηδ from the reverse network to the forward network of acts 8 and 12 can be performed in acts 7 and 11. As described above, by stealing the output value to the bus of the other network, the operation 5 in FIG.
The time required for 8 and 12 can be saved. Therefore, learning can be performed at higher speed.

Next, the neurocomputing system according to the present invention will be described. The following embodiment is one mode in which the information processing apparatus described so far is pursued to a practical level. First, the configuration is described in full using FIG. 12 to FIG. 19, and then the operation and the control necessary therefor are described using FIG. 20 to FIG. It should be noted that although there are some points that overlap with those of the above-described embodiments, they will be described once.

FIG. 12 is a diagram showing the entire system. In FIG. 12, 1000 is a host computer and 1001 is a digitizer, which is one of the input values of the host computer 1000. 1003
Indicates a control board. 1030 is WSI board, 1031
Is a WSI substrate, 1050 is a wafer, and wafer 1050 is a WSI substrate 1031
By bonding the WSI substrate 1031 to the WSI board and connecting the WSI substrate 1031 to the WSI board, it is possible to supply power to the wafer 1050 and input / output signals to / from the outside. Reference numeral 1080 is a mode signal line, 1081 is an address signal line, 1082 is a clock signal line, 1083 is an input data line, and 1084 is an output data line. Control board 10
03 and each WSI board 1030 is an internal bus 1020 of 1080 to 1084
Connect by. On the control board 1003, 1010 is an input layer memory, 1011 is an output layer memory, 1012 is a teacher signal memory, 1013 is a sequence RAM, and 1002 is a clock oscillator. 1021 indicates an external bus. Host computer 1000
Is accessible to each memory 1010 to 1013 on the control board 1003 through the external bus 1021, and can be arbitrarily read and written. 1085 is a decoder, 10
40 is an output buffer and 1041 is an input buffer. The input layer memory 1010 and the teacher signal memory 1012 can each receive the address signal line 1081 and send the contents of the memory to the input data line 1083 through the output buffer 1040. Controlled by 1085. The decoder 1085 mode signal line 1080 and the address signal line 1081 are input and decoded. Sequence RA
M1013 generates a mode signal and an address signal. 1086
Is a pointer, which inputs the clock sent from the pulse generator 1002 through the clock signal line 1082 and outputs 1
Increase by one. By repeating this, the address of the instruction written in the sequence RAM 1013 is designated. Sequence RA
Instructions of a mode signal and an address signal to be executed are sequentially written in advance in each address of M1013. Also,
1051 is a selector for sequence RAM 1013 or external bus 102
The command output from 1 is output to the mode signal line 1080 and the address signal line 1081. 1052 is a selector which outputs the value output from the output data line 1084 or the external bus 1021 to the input data line 1083.

The teacher signal of the output layer is written in the teacher signal memory 1012. The input layer memory 1010 functions as an input layer neuron, and the value of the input layer neuron is written therein.
The output layer memory 1011 inputs the output data line 1084 and the address signal line 1081 through the input buffer 1041 and takes in the output of the output layer neuron.

WSI board 1030 has mode signal line 1080 and address signal line 108
1. The clock signal line 1082 and the input data line 1083 are connected to the wafer 1050 through the input buffer 1041, and the signal output from the wafer 1050 is output to the output data line 1084 through the output buffer 1040 controlled by the decoder 1085. Further, the decoder 1085 performs decoding using the mode signal line 1080 and the address signal line 1081. Further, when writing data from the host computer 1000 to the latch, memory, etc. in the wafer 1050, FIG. 13 performed when the own board address is specified is a diagram showing the configuration of the wafer 1050. The wafer 1050 is composed of 1060 N blocks, 1061 IO blocks, 1062 B blocks, and 1063 P blocks. The blocks are arranged at equal intervals in the vertical (column) direction and the horizontal (row) direction. IO blocks 1061 are arranged on the lower side of the central row of the wafer 1050, and seven B blocks 1062 are arranged on the other central rows. Input / output to / from the outside is performed by the IO block 1061 on the lower side. N blocks 1060 are arranged other than the central row, and P blocks 1063 are arranged in the peripheral portion. The P block 1063 is for performing quality control in the semiconductor process of the wafer. N block 1060 has
A neuron block is installed, and a function conversion table is installed in the B block 1062. The IO block 1061 has a function of performing input / output with the outside in addition to the function of the B block 1062. IO block 1061, B block 1062, N block 10
Details of 60 will be described with reference to FIGS. 16 and 17.

FIG. 14 is a diagram showing the structure and arrangement of signal lines in the wafer 1050. A bonding pad 1201 is provided on the lower side of the IO block 1061, and a signal wire is connected to the outside by a bonding wire. The signal line to each N block 1060 is an IO block.
In 1061 and each B block 1062, first, a signal is transmitted in the vertical direction, and further, a signal is transmitted in each horizontal direction. N block
The output of the neuron block in 1060 is
It is collected in the IO block 1061 and the B block 1062, and function conversion is performed there, and it is sent to the IO block 1061 on the lower side for input / output and output to the outside.

FIG. 15 shows a method of supplying power to the blocks 1060 to 1062 of the wafer 1050 and the configuration of the WSI substrate 1031. In FIG. 15, 1301 to 1303 are bonding pads 1304 and bonding wires 1305 are connectors. 1310 is each block 10
It is a power supply bus for supplying power to 60 to 1062. By connecting the connector 1305 to the WSI board 1030, the power and signal lines are connected to the WSI board 1031. The power supply line is drawn from the bonding pads 1301 and 1304 through the WSI substrate 1031 to the collector 1305. In addition, the signal line is pulled out from the bonding pad 1303 to the connector 1305. Wafer 10
Within the block 50, power supply buses 1310 are provided in a grid pattern between the blocks 1060 to 1062, and the blocks 1060 to 1062 are connected to the power supply bus 1310. A bonding pad 1302 is provided at the end of the power supply bus 1310. Bonding pad 13 on wafer 1050
02 and the bonding pads 1301 and 1304 on the WSI substrate 1031 are connected by the bonding wire 1304,
Power can be supplied to each of the blocks 1060 to 1062.
Also, the bonding pad 1201 of FIG.
By connecting the bonding pad 1303 on the WSI substrate 1031 with the bonding wire 1304, the internal lotus 1020 and the IO block 1061 shown in FIG. 12 can be connected.

FIG. 16 shows the B block 1062. The IO block 1061 has an input / output function with the outside in addition to the function of the B block diagram 1062.
In FIG. 16, 1070 is a conversion table, 1071 is a control circuit, and 1072 to 1076 are selectors. IO block 1061, B
Block 1062 is an external or IO block connected to the bottom side
The mode signal line 1080, the address signal line 1081, and the clock signal line 1082 are input from 1061 or the B block 1062, and output to the B block 1062 connected to the upper side and the N block 1060 connected to the side. Input data line 1083 is input from the lower side, selector 1073, conversion table 1070, selector 107
4. Has a path that outputs to the left and right sides through the selector 1076.
Further, it is possible to output only to the left and right sides through the selector 1076 without passing through the conversion table 1070. The output data sensor 1084 input from the N block 1060 connected to the left and right passes through the selector 1072 selector 1073, the conversion table 1070, the selector 1074, and the selector 1075, and outputs to the B block 1062 or IO block 1061 connected to the lower side or the outside. Have a path. Further, the output data line 1084 input from the left and right passes through the selector 1072 and the selector 1075, and can also be output from the lower side. The selectors 1072 to 1076 are controlled by the control circuit 1071. The control circuit 1071 inputs the mode signal line 1080 and the address signal 1081 and generates control signals for the selectors 1072 to 1075. For the configuration of the control circuit 1071, see
It will be described later using the figure. Also, the conversion table 1070
Is connected to the address signal line 1081 and the input data line 1083 for writing.

FIG. 17 shows the N block 1060. The N block 1060 shown in FIG. 17 is arranged on the wafer 1050 on the right side of the center. The N block 1060 located on the left side is laterally symmetrical. In FIG. 17, reference numeral 1100 indicates a neuron block. Reference numerals 1101 and 1103 denote selectors, and 1102 denotes a control circuit. Neuron block in each N block 1060
Six 1100s are installed. Each neuron block 1100
The mode signal line 1080, the address signal line 1081, the clock signal line 1082, and the input data line 1083 are connected to the control circuit 11
02 decodes the address signal line 1081 and the selector 1101 selects the six neuron blocks 11 in the N block 1060.
One of the outputs of 00 is selected and output to the selector 1103.
The other input of the selector 1103 is the output data line 1084 output from the N block 1060 connected to the right side. The output of the selector 1103 is an N block connected to the left side.
Connect to 1060 or B block 1062 or IO block 1061. The control circuit 1102 controls the selector 1103.

FIG. 18 shows details of the neuron block 1100. This figure has almost the same configuration as that shown in FIG. First
In FIG. 18, reference numerals 1121 to 1125 denote latches. Reference numeral 1130 is a multiplier, 1131 is an α multiplier, and 1132 is an η multiplier. 1140 ~ 11
41 indicates an adder.

Reference numerals 1150 to 1152 denote selectors. 1160 is a memory, and 1161 is a pointer that points to the address of the memory. The memory 1160 stores the address of the neuron block 1100 of the connection source, its connection coefficient w, and the previous correction value Δw. Reference numeral 1170 represents a control circuit. The latch 1121 is used only when operating as the reverse network side, and captures f ′ (u) in the equations (4) and (5) sent from the input data line 1083.
Latch 1122 When operating as the forward network side, it takes in the learning signal δ in equation (3) sent from the input data line 1083, and when operating as the reverse network side it is sent from the input data line 1083 (3). Take x in the expression. The latch 1123 is used only when it operates as the output layer of the backward network, and captures the teacher signal t in the equation (4) sent from the input data line 1083. The selector 1150 inputs the coupling coefficient w of the memory 1160, the input data line 1083, the latch 1121, the latch 1122, and the latch 1124, selects two of them, and outputs them to the multiplier 1130. The multiplier 1130 multiplies the input value and inputs it. Selector 1151
Is the η multiplier 1132, the coupling coefficient w of the memory 1160, the multiplier 11
30, latch 1122, latch 1123, and latch 1124 are input, and two of them are selected and output to the adder 1140. Adder 11
40 adds or subtracts the input value and outputs it. latch
1024 takes in the output of the adder 1140. Selector 1152
The latch 1124 and the multiplier 1130 are input, one of them is selected and output to the latch 1125. The latch 1125 outputs to the output data line 1084 through the selectors 1101 and 1103 in FIG. The α multiplier 1131 inputs the previous correction value Δw of the memory 1160, multiplies it by a constant α, and outputs it. The adder 1141 is an α multiplier 11
31 and the multiplier 1130 are input, added, and output. η multiplier
1132 multiplies the output of the adder 1141 by a constant η and outputs the result. The memory 1160 contains the modified coupling coefficient w of the output of the adder 1140.
And the corrected value Δw of the output of the adder 1141 is fetched. The control circuit 1170 inputs the mode signal line 1080, the address signal line 1081, and the clock signal line 1082, and controls the selector and latch in the neuron block 1100 and the memory writing control. The WSI configuration has been described above.
As shown in the figure, it can also be constituted by a chip. In FIG. 19, 1400 is a chip board and 1401 is N
A chip, 1403 is a B chip, and 1404 is a connector. By replacing the chip board 1400 with the WSI board 1030 shown in FIG. 12, a neuron computing system can be similarly constructed. Also chips and WSI
It is also possible to use a mixture of the two. N chip 1401 has the same function as N block 1060, B chip 1403 is B block
It has the same function as 1062. The N chip and the B chip may be connected on the chip board 1400 in the same manner as the wafer 1050.

Learning is performed by repeating various operations in the system configuration of FIG. 12 described above.

FIG. 25 is a diagram showing a part of the mode signal output to the mode signal line 1080 of FIG. The simple operation contents for each mode signal are shown.

The system of FIG. 12 performs various operations according to the mode signal of this figure. Mode signal line 1080 1,3,5,8,9,11,13,1
By setting 5,17,19, the initial value of each latch and memory can be written. Learning operation is mode signal when
Use 24 to 32. The order of setting will be described later with reference to FIG.

According to FIG. 25, how the system of FIG. 12 operates will be explained, and accordingly, the sequence RAM 1013 of FIG.
The functions of the control circuit 1071 in FIG. 16, the control circuit 1102 in FIG. 17, and the control circuit 1170 in FIG. 18 will also be described.

First, referring to FIG. 20, IO block 1061, B block 10
62, N block 1060 and addresses for accessing the neuron block 1100 in the N block 1060 will be described. The address of each neuron block 1100 during the learning operation operates with a relative address. WSI is likely to be unable to use all neuron blocks 1100 due to defects. Further, in the present invention, the neuron blocks 1100 forming the forward network and the backward network have a one-to-one correspondence. Therefore, each neuron block 1100 can be set to any relative address. When setting a relative address, access with an absolute address. FIG. 20A shows the absolute address of each block in the wafer 1050. Further, FIG. 20 (b) shows the bit structure of the absolute address. Wafer 1050
Since there are a total of 288 neuron blocks 1100 inside, the absolute address within the wafer is 9 bits. Also,
Add 4 bits to the board address. The board address is necessary for decoding 1085 in FIG. 12 and need not be sent to the wafer 1050. When connecting many boards, increase the bits of the board address. Of the absolute address within the wafer, 3 bits indicate the row address, 1 bit indicates the right / left address, 2 bits indicate the column address, and 3 bits indicate the neuron block address in the N block.
The row address indicates rows 0 to 7 from the lower stage of the wafer 1050. The right / left address indicates right or left with respect to the center. The column address indicates which column. The neuron block address in the N block indicates any one of the 6 neuron blocks in the N block 1060. The neuron block address of the address in the N block is shown in FIG. 20 (b). Further, FIG. 20 (c) shows an absolute address in the chip board 1400.

FIG. 21 shows the control circuit 1071 of FIG. Further, FIG. 22 shows the control circuit 1102 of FIG. 17 and FIG. 23 shows the control circuit 1170 of FIG.
Indicates. First, the address decoding method will be described with reference to these three figures.

In FIG. 21, reference numeral 1501 is a column decode for decoding the column address of the absolute address.

Reference numeral 1502 denotes a row decoder which decodes an absolute address row address. Reference numeral 1503 denotes a decoder for decoding the mode signal line. Reference numerals 1506 to 1507 denote latches. Reference numeral 1504 represents an inverter circuit, and 1505 represents an AND circuit. Reference numerals 1511 to 1512 denote subtraction circuits, which subtract the relative address sent to the address signal line 1081 and the values previously written in the latches 1506 and 1507, and output the sign of the result.

Reference numerals 1509 to 1510 denote selectors for selecting and outputting either the result of the decoder by the absolute address or the result of the decoding by the relative address. Selector 1509 from decoder 1503,
The control line connected to the 1510 selects either an absolute address or a relative address. Reference numeral 1520 indicates an N block select line.

In FIG. 22, 1607 is a decoder for decoding the mode signal line 1080.

1604 is a latch, which can capture the output of the input data line 1083. The control is performed by the decoder 1607. Reference numeral 1603 denotes a subtraction circuit, which subtracts the relative address sent to the address signal line 1081 and the value previously written in the latch 1604, and outputs the sign of the result. Reference numerals 1601 to 1602 denote selectors. The selector 1601 selects and outputs either the N block select line 1520 which is the decoding result of the absolute address of the column address or the decoding result of the relative address. The selector 1602 selects the decoding result of the absolute address and the decoding result of the relative address in the N block. The control line connected to the selectors 1601 and 1602 from the decode 1607 selects an absolute address or a relative address. The N block select line 1520 is twisted and connected to the adjacent block is the N block in each row.
This is because 1060 can have the same pattern. As a result, the man-hours for designing the semiconductor mask can be reduced. A decoder 1605 decodes the address in the N block of the absolute address on the address signal line 1081.

An encoder 1606 inputs the result of decoding the address in each of the six neuron blocks 1100.

In FIG. 23, reference numeral 1701 denotes a latch which takes in its own relative address output to the input data line 1083. In addition to the relative address, whether it operates as the forward network side, the reverse network side, or the reverse network side, further,
Information indicating whether it is the middle layer or the output layer is also fetched. 17
Reference numeral 03 is a comparison circuit, which compares the relative address output to the address signal line 1081 with the relative address of the latch 1701 that has been written in advance and determines whether or not they match.
Send to 02. The result is also output to the decoder 1605 shown in FIG. Reference numeral 1704 is a comparison circuit, which compares the connection source address of the memory 1160 of FIG. 18 with the relative address output to the address signal line 1081. 1702 is a decoder, which is a comparison circuit 1703,
Latch 1701, mode 1081, decoder 1605 in FIG. 22, clock 4 signal line 1082, comparison circuit 1704, selector 16 in FIG.
Input 01 to control the selector, latch, memory, pointer, and latch 1701 in the neuron block 1100.

Although only the items necessary for explaining the operation are shown in FIG. 25, a mode signal for performing writing / reading of a latch, a memory, etc. is also set for inspection.

First, the relative address and the forward network or the reverse network,
In addition, 2 bits of information indicating the intermediate layer or the output layer is set to the latch 170 of the control circuit 1170 of each neuron block 110 of FIG.
Write to 1. The host computer 1000 sets the selectors 1051 and 1052 in FIG. 12 to the external bus 1021 side. The mode signal line 1080 is set to 17 and the absolute address of the neuron block 1100 to be accessed is set to the address signal line 1081. The value to be written is set in the input data line 1083. WS
In the I board 1030, the decoder 1085 decodes the board address, and the wafer 10
Control to write 50 latches, memory, etc.

For absolute address operation, the row decoder 1502 of FIG.
Will decode the row address and will tell if the row is selected. Further, the column decoder 1501 selects a column address, and one to four N blocks 1060 in the row direction are selected by an N block select line 1520 connected to the column address. , N block 1060 on the right or left side is selected. The result of the row decoder 1502 and the right / left address are ANDed by the AND circuit 1505 to be the enable signal of the left decoder 1501. Further, the result of the row decoder 1502 and the logical inversion of the right / left address by the inverter circuit 1504 are ANDed by the AND circuit 1505, and the result is decoded by the AND circuit 1505.
This is the enable signal for 1501. Selected N blocks 10
In the 60, the decoder 1605 shown in FIG.
The result is output to the control circuit 1170 in FIG. 23 in 00. The selection result of the N block of the selector 1601 is also the control circuit 11.
Output to 70. Decoder 1702 of select line control circuit 1170
Under the control of (1), if the mode signal is 17 and the self is selected, the value output to the input signal line 1083 is fetched in the latch 1701. By the above operation, data is written in the latches 1701 of all neuron blocks 1100. However, relative addresses within the same wafer are increased from smaller absolute addresses to larger relative addresses. There may be an empty number. This is related to the control method of the output data line 1084 described below.

Next, the latches 1604 of FIG. 22 of all N blocks 1060 are set. Set the mode signal to 15 and set the latch 1604 to N
The largest value of the relative addresses set in the neuron block 1100 in the block 1060 is written. The mode signal line 1080 is set to 15 by the host computer 1000, and the absolute address of the N block 1060 to be accessed is set to the address signal line 1081. It is not necessary to set the neuron block address in the N block. The value to be written is set in the input data line 1083. Control of the output data line 1084 in the N block 1060 during operation by the relative address, that is,
The selector 1103 shown in FIG. 17 is controlled by the subtraction circuit 1603 shown in FIG.
Thus, it is performed by the sign bit of the result of subtracting the latch 1604 from the address signal line 1081. Since the maximum relative address of the neuron block 1100 in the N block 1060 is written in the latch 1604, the neuron block 1100 outside the N block 1060 has a relative address set larger than the value written in the latch 1604. Has been done. Therefore, when the address signal line 1081 has a value larger than that of the latch 1604, the selector 1103 in FIG. 17 selects the output data line 1084 connected to the adjacent N block 1060.

Next, the maximum relative address set in the neuro block 1100 of that row is written in the latches 1507 of FIG. 21 of all IO blocks 1061 and B blocks 1062. The mode signal line 1080 is set to 11 by the host computer 1000, and the absolute address row address of the IO block 1061 or B block 1062 to be accessed is set to the address signal line 1081. It is not necessary to set right / left addresses, column addresses, and addresses within N blocks. The value to be written is set in the input data line 1083.

Output data sensor in IO block 1061 and B block 1062
The control of 1084, that is, the control of the selector 1075 of FIG.
The subtraction circuit 1512 shown in the figure performs the operation according to the sign bit of the result obtained by the latch 1507 from the address signal line 1081. Latch 15
Since the maximum relative address of the neuron block 1100 in that row is written in 07, the neuron block 1100 above that block is set with a relative address larger than the value of the latch 1507. for that reason,
When the address signal line 1081 has a value larger than that of the latch 1507, the selector 1075 shown in FIG.
Select the output data line 1084 connected to.

Next, for all IO blocks 1061 and B blocks 1062, the latch 1506 in FIG.
Write the maximum relative address set in. The mode signal line 1080 is set to 13 by the host computer 1000, and the IO block 1061 or B block 1062 is accessed.
The row address of the absolute address of is set to the address signal line 1081. It is not necessary to set right / left addresses, column addresses, and addresses within N blocks. The value to be written is set in the input data center 1083. The control of selector 1072 in FIG.
The subtraction circuit 1511 shown in the figure performs the operation based on the sign bit of the result obtained by subtracting the latch 1506 from the address signal line 1081. latch
1506 contains the neuron block 1100 to the right of that row.
Since the maximum relative address of is written, a relative address larger than the value of the latch 1506 is set in the left neuron block 1100. Therefore, when the address signal line 1081 has a value larger than that of the latch 1506, the selector 1072 in FIG. 16 selects the output data sensor 1084 connected to the left side.

Next, the conversion table 1070 of FIG. 16 of the IO block diagram 1061 and B block 1062 is written. The conversion table 1070 of the wafer 1050 forming the forward network uses the f conversion of equation (1), and the conversion table 1070 of the wafer 1050 forming the reverse network uses the f ′ conversion of equations (4) and (5). To do.

The mode signal line 1080 is set to 9 by the host computer 1000.
I / O block to access address line 1081
1061 and B block 1062 sets the absolute address row address, and sets the right / left address, column address, and bit in the N block address to the address to be accessed in the conversion table 1070. If 7 bits are insufficient, the number of bits of the address signal line 1081 is increased. Input data line 10
Set the value you want to write in 83. IO block 1061, each B
In block 1062, the row decoder 1502 shown in FIG. 21 decodes the row address, and if the row address is selected, the value output to the input data line 1083 is written in the conversion table 1070.

Next, the output of the input layer neuron is written in the input layer memory 1010 of FIG. 12 and the teacher signal is written in the teacher signal memory 1012. In general, since the neural network trains various patterns, the input layer memory 1010 is segmented for each pattern. Similarly, the teacher signal memory 1012 for writing the teacher signal corresponding to each pattern is also segmented. Change the segment to be accessed for each pattern. The host computer 1000 sends a mode signal of 1,
3, the address signal is set to the line 1081, the address is set to the signal line 1081, and the value is written in each memory.

Next, the initial value of the coupling coefficient is written in the memory 1160 of the neuron block 1100 to be used. Set the mode signal to 19, first set the address of the neuron block 1100 that accesses the address signal line 1081, then set the write address of the memory 1160 to the address signal line 1081 and set the value to write to the input data line 1083. To do. in this way,
The address signal line 1081 may be used in a time division manner, or the pointer 1161 may be set to 0 at the beginning to write from address 0, and the pointer may be sequentially advanced by 1 to be written into the memory 1160. In memory 1160, connect neuron blocks
The relative address of 110, the coupling coefficient w with it, and the previous correction value Δw are written. However, the initial value of Δw may be 0. Further, when the learning operation is interrupted halfway, the intermediate result is read to the host computer 1000, and when the learning is restarted, the necessary value is written accordingly. Also, the initial values of the corresponding coupling coefficients of the forward network and the backward network must be the same.

Next, as shown in FIG. 24, a series of mode signals for learning and a relative address of the neuron block 1100 for outputting are written in the sequence RAM 1013.

Figure 26 shows an example. In FIG. 26, a sequence RAM 101 for learning a neural network having three input layer neurons, two intermediate layer neurons, and two output layer neurons.
Here is an example of 3. FIG. 26 shows learning of one pattern, but when there are two or more patterns, it is necessary to change the segments of the input layer memory and the teacher signal memory each time the pattern is changed.

The learning operation will be described below with reference to FIG. The relative address of the input layer neuron is 0 to 2, the relative address of the intermediate layer neuron is 3 to 4, and the relative address of the output layer neuron is 5 to 6. Two boards are used, where the board address forming the forward network is 3, the board address forming the reverse network is 4.

The backward network does not require input layer neurons, so a total of eight neuron blocks 1100 are used. The board addresses of the input layer memory and the teacher signal memory are 0 and 1, respectively.

First, the pump 1086 is reset. Sequence RAM 1013
Outputs the command and the address written in the address 0 to the mode signal line 1080 and the address signal line 1081, respectively. Therefore, the mode signal line 1080 is set to 24 and the address 5 is set to the address signal line. Also, the board address is set to 1. The decoder 1085 decodes the board address and outputs the output of the teacher signal memory 1012 to the input data line 1083 through the output buffer 1040. The teacher signal memory 1012 outputs the value of the address 5 designated by the address signal line 1081. The neuron block 1100 having a relative address of 5 in the output layer on the backward network side outputs its own teacher signal output to the input data line 1083 to the eighteenth signal.
Capture in latch 1123 as shown. Similarly, the neuron block 100 in the output layer of the backward network with relative address 6
Also to.

Here, in order to write data in a latch, a memory, etc., an address hold time or write data hold time is required. Therefore, the timing of the write enable signal to them is generated by the clock signal of the clock signal line 1082.

Next, when the pointer 1086 inputs a clock from the clock signal line 1082, the pointer 1086 in FIG. 12 advances by 1 and moves to the next instruction. The mode signal line 1080 is set to 25, and the pointer 1161 in FIG. 18 is reset. Also, the contents of the latch 1124 in the neuron block 110 of FIG. 18 are reset.

The next instruction performs a forward operation. Mode signal line 1080
26, the address signal lines are connected to the neuron blocks from the input layer to the output layer in ascending order of relative address.
1100 specifies. As the board address, board address 0 is set in the input layer, and board address 3 is set in the intermediate layer and the output layer. The designated input layer memory 1010 outputs the value to the input data line 1083 and broadcasts it to all neuron blocks 1100. Neuron block
In the 1100, the mode signal line 1080 set to 26
The decoder 1702 shown in the figure decodes and controls each selector of the neuron block 1100 shown in FIG. The selector 1150 multiplies the coupling coefficient w of the memory 1160 and the input data line 1083 by the multiplier 1130.
Output to. The multiplier 1130 performs a synapse operation, that is, a multiplication between the coupling coefficient w and the output x of the neuron. The selector 1151 writes the output of the adder 1140 into the latch 1124 which outputs the multiplier 1130 and the latch 1124 to the adder 1140. The selector 1152 selects the selector 114. Latch 1125 captures that value. By the comparison circuit 1704 of the control circuit 1170 of FIG. 23, the connection source address of the memory 1160 and the address signal line 1081
The outputs of the latch 1124 and the multiplier 1130 are added, and the results are written in the latch 1124. Then, the point 1161 shown in FIG.
Advances the value of by one. In this way, the address signal lines are sequentially
Output source relative address sent to 1081 and input data line 10
The output sent to 83 is input and cumulative addition is performed. The memory 1160 is written in the ascending order of the source address. Further, the address signal line 1081 and its own relative address written in the latch 1701 of FIG. 23 are compared by the comparison circuit 1703. Selector 110 in FIG. 17
It is sent to the selector 1072 of FIG. 16 through 1, 1103 and the output data line 1084. In the decoder 1503 of FIG. 21 of the IO block 1061 and the B block 1062, when the mode signal is set to 26, the selector 1073 selects the selector 1072 side and the selector 1074 selects the conversion table 1070. Further, the selector 1076 selects the input data line 1083 side so that the input data line 1083 sent from the lower side is sent to the side N block 1060 as it is. The selectors 1072 and 1075 control the address signal line 1081 by the method described above. This allows
The internal energy value u which is the output of the neuron block 1100 whose relative address is specified is the conversion table 10 of that row.
It is sent to 70, converted into the output x of the neuron, and sent to the control board 1003 of FIG. 12 through the output data line 1084. The selector 1052 in FIG. 12 outputs the value of the output data line 1084 as it is to the input data line 1083.

Also, on the reverse network side in this mode,
In FIG. 18, when the relative address of itself is output to the address signal line 1081, the value of the input data line 1083 is taken into the latch 1122.

In the next operation, the mode signal line 1080 is set to 27, and the relative addresses of the intermediate layer and the output layer are sequentially set to the address signal 1081. The neuron block 1100 of the forward network specified with the relative address again returns the contents of the 18th latch 1125, that is, the internal energy value u to the IO block of FIG.
1061, B block 1062. In mode 27, the selector 1074 of the IO block 1061 and B block 1062 is a selector
Select 1072. Therefore, the internal energy value u is output to the lower output data line 1084 as it is. The decoding method of the address signal line 1081 is the same as in mode 26. The internal energy value u passes through the control board of FIG. 12 and is output to the output data line 1084. On the other hand, in the reverse direction IO block 1061 and B block 1062, the selector 1073 is set to the input data sensor 1083 side,
1074 is the conversion table 1071 side, selector 1076 is the selector
Set on the 1074 side. The internal energy value u sent through the input data line 1083 is converted into f ′ (u) by the conversion table 1071 and becomes f ′ (u), which is output to the side N block 1060 through the input data line 1083. In the reverse direction network neuron block 1100 of FIG. 18, if the address output to the address signal line 1081 and its own relative address match, f ′ (u) output to the input data line 1083 is taken into the latch 1121.

Next, the mode signal line 1080 is set to 28. The neuron block 1100 in the output layer of the backward network sets the selector as follows. In FIG. 18, the selector 1151 outputs the latch 1122 and the latch 1123 to the addition and subtraction 1140. Adder
Calculate (t−x) by 1140 and write to latch 1124. Selector 1150 multiplies latch 1124 and latch 1121 by multiplier 11
Output to 30. The multiplier 1130 causes the latch 1124 (t−
x) is multiplied by f ′ (u) of the latch 1121 to calculate the learning signal δ of the output layer. The selector 1152 outputs the output of the multiplier 1130 to the latch 1125. The learning signal δ is written in the latch 1125.

In the next instruction, the mode signal line 1080 is set to 25, and the pointer 1161 in FIG. 18 is reset. Also, the latch 1124 in the neuron block 1100 in FIG. 18 is reset.

In the next instruction, the mode signal line 1080 is set to 29, and the relative address of the output layer is set to the address signal line 1081 in ascending order. The neuron block 1100 in FIG. 18 of the reverse network with relative addresses specifies the contents of the latch 1125, that is, the learning signal δ of the output layer, in the IO block 10 in FIG.
61, B block 1062. In mode 29, the selector 1074 in FIG. 16 of the IO block 1061 and B block 1062 selects the selector 1072, so the learning signal δ is output to the output data line 1084 as it is. The decoding of the address signal line 1081 is the same as in mode 26. Learning signal δ
Is output to the input data line 1083 through the control board 1003 shown in FIG. In the IO block 1061 and B block 1062 of the forward network, the selector 1076 of FIG. 16 is set to the input data line 1083 side connected to the lower side.
The learning signal δ sent from the reverse network is output to the side N block 1060. In the neuron block 110 of FIG. 18 of the forward network, the address signal line 1
If the address output to 081 matches its own relative address, the learning signal δ output to the input data line 1083 is captured in the latch 1122.

In the middle layer neuron block 1100 of the reverse network, the selector 1150 outputs the coupling coefficient w of the memory 1160 and the input data line 1083 to the multiplier 1130. Multiplier 1130
Thus, the engagement coefficient w is multiplied by the learning signal δ. The selector 1151 outputs the multiplier 1130 and the latch 1124 to the adder 1140. The latch 124 captures the output of the adder 1140. 23rd
The comparison circuit 1704 in the figure compares the coupling source address output from the memory 1160 with the address signal line 1081, and when they match, the outputs of the latch 1124 and the multiplier 1130 are added and written to the latch 1124. Then, the value of the pointer 1161 is advanced by one. In this way, they are sequentially sent to the address signal line 1081 and the input data line 1083. The relative address of the output neuron block 1100 and its output are input, and Σw · δ is cumulatively added.

The next instruction sets the mode signal line 1080 to 30. The neuron block 1100 in the output layer of the backward network sets the selector as follows. In FIG. 18, a selector 1150 outputs a latch 1124 and a latch 1121 to a multiplier 1130 by a multiplier 1130.
(U) is multiplied to calculate the learning signal δ of the intermediate layer. The selector 1152 outputs the output of the multiplier 1130 to the latch 1125. The learning signal δ is written in the latch 1125.

Next, the mode signal line 1080 is set to 29, and the learning signal δ of the intermediate layer is broadcasted to all the neuron blocks 1100. The forward network neuron block 1100
The value of the address signal line 1081 is compared with the own relative address comparison circuit 1703 written in the latch 1701 shown in FIG.

If there are two or more intermediate layers, or if the mode is
30 commands, calculate learning signal δ, and then mode 29
To send to the forward network.

In the next instruction, the mode signal line 1080 is set to 25, and the pointer 1161 in FIG. 18 is reset.

Next, the mode signal line 1080 is set to 31, and the address signal lines 1081 are designated from the input layer toward the intermediate layer in ascending order of relative address. Regarding the input layer, the input layer memory 10
Output from 10. With respect to the intermediate layer, the neuron block 1100 in the intermediate layer of the forward network f-converts the internal energy value u of the conversion table 1070 of the IO block 1061 and the B block 1062, and broadcasts the value. Each selector of the neuron block 1100 in FIG. 18 of the forward network is controlled as follows. The selector 1150 outputs the latch 1122 and the input data line 1083 to the multiplier 1130. The multiplier 1130 multiplies the learning signal δ of the latch 1122 by the output x of the neuron. The selector 1151 outputs the coupling coefficient w of the memory 1160 and the η multiplier 1132 to the adder 1140. Further, the α multiplier 1131 inputs the previous correction value Δw ′ in the memory 1160, multiplies it by α, and outputs αΔw ′. The adder 1141 inputs the output δ · x of the multiplier 1130 and the output αΔw ′ of the α multiplier 1131, adds them, and outputs them to the η multiplier 1132. η multiplier 11
The 32 output is the modified value. In Δw ′ of memory 1160,
Write the value before multiplying by η.

w _ji ^{n + 1} = w _ji ^η + η · Δw ′ _ji ^{n + 1} (6) Δw ′ _ji ^{n + 1} = δ _j · x _i + α · Δw ′ _ji ^{n + 1} (7) The constant η has a wide range of values. By doing so, the number of bits of Δw ′ written in the memory 1160 can be saved. The adder 1140 corrects the coupling coefficient w and writes the correction result in the memory 1160. Further, the above correction operation is performed by the comparison circuit 1704 in FIG.
This is performed when the address of the connection source of 60 and the address signal line 1081 match.

Next, the mode signal line 1080 is set to 25, and the pointer 1161 in FIG. 18 is reset.

Next, the mode signal line 1080 is set to 32, the relative address of the neuron block 1100 in the output layer of the backward network is set to the address signal line 1081, and the learning signal δ is broadcast. If there are two or more intermediate layers, the learning signal δ of the intermediate layer is also broadcast. Each selector of the neuron block 1100 of FIG. 18 of the backward network is set in the same manner as the correction of the coupling coefficient w of the forward network of mode 31. Latch 11 by multiplier 1130
The output x of the 22 neurons and the learning signal δ output to the input data line 1083 are multiplied to obtain the η multiplier 1132 and the adder 114.
The corrected value Δw ′ and the corrected coupling coefficient w are calculated by 0, the α multiplier 1131 and the adder 1141 and written in the memory 1160.

The learning operation can be performed by changing the segments of the input layer memory and the teacher signal memory for each pattern for each pattern. Further, the output layer memory 1011 captures the output of the output layer neuron of the forward network output to the output data line 1084. Host computer 10
00 can read the output layer memory 1011 from the external bus 1021 and check whether the output is as expected. As an example of the application of FIG. 12, for example, by the digitizer 1001 of FIG. 12, input handwritten characters, normalize the coordinates of the characters input by the host computer 1000,
A system for recognizing handwritten characters can be created as an input of a neural network. In the system shown in FIG. 12, the time required for learning once for each pattern of a hierarchical neural network of p input layers, q intermediate layers, and r output layers is
When the cumulative addition time required for one time is t, it is approximately (2p + 4q + 5r) · t (s) (8). For example, if p is 256 (16 dots x 16 dots), q is 100, r is 26, and t is 100 ms for handwritten character recognition,
One pattern can be learned once in about 0.1 ms. Even if it takes 1000 times to learn one character,
You can learn 10-character patterns in just 1 second.

[Effect of the Invention] According to the present invention, a neural network can be learned at high speed. Further, it is convenient to form a wafer scale integrated circuit, and it is also possible to form a plurality of wafers. Also, the number of designing steps is small. Therefore, a large-scale neural network can be calculated at high speed, and a compact wafer-scale integrated circuit for a neurocomputer can be manufactured.

[Brief description of drawings]

FIG. 1 is a diagram showing an embodiment of the present invention, FIG. 2 is a diagram showing a neuron model constituting a neural network, FIG. 3 is a diagram showing a three-layer hierarchical neural network, and FIG. 4 is a neural network. FIG. 5 is a diagram showing a dedicated hardware wafer for fast calculation, FIG. 5 is a diagram explaining the details of the operation of the present invention, FIG. 6 is a diagram showing the flow of calculation according to the present invention, and FIG.
FIG. 8, FIG. 8 and FIG. 9 are diagrams showing details of an example of functional blocks used in the present invention, FIG. 10 and FIG. 11 are diagrams showing another embodiment of the present invention, and FIG. FIG. 13 is a diagram showing the entire neuro-computing system of the present invention, FIG. 13 is a diagram showing the structure of a wafer of the neuro-computing system of the present invention, FIG. 14 is a diagram showing the structure and arrangement of signal lines on the wafer, and FIG. Shows the method of supplying power to each block of the wafer and the structure of the WSI substrate, FIG. 16 shows the B block (IO block), FIG. 17 shows the N block, and FIG. 18 shows the neuron block. Figure showing details, No. 19
FIG. 20 is a diagram for explaining the case where the system of the present invention is configured by a chip, and FIG. 20 is an IO block, B block, N
FIG. 21 is a diagram for explaining addresses for accessing neuron blocks in blocks and N blocks, and FIG.
FIG. 22 is a diagram showing the control circuit of FIG. 22, FIG. 22 is a diagram showing the control circuit of FIG. 17, FIG. 23 is a diagram showing the control circuit of FIG. 18, FIG. 24 is a diagram explaining a sequence RAM, and FIG. FIG. 26 is a diagram showing an example of a mode signal and its operation, and FIG. 26 is a diagram showing an example of an instruction to write to the sequence RAM of FIG. Explanation of symbols 120F, 121F, 120B, 121B …… Computing unit, 123,124 …… Control circuit, 125-127 …… Selector, 128 …… Buffer, 130
...... Control signal generation circuit, 131,132 …… Function conversion circuit, 140
...... Output bus, 141 …… Input bus, 142 to 145 …… Control signal line, 150 …… External input terminal, 151 …… External output terminal, 16
0 …… Multiplier.

Front page continued (72) Inventor Masayoshi Yagyu 1-280, Higashi Koikekubo, Kokubunji, Tokyo Inside Central Research Laboratory, Hitachi, Ltd. (72) Minoru Yamada 1-280, East Koikekubo, Kokubunji, Tokyo Inside Central Research Center, Hitachi Ltd. (72) Inventor Katsunari Shibata 1-280, Higashi Koigokubo, Kokubunji, Tokyo, Central Research Laboratory, Hitachi, Ltd. (56) Reference JP-A-60-219837 (JP, A)

Claims (18)

[Claims] 1. An input layer configured by at least one calculation unit, to which a signal from the outside is input, an intermediate layer which inputs an output of the input layer and outputs an output value corresponding to the input value, and the input layer. Or a first hierarchical network having an output layer for inputting the output of the middle layer and outputting an output value corresponding to the input value, wherein the first network determines the output value in the forward direction from the input layer toward the output layer. A hierarchical network comprising at least an intermediate layer and an output layer connected to the intermediate layer, the hierarchical network being configured by at least one calculation unit and corresponding to the first network. The second network that determines the output value in the opposite direction and the first and second networks communicate with each other to modify the output value of the calculation unit of each network. The information processing apparatus characterized by comprising means for modifying the value, the. 2. The calculation unit calculates a sum of values input from another calculation unit, weighting the input values,
The information processing apparatus according to claim 1, comprising a neuron calculation unit that determines an output value corresponding to the summation layer. 3. The information processing apparatus according to claim 1, wherein the input layer of the first network has a function of storing an input signal from the outside. 4. The information processing apparatus according to claim 1, wherein the plurality of calculation units forming the intermediate layer or the output layer are integrated on one wafer scale integrated circuit. 5. The first network comprises at least one
The information processing apparatus according to claim 1, wherein the information processing apparatus is integrated on one first wafer scale integrated circuit, and the second network is integrated on at least one second wafer scale integrated circuit. 6. An input layer of the first network is configured as a first functional block, an intermediate layer and an output layer of the first network are configured as a second functional block, and an intermediate layer of the second network. The information processing apparatus according to claim 1, wherein the output layer and the output layer are configured as a third functional block. 7. A communication between the first functional block and the second functional block, a communication between the second functional blocks, and a third functional block having at least one data path. 7. The information processing apparatus according to claim 6, wherein the communication between the second functional block and the third functional block is performed by the data path. 8. The information processing apparatus according to claim 4, wherein the second functional block and the third functional block are configured by the same circuit. 9. An input layer comprising at least one calculation unit, to which a signal from the outside is input, an intermediate layer which inputs an output of the input layer and outputs an output value corresponding to the input value, and the input layer. Or a hierarchical network having an output layer for inputting an output of the middle layer and outputting an output value corresponding to the input value, wherein the input layer is a first functional block and the middle layer and the output layer are a second function. The first functional block and the second functional block hold weight values for output values on the input layer side, and the first functional block and the second functional block are arranged in a forward direction from the input layer to the output layer. A first network for determining an output value of a second functional block, and at least one calculation unit, corresponding to the first network, at least an intermediate layer and A hierarchical network having an output layer connected to an inter-layer, wherein an intermediate layer and an output layer are composed of a third functional block corresponding to the second functional block.
The second functional block holds a weight value for the output on the output layer side and determines the output value of the third functional block in the reverse direction from the output layer to the intermediate layer, and the second function. A block and means for communicating the third functional block with each other, and the weight value held by the second functional block and the weight value held by the third functional block are made to correspond one-to-one. , An information processing device characterized by modifying the same value. 10. A communication between the first functional block and the second functional block, a communication between the second functional blocks, and a third functional block having at least one data bus. 10. The information processing apparatus according to claim 9, wherein the communication and the communication between the second functional block and the third functional block are performed by the data bus. 11. A plurality of functional blocks connected to at least one first data bus and at least one second data bus, and a value converted from the functional block to the first data bus by a function. Then, the first function calculating means for outputting to the second data bus and the value output from the functional block to the first data bus are converted by the derivative of the above function to the second data bus. It has a second function calculating means for outputting and a means for controlling the output from the functional block to the first data bus, and the functional block calculates the sum of the values weighted to the input value, A means for calculating a neuron model for determining an output value corresponding to the sum value, a means for outputting the output of the functional block to the first data bus, and a means for calculating the first or second function. An information processing apparatus comprising means for inputting a value output to the second data bus. 12. A hierarchical network comprising at least one calculation unit, comprising an input layer, an intermediate layer and an output layer, wherein the input layer is a first functional block and the intermediate layer and the output layer are a second functional block. The first functional block is connected to the first data bus, the second functional block is connected to the first data bus and the second data bus, and is output from the input layer. The first functional block and the second functional block in the forward direction toward the layer.
A first network for determining output values of the functional blocks of, and a hierarchical network comprising at least one calculation unit and having at least an intermediate layer and an output layer corresponding to the first network, The layer and the output layer are composed of a third functional block corresponding to the second functional block, and the third functional block is connected to the first data bus and the second data bus,
A second network that determines the output value of the third functional block in the reverse direction from the output layer to the intermediate layer, and the value output to the second data bus is converted by a function to generate the first data. A first function calculating means for outputting to the bus; a means for directly outputting the value to be output to the second data bus to the first data bus; and a value to be output to the second data bus. And a second function calculating means for converting by a derivative of a function and outputting to the first data bus, wherein the second functional block stores a weight value of a neuron model for weighting an input value. Means for calculating the total sum of the weighted input values to perform inner product calculation of the neuron model, and means for rewriting and correcting the weight value, wherein the third functional block is Means for holding a weight value, means for calculating a value necessary to correct the weight value, and means for rewriting and modifying the weight value, and the second functional block corresponds to the third function. An information processing apparatus, wherein the second functional block and the third functional block communicate with each other and modify the weight values held by the second functional block and the third functional block to the same value as the corresponding weight value. 13. The information processing device according to claim 12, wherein the plurality of calculation units forming the intermediate layer or the output layer are integrated on one wafer-scale integrated circuit. 14. The first network is integrated on at least one first wafer-scale integrated circuit, and the second network is integrated on at least one second wafer-scale integrated circuit. An information processing apparatus according to claim 12, 15. The information processing apparatus according to claim 14, wherein the first wafer scale integrated circuit and the second wafer scale integrated circuit are configured by the same circuit. 16. The information processing apparatus according to claim 13, wherein the first functional block and the second functional block have the same circuit configuration. 17. A learning method for an information processing apparatus having a hierarchical network including an input layer, an intermediate layer, and an output layer, the method comprising: a forward direction from the input layer to the output layer; a forward direction to the output layer. The output value is determined in the above, and the intermediate layer and the output layer hold the first weight value for the output value on the input layer side. The input layer and the intermediate layer retain the second weight for the learning signal value on the output layer side, and the first weight value and the second weight value are corrected to the same value by the learning. A learning method for an information processing device, comprising: 18. A first data bus (140) and a second data bus (141) for inputting data from outside the information processing apparatus and outputting data to the outside.
And holding the weight value of the neuron model, inputting data from the second data bus (141), weighting each input data according to the weight value of the held neuron model, and weighting A first that outputs the output corresponding to the inner product to the first data bus (140) by calculating the sum of data to calculate the inner product of the neuron model
Calculation unit (120F), second calculation unit (121
F), the third calculation unit (121B), and the fourth calculation unit (120B), and the inner product of the neuron models calculated by the first calculation unit (120F) and the second calculation unit (121F) From the first data bus (140),
It is calculated by a first function calculating means (131) for converting by a function and outputting to the second data bus (141), the first calculating unit (120F) and the second calculating unit (121F). And a second function calculating means (132) for inputting the inner product of the neuron model from the first data bus, converting the inner product by the function, and outputting the second product to the second data bus (141). One calculation unit (120F), the second calculation unit (121F), the third calculation unit (121B), the fourth calculation unit (120F), the first function calculation means (131), and A learning method for an information processing apparatus, comprising: a control means (130, 125, 126, 127) for controlling the second function calculation means, wherein the information processing apparatus inputs data from the outside to perform the first calculation. Unit (120F), second calculation unit above (121F) and a forward network having the first function calculating means (131),
Determining the output value of the forward network, inputting the expected output value for a given input, the inverse having the third computing unit (121B) and the fourth computing unit (120B) The learning signal value of the backward network is determined by performing calculation by the directional network and the second function calculating means (132), and the output value of each of the first to fourth calculation units and the learning signal. A learning method for an information processing apparatus, characterized in that the weight value is corrected by adding a value proportional to the product of the values to the weight value of the neuron model.