JPH0149985B2 - - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION The present invention relates to a self-organizing storage device that has a multi-layered structure of information processing layers and organizes the connections between each layer through learning. It is designed to self-organize by checking the pattern of

In general, as self-organizing storage devices aimed at improving pattern recognition functions, for example, the one described in Japanese Patent Application Laid-Open No. 51-35247 "Self-organizing Formation Method" filed by the applicant has been proposed. However, such conventional devices are configured to obtain the final output only by processing the output from each information processing layer in the forward direction. Therefore, even if information about previously stored patterns is input during the self-organization process, such stored pattern information is not involved in self-organization, and the newly input pattern is There was a problem that if there was a lot of overlap with the old and new patterns and they were similar, the old and new patterns would be considered the same.

In other words, in the conventional device described in Japanese Patent Application Laid-Open No. 51-35247, the information signals to be processed in each information processing layer flow only in the forward direction from the input layer to the deeper layers. It was structured as follows. However, when high-order pattern information processing actually occurs in the human brain, information processing proceeds not only in the forward direction as described above, but also by repeatedly matching past memory patterns with new input patterns. The flow of information in the opposite direction is also recognized to play an important role. Therefore, conventional self-organizing memory devices configured to perform information processing based only on forward information flow cannot sufficiently simulate the actual information processing mode in the human brain. The drawback was that there was no.

It is an object of the present invention to eliminate the above-mentioned conventional drawbacks, add the reverse information flow to the forward information flow, and make it possible to perform information processing in a manner that closely approximates the information processing in the human brain. An object of the present invention is to provide a self-organizing storage device that has the following features.

That is, the self-organizing memory device of the present invention sequentially arranges a plurality of information processing layers in which a plurality of excitatory and inhibitory nonlinear threshold elements are mixed and arranged in two dimensions, and the information processing layers are mutually arranged. By providing forward excitatory connections and inhibitory connections and reverse feedback inhibitory connections between them, the connections between the information processing layers are self-organized in correspondence with input information. That is.

That is, the self-organizing storage device of the present invention has the first
As the principle structure will be described later with reference to the figure, in addition to the conventional forward coupling between the information processing layers, a feedback-type inhibitory coupling is provided. This system is designed to self-organize the connections between the information processing layers at each stage while emphasizing the differences between previously stored patterns and new input patterns. Even in contrast, it is possible to maintain a good pattern separation function by self-assembly for a short period of time.

The present invention will be described in detail below with reference to the drawings.

In general, an information processing device (neural cell model) that imitates neurons that are components of human cerebral tissue, that is, a neural device, performs nonlinear processing on an input analog signal to form a 0 or positive analog output signal. Use analog elements, etc. Attempts have been made to create a multilayer structure by continuously connecting information processing layers in which such neural elements are arranged two-dimensionally in multiple stages, and to create an information processing device with functions similar to the human cerebrum. Such an information processing device is generally called a neural circuit model or neural network.

When connecting neural elements between each information processing layer of an information processing device as described above, it is not necessarily necessary to give each neural element the possibility of connecting with all neural elements in other layers. Without,
The possibility of connection is given only between neural elements existing within a limited range of minute areas, and such minute areas are called the connectable area of that neural element. Even if there is a bond, the strength of the bond is weighted depending on the center portion and the peripheral portion.

When considering the self-organization of neural circuits by changing the strength of the connections between neural elements between the information processing layers according to the pattern of input information as described above, and strengthening the so-called connection strength, it is important to Strengthening means to form a connection when there is no connection between neural elements and the connection strength is 0, and to strengthen the connection if the connection between neural elements has already been formed. It means to strengthen.

On the other hand, as neural elements, there are excitatory elements that have an excitatory effect of increasing input to other neural elements through connections from that neural element, and inhibitory effects that decrease input to other neural elements through connections from that neural element. Two types of suppressive elements are set.
In addition, regarding the setting, for example, the excitatory element may form a positive polarity input signal, and the inhibitory element may form a negative polarity output signal, or both may be set. The neural element also forms a positive output signal, and the neural element that receives these positive output signals is provided with two types of input terminals, and the output is increased by the input signal from one of the input terminals, and the output is increased by the input signal from one of the input terminals. It is also possible that the output is decreased by the input signal from the input terminal of the .

In a neural network with a multilayer structure in which interlayer connections are made as described above, conventionally, only excitatory and inhibitory forward connections are provided between layers, whereas in the device of the present invention, as shown in FIG. As shown, feedback-type inhibitory connections are added to pair with forward excitatory connections. That is, in the principle configuration shown in the figure, three information processing layers U ₂ to _{U 4} are arranged in cascade following the input layer U ₁ into which pattern "A" is input, and the excitability of each information processing layer is Forward excitatory variable connections a and feedback inhibitory variable connections b are sequentially provided between the excitatory nerve elements in the minute regions of the preceding layer corresponding to the nerve elements in sequence.

Therefore, in strengthening the connections between neural elements, for each information processing layer, the neural elements that formed the maximum amount of output or showed the best response within the above-mentioned connection variable region are selected. , that is,
It follows the principle that only the input-side connections of neural elements that have formed an output that best matches the input are strengthened only when the input is present.

When connecting the information processing layers U ₁ to _UL in the neural network according to the above-mentioned principle, three adjacent layers of the information processing layers U ₁ to _UL , i.e. , l-1st information processing layer U _l-1 , l-th information processing layer U _l and l+1th information processing layer U l -1
FIG. 2 shows the manner of coupling between the th information processing layer U _l+1 . In the figure, an excitatory nerve element is shown as a u element, and two types of inhibitory nerve elements are shown as a v element and a w element.

In addition, regarding the arrangement of neural elements in each information processing layer in the neural network, the jth u element placed in the U _l layer is expressed as u ^l _j , and connections are made between this u ^l _j element. Possible bonding area S ^l-1 _j of the previous stage U _l-1 layer
The u element placed within is expressed as u ^l-1 _i , that is, S ^l-1 _j .

Therefore, in the connection mode shown in the second diagram,
Each information processing layer, e.g. each U element in the U _l layer, e.g.
The u ^l _j element is a group of u elements u ^l-1 _i in the previous stage U _l-1 layer,
That is, it receives variable excitatory coupling a ^l _ij from S ^l-1 _j , but as shown in Figure 2, the w element in the U _l layer
w ^l _j also receives a fixed excitatory connection C ^l _ij from the same u element group u ^l-1 _i , that is, S ^l-1 _j , and its output is generated by a variable forward inhibitory connection d ^l _j . The above-mentioned u element u ^l _j
It is reported that.

Also, in addition to each forward link as described above,
A u element in each information processing layer, for example, an u ^l _j element, passes through a v element v ^l _j to a group of u elements in the preceding information processing layer, for example, u ^l-1 _i , that is, S ^l A variable feedback inhibitory connection b ^l _ji toward ^-1 _j is provided in pair with a forward variable excitatory connection a ^l _ij , and the strength of the variable feedback inhibitory connection b ^l _ji is always determined in conjunction with the strength of the variable forward excitatory connections a ^l _ij .

Next, the process of self-organizing the connections between the information processing layers in the neural network configured as described above will be explained.

First, the reactions of each neural element in each information processing layer are listed and shown in the following equations (1) to (8).

Here, ψ{X}=max(x, o) (2) Here v ^l+1 _k (t, τ)=u ^l+1 _k (t, τ) l=0, 1, 2,..., L-1 (8) Therefore, the pattern input to the neural network is
It is presented in the form of sampling every unit time. In other words, in equations (1) to (8) above, the time after presentation of the input pattern is expressed as τ, but for simplicity, the time τ is the time at which the signal is transmitted between adjacent information processing layers. It is assumed that discrete values are taken for each delay time Δ that occurs. Also, the tth input pattern given is represented by {P _j (t)},
After the input pattern {P _j (t)} is presented, the output of the j-th u element u l _j in the l-th information processing layer U _l at the time τ is expressed as ^{u l j} ₍ t, τ)
It is expressed as. Note that the same notation as described above is used for the output display of the suppressive elements v ^l _j and w ^l _j .

In addition, the input to the u element in the input layer U ₀ due to one pattern presentation is given by time as shown in equation (1).
Assume that it becomes 0 after lasting KΔ.
However, even if the output of the u element in the input layer U ₀ becomes 0 at time τ>KΔ, the signal itself has a delay when propagating between layers, so the output of all the u elements in the neural network becomes 0. It becomes 0 at time τ
>(K+L) after Δ has elapsed. Note that the symbol L represents the number of the final layer _UL , and means that the L-th information processing layer _UL is the final layer. Also,
Since the outputs of all u elements must become 0 just before the next pattern {P _j (t+1)} is presented, the time (K+L)Δ is shorter than the pattern presentation interval, It is assumed that the following equation (9) is satisfied.

(K+L)Δ<1 (1) Here, to explain in detail the output of the u element in the input layer U ₀ given by equation (1), if the t-th pattern {P _j (t)} is presented At the point when the input layer
The output u ⁰ _j (t, o) of the j-th u element in U ₀ is the j-th element P _j (t) in the input pattern {P _j (t)}.
receives excitatory input from However, when the delay time Δ has elapsed and the u element u ¹ _k in the next stage information processing layer U ₁ begins to form an output according to equation (3), the output is shown in equation (8). As shown above, it is transmitted to the inhibitory neural element v ¹ _k via a fixed connection with a strength of 1.
As a result, the u element u ⁰ _j in the input layer U ₀ receives an inhibitory input via the feedback inhibitory coupling b ^l _kj via its inhibitory element v ¹ _k . At the same time, the u element u _{0 j} in the input layer U ⁰ is connected to the side via the fixed coupling h ⁰ _nj from _the surrounding suppressive element v ⁰ _n (mεH ⁰ _j ) in the same input layer U 0. Suppression comes into play.

In equation (1), the combinable region S ¹ _j indicates the range in which the suppressive element v that sends a feedback suppressing signal to the u element u ⁰ _j in the input layer U ₀ exists, and , the side suppression region H ⁰ _j indicates the range in which suppressive elements v for applying side suppression between the u elements in the input layer U ₀ exist.

Next, to describe what is meant by the output of the u element in each information processing layer after information processing layer U ₁ given by equations (3) and (4), the lth information processing layer U The jth u element u ^l _j in _l is u^ ^l _j (t,
τ), the feedback inhibitory input from the v element in the next information processing layer, and the side inhibitory input from the surrounding v elements in the same information processing layer _U Form.

Among the various types of connections mentioned above, the first is the information processing layer.
In detail, the forward input received by the u element u ^l _j in U _l is that the inhibitory input from the inhibitory element w ^l _j in the information processing layer U _l is It acts in a divisional manner on the excitatory input that comes from a group of elements U _i (iεS ^l-1 _j ) in U _l-1 , and its output value is given by equation (5). shall be determined. In addition, a ^l _ij (t) and d ^l _j (t) represent the strength of forward excitatory and inhibitory variable connections at the t-th pattern presentation, respectively, and each time a new pattern is presented, each time, according to the principles described below. In addition, the information processing layer
The inhibitory element w ^l _j in U _l receives a fixed coupling from a group of u elements u ^l-1 _{i ,} that is, S l-1 _j , in the preceding information processing layer U ^l-1 , and The output is formed according to the average value of the outputs, and shows the value shown in equation (6).
Here, it is assumed that the strength of the fixed excitatory connection c ^l _ij is adjusted as shown in equation (7). Note that the connectable region S ^l-1 _j indicates a range in which a group of u elements in the preceding information processing layer U _l-1 that are connected to the u element u ^l _j exist. Furthermore, as is clear from equations (5) and (6), the combinable area of the u element u ^l _j in the information processing layer U _l is exactly the same as the combinable area of the inhibitory element w ^l _j . We are doing so.

Next, to explain in detail the feedback type inhibitory input that the u element u ^l _j in the information processing layer U _l receives, the u element u ^l _j
As shown in equation (3), since the u element itself sends out an output, the v element in the next stage information processing layer U _l+1 that sends out an output after the delay time Δ has elapsed.
It receives a feedback inhibitory input from v ^l+1 _k and sends out the output shown in equation (8). In equation (3), b ^l+1 _kj represents the feedback suppressive variable coupling from the v element v ^l+1 _k , and S ^{l+1 j} _{represents the} Next-stage information processing layer U _l+1 that sends out feedback-type suppression signals
It shows the range in which the v elements within exist.

Next, to explain in detail the inhibitory input that the u element u ^l _j in the information processing layer U _l receives, the u element u ^l _j is
At the same time as receiving the feedback-type inhibitory input mentioned above,
Receives side inhibition input from surrounding v elements in the same information processing layer _Ul . Note that v element v ^l _n , that is,
The suppressive fixed coupling h ^l _nj coming from H ^l _j has the effect of applying side suppression between the u elements in the same layer, and H ^l _j is the side suppression area in the information processing layer U _l . It represents.

This concludes the explanation according to equations (1) to (8) regarding the outputs of the u element, v element, and w element in each information processing layer.

Next, we will explain in detail the strength of the inhibitory variable coupling b ^l+1 _kj that acts in a feedback manner on the u element in each information processing layer, for example, the j-th u element u ^l _j in the information processing layer U _l . , the strength of the feedback inhibitory variable coupling b ^l+1 _kj is
It is determined as the strength relative to the total strength of the forward excitatory variable coupling a ^{l+ 1} _k already received by the u element u l+1 _k . In other words, feedback-type inhibitory variable coupling
It is assumed that b ^l+1 _kj is always determined in conjunction with the forward excitable variable connection a ^l+1 _jk as shown in the following equation (10).

In this equation (10), S ^l _k is the range in which there is a group of u elements in the information processing layer U _l that are coupled to the u element u l ⁺ 1 _k , that is, the range in which the u elements u ^l+1 _k can be coupled. Indicates the area.

Next, to explain in detail the principle of strengthening variable connections, all strengthening of variable connections occurs from neural elements within each information processing layer during the period after an input pattern is presented until the next pattern is presented. This is done according to the time integral value of the output. It is assumed that the elapsed time τ after pattern presentation takes a discrete value for each delay time Δ that occurs when a signal is transmitted between adjacent information processing layers, and as described above, the L-th In a neural network in which the information processing layer _UL is the final stage, equation (9) is satisfied. Therefore, after the presentation of the t-th pattern until the presentation of the next pattern, the time integral values of the u element and the w neural element in each information processing layer are respectively expressed as u〓
When expressed by ^l _j (t) and w ^l _j (t), the values are determined in the form shown in the following equations (11) and (12).

u〓 ^l _j (t)=∫ ^(K+L) 〓 _O u ^l _j (t, τ)δτ When l=0, 1, 2,..., L (11) w〓 ^l _j (t)=∫ ^(K+L) 〓 _O w ^l _j (t, τ) δτ When l=0, 1, 2, ..., L (12) Next, the j-th u in the l-th information processing layer U _l The forward input coupling of an element u ^l _j is strengthened only if there is no neural element in the vicinity of the u element u ^l _j that has a larger time integral value than the u element itself. In other words, let δ ^l _j (t) be a variable that takes the value 1 when it is reinforced by the t-th pattern presentation, and takes the value 0 when it is not reinforced.
Then, the value of the variable δ ^l _j (t) is as follows (13)
Determined by the formula.

δ ^l _j (t) 1j′εΩ ^l ₀ when u〓 ^l _j (t)≧u〓 ^l _j ′(t), then 0. Others When l=1, 2,…, L (13) In this equation (13), Ω ^l _j represents the neighborhood region of the u element ^ul _j .

Using the variable δ ^l _j (t) as described above, the forward variable excitatory connection a ^l _ij and the forward variable inhibitory connection d ^l _j
The respective reinforcement amounts for and are given by the following equations (14) and (1
It can be expressed as in equation (5), where q ^l in equation (14) is a parameter that determines the degree of reinforcement for each pattern presentation.

Δa ^l _ij (t)≡a ^l _ij (t+1)−a ^l _ij (t) =q ^l・c ^l
_ij・
u ^l-1 _i (t)・δ ^l _j (t) When l=1, 2,..., L (14) Therefore, the feedback type variable inhibitory connection b ^l+1 _kj is also strengthened, but as mentioned above, it is always in conjunction with the forward direction variable excitatory connection a ^l+1 _jk according to equation (10). It changes over time.

As described above, in the neural network according to the present invention, the forward variable connections are the neural elements in each information processing layer during the period after one pattern is presented until the next pattern is presented. is strengthened depending on the time integral value of the output. Therefore, a new property not found in the prior art is maintained in that the forward coupling from the u element whose output is not suppressed via the feedback suppressive coupling is further strengthened.

In other words, according to the principle of strengthening connections in neural networks according to the present invention, one necessary condition is that the u elements in the input side information processing layer continue to send out outputs, as shown in equation (14). I am doing it. Therefore, the output cannot be suppressed by feedback type suppression,
The forward coupling from the u-element that continues to send out a large output will be strengthened. On the other hand, feedback type suppression is (10)
As is clear from the equation, it has a large effect on the u element, which is already sending out excitatory connections in the forward direction that are sufficiently strengthened compared to the surrounding connections.
In other words, in the part where the circuit network that extracts the features of the given pattern is completed, the feedback type suppression acts greatly and suppresses the output of the u element in the input side information processing layer, so the coupling is not strengthened any further. It will not proceed.

Furthermore, the excitatory and inhibitory forward variable connections a ^l _ij
It is assumed that both d l _j and d ^l j take small positive values as shown in the following equations (16) and (17), respectively, even in the initial state. Also, α ^l in the formula is
It is a positive parameter that determines the size of the initial value.

a ^l _ij (0)=α ^l・c ^l _ij When l=1, 2,..., L (16) d ^l _j (0)=αl/2 When l=1, 2,..., L (17 ) In order to specifically illustrate the mode of operation of the neural network according to the present invention explained above, the input information receiving layer
Examples of computer simulation results performed on a four-stage neural network consisting of U ₀ and information processing layers U ₁ to _{U 3} are shown in Figures 3 and 4 a and b.
and shown in FIGS. 5a and 5b, respectively.

FIG. 3 shows four types of patterns "X", "Y", "T", and "Z", which were used in the above-mentioned computer simulation and have many mutually overlapping parts. In this computer simulation, an information signal consisting of such a test pattern is transmitted to the receiving layer U ₀
We compared and examined the process in which the connections between each information processing layer gradually change and self-organization is gradually completed in response to repeated presentations.

Taking as an example the case where pattern "Z" is presented among the four types of test patterns shown in FIG. 4a and 4b show a comparison with the apparatus of the present invention using the present invention. In addition, in Figure 4 a and b, in order to make it easier to understand the progress of self-organization, the information signal flows from the inner layer to the previous layer, contrary to the normal flow of the information signal. Assuming that the final information processing layer
In U ₃ , the range that ultimately affects the output of the self-organized U element that selectively responds to the pattern “Z” is reproduced on the receptor layer U ₀ to illustrate its progress. There is. Note that the numbers appended to each row in the diagram represent the number of times the pattern is presented. As is clear from the results shown,
The number of pattern presentations required for the self-assembly to be completed and the presentation pattern "Z" to be distinguishable was more than 10 times in the conventional device shown in figure a, whereas in figure b In the device of the present invention shown in Figure 2, 6 to 7 presentations were sufficient.

Next, the four types of test patterns “X”, “Y”, “T”, and “Z” shown in the third figure are presented 10 times each to allow self-organization of the connections between the information processing layers, and then the final The results of examining which of the four test patterns each U element in the information processing layer _U3 responds to are compared between the device of the present invention and the conventional device, as shown in Figure 5a. Shown in b. In the illustrated example, the symbol of the pattern to which each u-element arranged in a matrix responds is shown within the frame of each u-element. As is clear from the results shown in the figure, the number of U elements that respond to two or more types of patterns was 29 in the conventional device shown in FIG. This result confirms that in the device of the present invention, much better pattern selectivity than the conventional one can be obtained through self-organization of the connections between the information processing layers.

As is clear from the above description, according to the present invention, a self-organizing storage device used for pattern recognition can perform information processing that mimics the human memory process of matching past memories with new input patterns. Therefore, by adding feedback type connections, it is possible to achieve a much higher level of pattern recognition than in conventional neural networks using only forward connections.

Therefore, the self-organizing storage device of the present invention can be applied to a wide range of technical fields such as pattern recognition devices and information retrieval devices that can perform highly sophisticated pattern recognition of graphics, characters, sounds, etc.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the principle configuration of the self-organizing storage device of the present invention, FIG. 2 is a diagram showing an example of the detailed configuration of the connection between the information processing layers, and FIG. Figures 4a and 4b are diagrams showing examples of test patterns subjected to computer simulation; Figures 4a and 4b are photographs of oscilloscope figures showing examples of the results of the computer simulation in comparison with those obtained by conventional equipment; and Figure 5. Figures a and b are diagrams respectively showing other examples of the results of the computer simulation in comparison with other examples using the conventional device. U...information processing layer, u...excitatory element, v, w...inhibitory element, S...combinable region, H...side inhibitory region,
a...Variable excitatory connection, b...Variable feedback inhibitory connection, c...Fixed excitatory connection, d...Variable forward inhibitory connection, h...Fixed inhibitory connection.

Claims (1)

[Claims] 1 A plurality of information processing layers in which a plurality of excitatory and inhibitory nonlinear threshold elements are mixed and arranged in two dimensions are sequentially arranged, and forward excitatory connections and inhibition are established between the information processing layers. 1. A self-organizing storage device, characterized in that the connections between the sequential information processing layers are self-organized in correspondence with input information by providing symmetrical connections and feedback-type inhibitory connections in the opposite direction.