JP3310009B2 - Neural network system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network system and, more particularly, to a neural network system which adjusts the contribution of a plurality of sub-neural networks by feeding back the output of the neural network. .

[0002]

2. Description of the Related Art A neural network is an arithmetic device simulating a neural network of a human brain, and is currently used in various fields such as recognition and control. However, a single neural network constructed by a learning algorithm called back propagation, which is widely used at present, has some practical problems.

One of the problems is that when a neural network is used for recognition processing or the like, its performance is insufficient compared with human recognition. In order to solve this problem, structuring of a neural network and introduction of a feedback mechanism have been performed. Although this method is effective for improving recognition performance,
When creating a feedback mechanism, a new problem arises in that the method of feedback must be strictly described by an expert.

[0004] This problem will be described below using a neural network for recognizing characters as an example. A neural network is composed of units (artificial neurons) corresponding to nerve cells and links corresponding to synapses connecting them. For character recognition, for example, as shown in FIG.
A three-layer neural network that forms an intermediate layer and an output layer is most often used. In this neural network, when a pattern of a certain character (3 in FIG. 10) is input to the input layer,
The output unit corresponding to the type of the character recognizes the character by outputting a large output. For example, if only the third output unit outputs a large output, the input character pattern is determined to be “3”. If only the eighth output unit outputs a large output, the input character pattern is “3”. 8 ".

However, when a character pattern similar to both "3" and "8" is input as shown in FIG. 11, a third output unit and an eighth output The units will both be in some state of output and the character type cannot be determined. In this regard, if it is a human, "3" and "8" are unconsciously
Focusing on the character center portion important for distinguishing between characters, the character pattern whose character pattern is “3” can be classified.

The structuring and feedback mechanism of the neural network simulate the above-mentioned automatic change of the human viewpoint. This means, for example, that a character pattern is divided into several parts as shown in FIG. 12 and those partial patterns are separated into separate sub-neural networks (NNi; i = 1,..., N).
And then combine the outputs to create an output. In this case, the weight coefficient of the link of the output of each sub-neural network to the integration unit indicates the importance of the information of that part, and changing this corresponds to which part of the character pattern to watch. The feedback mechanism changes the weight coefficients of these links from the output pattern. In this example, the output unit "3"
When the same output is simultaneously output to and "8", the weight coefficient of the link from the neural network corresponding to the central part of the character to the integrated unit is increased. Then, the information occupied by the information of the character center portion among the information entering the integrated portion increases, and is determined to be “3”.

Although such a method is an effective method corresponding to the human recognition process, the feedback method must be designed by a human, as can be seen from this example. A similar problem to this character recognition example is that other applications,
For example, it occurs in diagnosis and monitoring, and the design of a feedback mechanism for solving the problem requires more specialized knowledge than character recognition.

[0008]

As described above,
A conventional neural network using a feedback mechanism has a problem that, when constructing a feedback mechanism, it is necessary to have specialized knowledge about a learning target.
SUMMARY OF THE INVENTION An object of the present invention is to provide a neural network system in which a feedback mechanism can be easily designed.

[0009]

To achieve the above object, a neural network system according to one aspect of the present invention provides a first neural network and a first neural network by feeding back an output signal of the first neural network. The structure in the neural network of
For example, a feedback mechanism is provided for adjusting (correcting) the contribution rate of the sub-neural network constituting the first neural network.

[0010] The feedback mechanism includes a second neural network such that when a plurality of case patterns are input to the first neural network, the output of the first neural network substantially matches any of the teaching outputs. Then, the relationship of the structure that substantially matches the original output of the neural network and the teaching output is stored in the second neural network.

When the discriminated case pattern is input to the system, the second neural network operates according to the output signal of the first neural network and the learning content.
Adjust the structure of the neural network. The output signal of the first neural network after the adjustment of the structure is the determined output signal.

[0012]

The feedback mechanism searches for the structure of the first neural network such that when the case pattern is input to the first neural network, the output of the first neural network substantially matches one of the teaching outputs. I do. The second neural network is configured such that, for a plurality of case patterns, the first output of the first neural network (the output before modifying the structure) and the first output whose output substantially matches any of the teaching outputs Learn the structure of this neural network as input and output.

When the discriminated pattern is input to the first neural network, the first neural network outputs a signal once in a state where the feedback mechanism is not operated. Thereafter, the feedback mechanism adjusts the structure of the first neural network according to the output signal of the first neural network. By adjusting the structure, the output of the first neural network also changes.
The output signal of the first neural network after the structure adjustment is the output signal of the system.

Since the second neural network has a kind of interpolation function for complementing between learned patterns, even if an unlearned discriminated case pattern is input, the first neural network has the first function.
Can appropriately modify the structure of the neural network, and can obtain an appropriate output as a system.

Therefore, the neural network system of the present invention provides a representative pattern and a corresponding teaching output,
By simply preparing a plurality of case patterns and a plurality of case patterns, a feedback mechanism can be constructed without using specialized knowledge on a learning target.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a neural network system according to an embodiment of the present invention will be described with reference to the drawings.

First, an outline of the neural network system according to the present embodiment will be described. The learning target of the neural network system of the present embodiment is the structured neural network shown in FIG. 3, and the neural network of the present embodiment recognizes a voltage pattern in a power system and evaluates the pattern. In this example, nodes in a certain area are grouped into one group, and the input pattern is composed of partial vectors characterized by a total voltage intensity for each area. FIG. 5 shows a power system model, and FIG. 6 shows a part of a representative voltage pattern.

The evaluation layer of the neural network shown in FIG. 3 corresponds to twelve evaluation items. For example, the output unit 1 evaluates the “daytime pattern”, the output unit 5 evaluates the “voltage drop”, and the output unit 10 evaluates the “area 5”. In this neural network, when a certain voltage pattern is input to the input layer as a case pattern, an output unit corresponding to an evaluation item valid for the pattern outputs a large output. If the input pattern is close to any of the eight representative patterns in FIGS. 6A to 6F, the input pattern is regarded as being similar to the representative pattern, and the same evaluation output as the representative pattern is output to the output unit. Given by

However, as in the case of a character pattern that is difficult to distinguish in character recognition, a voltage pattern considerably different from a representative pattern learned by a neural network may be input. In such a case, a feedback mechanism must be used as in the case of character recognition. This embodiment makes it possible to automatically construct this feedback mechanism without requiring much specialized knowledge.

Next, the details of the structured neural network used in this embodiment will be described with reference to FIG. As is generally known, the structured neural network shown in FIG. 3 includes an input layer, a feature layer, an association layer, an evaluation layer serving as an output layer, and a feedback mechanism. Assume that the construction has already been completed by the method.

The voltage pattern is input to the input layer after being divided into groups of partial voltage patterns of areas 1 to 5 in the power system of FIG. Each unit in the input layer outputs the input signal as it is. The feature layer is also for extracting the features of the partial vectors constituting the voltage pattern vector,
Each unit of the characteristic layer receives the output of each unit of the input layer, calculates and outputs a reaction strength according to the reaction characteristics and the connection weight. The association layer is for combining the outputs of the characteristic layers. Each unit of the association layer receives the output of each unit of the characteristic layer, selects and outputs the output having the maximum intensity.

The output of the association layer is supplied to the evaluation layer. The evaluation layer provides the output of the neural network in response to the output of the association layer. The evaluation layer includes an AND unit. As shown in FIG. 4, the AND unit adds a plurality of inputs (from the association layer) with predetermined weights W1 to Wn, converts the added value by a predetermined sigmoid function, and outputs the result. The feedback mechanism corrects the load value. By modifying the weight value, the contribution rate of the sub-neural network is modified. In the present embodiment, the input layer to the association layer form a set of sub-neural nets (corresponding to NN1 to NNn in FIG. 12) corresponding to each group (areas 1 to 5), and the AND-type unit is an integration unit. (Corresponding to the integration unit in FIG. 12)
It has become. The weight coefficients W1 to W5 (FIG. 3) of the link from the association layer to the AND type unit indicate the relative importance of the information of each group. The feedback mechanism receives the output of the evaluation layer and outputs a relative contribution rate f _n . The relative contribution ratio f _n is defined by the following equation. _{^{w n = f n / Ng (}} 2) where, w ⁿ is the weight coefficient of the link to the AND type unit from the feature layer of group n, Ng is the number of groups.

This network can control the interpolation function of the neural network by changing the link weight coefficient (relative contribution rate) by feedback. Next, the configuration of the entire system according to the present embodiment will be described.

As shown in FIG. 1, the system according to the present invention has a structured neural network unit 7 for realizing the input / output characteristics of the structured neural network shown in FIG. 3, an input pattern (representative pattern and case pattern) and a representative pattern. A pattern database 8 for storing a teaching pattern (output) paired with the above, a data selector 10 for selecting input data, an evaluator 9 for evaluating a load value determined by a feedback mechanism based on an output value at that time, and an output device. 11, an output unit 12, an input device 13, and an input unit 14.

The representative pattern is a pattern obtained by learning the relationship between them along with a teaching pattern attached to the neural network, and the case pattern is an arbitrary pattern that is slightly different from the representative pattern.

The structured neural network unit 7 includes a state table 1 indicating the output states of the units, a load value table 6 for storing weight coefficients of links between the units, a unit state calculation unit 4 for determining the states of the units, It comprises a load value calculating section 3 for updating the load value, and a feedback mechanism 5 for adjusting the load value.

Further, as shown in FIG. 2, the feedback mechanism 5 has a parameter adjuster 15 for sequentially changing and generating a combination {f _n } of relative contribution rates, which will be described later, and an output of the structured neural network unit 7. storing a parameter storage unit 16 for storing a combination of relative contribution {f _n} when substantially coincide, the combination of the output and the relative contribution of the structured neural network unit 7 {f _n} to a predetermined teaching pattern The neural network unit 17 performs the following. Next, the operation of the neural network system according to the present embodiment will be described. First, the relationship between the components will be described according to the process of calculating the output of the neural network unit 7 for a certain input pattern.

As for the input pattern to the neural network unit 7, one of the input patterns in the pattern database 8 is selected by the data selector 10, or is input from the input device 13. This input pattern is input unit 1
4 and input to the neural network unit 7. In addition,
The neural network unit 7 is a known device that simulates the characteristics of the structured neural network of FIG.

Outputs of all units are written in the unit status table 1 of the neural network unit 7. In this embodiment, the input pattern is written as it is to the input layer of the table as the output of the input layer unit.
The weight value of the link between the input layer unit and the characteristic layer unit is registered in the load value table 6. Using the weight coefficient group registered in the load value table 6 and the output group of the input layer unit registered in the unit state table 1, the calculation unit 4 calculates the output of the characteristic layer unit, and calculates the characteristic layer unit of the unit state table 1. Write to.

The weight value of the link between the characteristic layer unit and the association layer unit is also registered in the load value table 6,
The operation unit 4 calculates the output of the association layer unit in the same manner,
Write to the united layer section of unit status table 1. Since then
Similarly, the output of the evaluation layer unit is finally written in the evaluation layer section of the unit state table 1. This value is
The output of the neural network unit 7 is output from the output unit 12.

Next, a method of constructing the feedback mechanism 5 in the present embodiment will be described. First, a certain case pattern is provided from the pattern database 8 to the neural network unit 7 through the data selector 10 and the input unit 14. The neural network unit 7 outputs the output unit 12 according to the aforementioned procedure.
Is given the output value of the evaluation layer unit. At first, the feedback mechanism 5 does not work. That is, at this time, the relative contribution rates f _n are all equal to 1.0. If the given case pattern is far from the representative pattern, the output value of the evaluation layer unit for this pattern will be an ambiguous output pattern different from the output pattern (teaching pattern) for the representative pattern. Therefore, in the present embodiment, a combination {f _n } of relative contribution ratios that makes this output pattern closer to one of the teaching patterns is automatically searched.

The search for the appropriate relative contribution rate {f _n } is performed, for example, as follows. First, the parameter adjuster 15 in the feedback mechanism 5 generates {f _n } candidates. In the present embodiment, the parameter adjuster 15 sets f ₁ , f ₂ , f ₃ , and f ₄ of the relative contribution rates f _n (n = 1,..., 5) to (3)
Varied by the formula, f ₅ is set to satisfy the above equation (2). _{f n = f n + 0.2 *} R (3) (n = 1, ..., 4)

R in equation (3) is a number that takes -1 and 0 and 1 with equal probability, and is formed by a pseudo-random number. The parameter adjuster 15 changes {f _n } one after another according to this equation, and the feedback unit 5 changes the load value table 6 accordingly, and accordingly, the output pattern given by the structured neural network unit 7 to the output unit 12 also changes. I do. The evaluator 9 stores the successively changing output patterns in the pattern database 8.
Is compared with all the teaching patterns. In the present embodiment, this comparison is performed by considering the output pattern and the teaching pattern as vectors and calculating the vector distance.
If the output pattern created by a certain {f _n } happens to substantially coincide with a certain teaching pattern (that is, if the distance between both vectors is equal to or less than a predetermined reference value),
This {f _n } can be regarded as good feedback for the output pattern of the neural network unit 5 before the feedback is introduced. Parameter storage unit 16 stores a combination of f _n (output in a state where the feedback mechanism 5 is not working) output of the neural network at this time.

FIG. 7 shows how the distance between the teaching patterns 1 to 3 and the output pattern changes when the relative contribution ratio f _n is changed by the above-described method when certain case data is input. In this case, when the number of updates of the relative contribution rate is 139, the distance between the output pattern and the pattern 2 is “0”, and the output pattern and the pattern 2 match. Thus, the parameter storage unit 16 stores the parameter storage unit 16 outputs of the neural network (the position feedback mechanism 5 is not working) a combination of f _n at this time.

After the correct feedback has been obtained for all the case data by the above means, an output without feedback and an appropriate relative contribution corresponding thereto are given to the neural network 17 of the feedback mechanism 5 as an input / output pair. To learn. As a learning method, for example, back propagation can be considered. After learning, the neural network 17 of the feedback mechanism outputs an appropriate relative contribution rate to the parameter adjuster 15 in accordance with the output 12 of the neural network of the main body (a state in which the feedback is not working). The relative contribution is adjusted. By adjusting the relative contribution, the output 12 changes to an appropriate content, and this is the finalized (or final) output of the neural network unit 7, that is, the output of the system. The operation of the above system is summarized as follows (assuming that the learning of the neural network unit 7 excluding the feedback mechanism has been completed).

Step 1: Turn off the feedback mechanism 5,
The case pattern is input to the neural network unit 7 and the output of the neural network unit 7 is obtained. Next, the feedback mechanism 5 is turned on to sequentially change the set of contribution rates. The output of the neural network unit 7 also changes according to the change in the set of contribution rates. A set of contribution rates when the output of the neural network unit 7 substantially matches any of the teaching output patterns is stored together with the original output of the neural network unit 7. Repeat step 1 for all case patterns.

Step 2: The neural network unit 17 of the feedback mechanism 5 learns the relationship between the original output of the neural network unit 7 obtained from the case pattern and the contribution ratio as an input / output relationship. Step 3: Supply the processing target signal (pattern) to the neural network unit 7 with the feedback mechanism 5 turned off. Step 4: Supply the output of the neural network unit 7 to the neural network unit 17. step 5: The contribution rate in the neural network unit 7 is corrected according to the output of the neural network unit 17.

Step 6: With the contribution ratio corrected, the neural network unit 7 processes the signal to be processed again, and the output signal is used as the (final) output signal of the system. (Experimental result)

A network having the above-described feedback mechanism was actually constructed, and an experiment was conducted. As an experimental example, an example of inputting the pattern of FIG. 8 is shown. The pattern of FIG. 8 is a mixture of the representative patterns of FIGS. 6C and 6F at 4: 6.

In response to this input, the neural network unit 7 determines that the relative contribution ratio is uniform (f _n = 1.0,
n = 1, ..., 5; without feedback)
Although the output signal of (a) was output, the feedback mechanism obtained by the above procedure works, and the relative contribution rate is f ₁ to
By adjusting f ₄ = 0.8 and f ₅ = 1.8, the final output shown in FIG. 9B was obtained.

With respect to the input pattern of FIG. 8, the network has determined that there is no tendency of voltage drop as shown in FIG. 9 (a). However, when the feedback is applied, the degree of attention to the feature of the input pattern changes due to the change in the relative contribution rate, and the network evaluates that a voltage drop is observed in the area 5 as shown in FIG. 9B. This is similar to an expert focusing on the subtle voltage drop in area 5, confirming that the feedback mechanism is working effectively.

As described above, according to this embodiment,
Since the feedback mechanism is equipped with a neural network and a means for automatically searching for an appropriate contribution rate, by preparing a representative pattern and its accompanying teaching pattern and some case patterns, the designer can obtain specialized knowledge on the learning target. An accurate feedback mechanism can be easily constructed even if the user does not have a basic knowledge.

After the learning of the neural network unit 17 of the feedback mechanism is completed, the parameter adjusting unit 15 and the parameter storage unit 16 are provided as long as the function of rewriting the load value table 6 in accordance with the output of the neural network unit 17 can be ensured. May be removed.

Further, the present invention is not limited to the above embodiment. For example, another circuit such as a converter for sharpening the output may be provided at the output stage of the neural network unit 7. Further, for example, in the above-described embodiment, an example in which the present invention is applied to a system for evaluating a voltage pattern in a power system has been described. However, the present invention is not limited to this, and can be applied to other wide fields such as recognition and classification. .

In the above embodiment, the present invention is applied to a structured neural network system integrating outputs of a plurality of sub-neural networks. However, it is obvious that the present invention is applied to a system consisting of a single neural network. It is possible. In this case, for example, the link weight of any one or a plurality of layers may be changed.

Further, in the above embodiment, the contribution rate when integrating the outputs of a plurality of sub-neural networks is corrected by the feedback mechanism. However, the weight coefficient of the link between the units in the sub-neural network and the tendency of the unit to fire are improved. You may make it change other structures and structures, such as (reaction characteristic). Also, the reaction characteristics of the unit are not limited to the sigmoid function type, but may be other known characteristics, for example, a bell-shaped response characteristic.

[0047]

As described above, according to the present invention,
By providing a function for automatically constructing a feedback mechanism in a neural network system having a feedback mechanism, an accurate feedback mechanism can be easily constructed even when a designer does not have specialized knowledge on a learning target.

[Brief description of the drawings]

FIG. 1 is a block diagram of a neural network system according to an embodiment of the present invention.

FIG. 2 is a block diagram of a feedback mechanism of the neural network system of FIG. 1;

FIG. 3 is a view for explaining a structured neural network constituting the neural network shown in FIG. 1;

FIG. 4 is a diagram illustrating units of an evaluation layer of the structured neural network, and is a diagram illustrating a relative contribution ratio.

FIG. 5 is a diagram illustrating a power system which is an application example of the present invention.

FIG. 6 is a diagram showing a representative pattern when applied to the state evaluation of the power system of FIG. 5;

FIG. 7 is a view for explaining parameter search in the feedback mechanism.

FIG. 8 is a diagram showing an input example in power system state evaluation.

9A and 9B are diagrams illustrating an output example in power system state evaluation, where FIG. 9A is a diagram illustrating an output when feedback does not work, and FIG. 9B is a diagram illustrating an output when feedback works.

FIG. 10 is a diagram illustrating a three-layer neural network.

FIG. 11 illustrates a similar character pattern.

FIG. 12 is a diagram for explaining the effects of structuring a neural network and feedback.

Continuation of the front page (56) References JP-A-3-250243 (JP, A) JP-A-62-106590 (JP, A) JP-A-4-84282 (JP, A) Kazuo Nishimura, "Structured neural network Power System State Evaluation ", Proceedings of the National Meeting of the Institute of Electrical Engineers of Japan, 1990, December 31, 1990, Vol. 9-140 to 9-141, (JST Document No .: S0653A) Kazuo Nishimura, "Trends in Neural Network Technology", Toshiba Review, Japan, Toshiba Corporation, December 1, 1991, Vol. 12, pp. 924-930, (Patent Office CSDB Document Number: CSNS 199900218001) Kazuo Nishimura, "Application of Neural Network to Power System Engineering", Toshiba Review, Japan, Toshiba Corporation, Published, September 1, 1991, Vol. 46, No. 9, pp. 747-750, (JPS CSDB Literature Number: CSNS199900226006) Yoshihiro Kojima et al., “Multi-Font Kanji Recognition Using Structured Neural Network (NARA)”, IEICE Technical Report, Japan, Electronic Information Published by The Institute of Telecommunications, September 20, 1991, Vol. 91, No. 231 (NC91-30 to 47), pp. 9-16, (Patent Office CSDB literature number: CNT199900038002) Yutaka Yamaguchi, et al., "Adaptive Control Using Fuzzy Associative Memory System", IEEJ Transactions C, Japan, The Institute of Electrical Engineers of Japan, 1991. March 20, Vol. 111, No. 1 pp. 40-46, (Patent Office CSDB literature number: CCNT199900320003) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06N 1/00-7/08 G06G 7/60 G10L 15/16 G06K 9/00 -9/03 G06K 9/46-9/52 G06K 9/62-9/82 G06T 7/00 G05B 13/02 CSDB (Japan Patent Office) JST file (JOIS)

Claims (1)

(57) [Claims] 1. A first neural network including a plurality of sub-neural networks and an integrating unit that integrates outputs of the plurality of sub-neural networks to obtain a final output, and according to an output of the first neural network, A neural network system having a feedback mechanism for adjusting a relative contribution ratio of output signals of the plurality of sub-neural networks, wherein the feedback mechanism comprises a representative input pattern to the first neural network and a teaching output corresponding thereto. Means for storing a pattern and a case pattern, and for each of the plurality of case patterns, an output when the case pattern is input to the first neural network substantially matches any of the teaching output patterns Means for determining the contribution rate at the time, A second neural network that learns the relationship between the output of the first neural network and the contribution ratio obtained from the neural network as an input / output relationship, and the contribution ratio according to the contribution ratio output by the second neural network. Means for automatically correcting the contribution rate according to the output signal of the first neural network, wherein the output signal of the first neural network after the contribution rate correction is the output signal of the system. A neural network system characterized by: