JP7036689B2 - Computer system - Google Patents

Description

The present disclosure relates to a computer system.

In order to efficiently design and operate social infrastructure and cities, it processes data in the real world and cyber space, analyzes or predicts the state of social infrastructure, etc., or controls or guides social infrastructure, etc. Technology is attracting attention. The data to be processed is composed of environmental sensing data such as temperature and humidity, log data of machines such as automobiles, and log data of humans or organizations such as mail and SNS.

In recent years, machine learning using neural networks has been attracting attention as a method for data analysis. Neural networks can easily realize high-speed and accurate data analysis. Various types of data, including system data as described above, are graph data that can be represented by a graph structure.

For example, International Publication No. 2016/174725 (Patent Document 1) discloses a technique capable of constructing a neural network of graph data. Specifically, Patent Document 1 is a computer that executes arithmetic processing using a neural network composed of a plurality of layers including one or more neurons, and connects a plurality of nodes and a plurality of nodes. A storage unit that stores graph data composed of one or more edges and sample data that stores one or more values input to the neural network, and a construction unit that builds a neural network using graph data. , The construction unit generates one or more neurons in each of the plurality of layers based on the plurality of nodes contained in the graph data, and based on the one or more edges contained in the graph data, Disclosed (eg, summary) that a neural network is constructed by creating connections between one or more neurons contained in each of multiple layers. In addition, Non-Patent Document 1 discloses an example of a relational graph convolutional network.

International Publication No. 2016/174725

M. Schlichtkrull et al., “Modeling Relational Data with Graph Convolutional Networks”, arXiv preprint arXiv: 1703.06103, 2017.

Among various types of neural networks known, a convolutional neural network (CNN) is known as a neural network particularly useful for image processing. The graph data has a relationship, and it is useful if the graph data can be analyzed by CNN.

In order to realize so-called deep learning in which a CNN is composed of a deep network, pooling or, more generally, an operation of reducing the degree of freedom of parameters toward the upper layer is required. If a large number of convolution layers are stacked without a pooling layer, only the number of parameters increases without reducing the degree of freedom that can be expressed (regularization is not applied), which causes overfitting when learning the parameters.

In the conventional grid data CNN (CNN on the grid), it is common to reduce the image size by an operation called pooling after the convolution operation (and application of the nonlinear activation function). However, it is difficult to realize the pooling operation in the CNN of the graph data.

If the pooling operation in the grid data is simply made to correspond to the graph data, it is considered that the node of the graph data should be appropriately deleted to reduce the size of the graph data. However, the guideline on how to select the node to be deleted and the node to be retained is not clear.

Therefore, a technique for realizing pooling that reduces the degree of freedom of parameters in the CNN on the graph is desired.

One aspect of the present disclosure is a computer system that performs a convolutional neural network on a graph, comprising one or more processors and one or more storage devices, wherein the convolutional neural network on the graph is one or more. Convolutional layer and one or more pooling layers, the one or more storage devices store the kernel weight data of the one or more convolutional layers, and the one or more processors in each convolutional layer. The value of each node is updated by a convolutional operation based on a kernel having a size of a predetermined number of hops, and in each pooling layer, the value of each node is set to the value of each node and within the pooling range of a predetermined number of hops from each node. It is updated by the pooling process based on the value of the node, and the size of the kernel of the convolutional layer behind the pooling layer is larger than the size of the kernel of the convolutional layer of the previous stage.

According to one aspect of the present disclosure, the pooling process can be appropriately performed in the convolution operation on the graph.

An example of the configuration of a computer system is shown. An example of a neural network is shown. An example of a convolutional neural network is shown. An example of the configuration of CNN on the graph is shown. An example of graph data is shown. Indicates one node and a node connected to the node. It is a part of the weight data and shows the value related to the convolution of the node. The update of the state value of the node is shown schematically. The details of the first convolution layer, the first pooling layer, the second convolution layer and the second pooling layer of CNN on the graph are shown. An example of graph data input to CNN on the graph is shown. An example of the convolution operation of the first convolution layer is shown. An example of the attributes of graph data is shown. Some data of the weight data of this example are shown. The update of the edge state value is shown by the first convolution layer. The pooling process by the first pooling layer is schematically shown. An example of an applicable Laplacian row example is shown in the graph data having the graph structure shown in FIG. The convolution operation for the node by the second convolution layer is schematically shown. It is a flowchart explaining the learning process which a computer executes. An example of a computer system that executes a distributed CNN is shown. Examples of dispersed CNN layers and columns are shown. An example of a conceptual configuration of a distributed CNN is shown. The propagation rule of the application node of the function f is shown. Shows the propagation rule of countable nodes. The propagation rule of the multiplication node is shown. The propagation rule of the variable replication node (branch node) is shown. A configuration example of the dispersed CNN and its processing are shown.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the present embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are given to the common configurations in each figure.

[System configuration]
FIG. 1 shows an example of the configuration of a computer system. As shown in FIG. 1, the computer system 100 is composed of a plurality of computers 101 and a storage system 102, and the plurality of computers 101 and the storage system 102 are connected via a network 103 so as to be able to communicate with each other. .. The computer system 100 of this embodiment includes three computers 101. In FIG. 1, only one computer is designated by reference numeral 100 as an example. The number of computers and storage systems is arbitrary. The computer system 100 may be composed of one computer 101.

As the network 103, a WAN (Wide Area Network), a LAN (Local Area Network), a SAN (Storage Area Network), and the like can be considered. The type of network 103 is not limited. The network connecting each computer 101 and the storage system 102 and the network connecting each computer 101 may be different networks.

The computer 101 includes a processor 110, a memory 111, a communication interface 112, and an input / output interface 118, and each configuration is connected to each other via a bus 114. Further, the input / output device 119 is connected to another device via the input / output interface 118. The processor 110 includes one or more CPUs 115 that perform arithmetic processing. In FIG. 1, only one CPU is designated by reference numeral 115. The CPU 115 realizes the function of the computer 101 by executing the program stored in the memory 111. Further, the process executed on the computer 101 is executed by one or more CPUs 115. It should be noted that one CPU 115 may execute a plurality of processes. The CPU 115 may be an arithmetic unit such as FPGA or GPU.

The memory 111 stores a program executed by the processor 110 and information used by the program. Further, the memory 111 includes a memory space allocated for one process executed by the processor 110. The memory space may be secured on the memory areas of a plurality of memories 111, or may be secured on the memory area of one memory 111. Further, the memory 111 may include a memory space for a plurality of processes. The program and information stored in the memory 111 will be described later.

The communication interface 112 communicates with an external device via the network 103. The processor 110 accesses other devices via the communication interface 112. The input / output interface 118 mediates communication between another device and the input / output device 119 via the network 103.

The storage system 102 includes a disk interface 133 and a plurality of storage devices 117, and each device is connected to each other via a bus 135. The disk interface 133 is an interface for connecting to a plurality of storage drives 134. The storage drive 134 is a storage device for storing various data, for example, an HDD or an SSD. In FIG. 1, only one storage drive is designated by reference numeral 134.

The memory 111 stores a program that realizes the data processing unit 120. The processor 110 that executes the data processing unit 120 executes a learning process by inverse error propagation and an analysis process by forward propagation. In the training process, the processor 110 uses the training data 122 to determine the weights of the edges in the constructed neural network. In the analysis process, the processor 110 inputs the data to be analyzed into the constructed neural network to perform predetermined analysis such as classification and regression.

The data processing unit 120 may be composed of a plurality of program modules. For example, the data processing unit 120 may include a construction module for constructing a neural network, a learning processing module for executing learning processing, and an analysis processing module for executing analysis processing. Further, each program module may be executed by a separate computer 101.

[neural network]
FIG. 2 shows an example of a neural network. The neural network 200 shown in FIG. 2 is composed of three layers, an input layer 201, an intermediate layer 202, and an output layer 203. Each layer is composed of one or more neurons 211. Also, the neuron 211 in each layer connects to at least one neuron 211 in the other layer. Specifically, the neuron 211 of the input layer 201 is connected to at least one neuron 211 of the intermediate layer 202, and the neuron 211 of the intermediate layer 202 is connected to at least one neuron 211 of the output layer 203. The side of the input layer 201 is the front stage side, and the side of the output layer 203 is the rear stage side.

Edge 212 represents the output of data between neurons 211. In the analysis process, the neural network 200 forward propagates the input data from the input layer 201 to the output layer 203 via the intermediate layer 202. The value input to each neuron 211 is multiplied by the weight assigned to the edge and output to neurons 211 in other layers connected by the edge 212. The input to input 211 may include a bias. The weight represents the strength of the connection between the neurons 211 and is determined by the learning process described later. In the learning process, the weight of each layer is updated by the error back propagation of the loss signal.

Returning to FIG. 1, the memory 111 stores the graph data 121, the training data 122, and the weight data 123. The graph data 121 is data having a graph structure composed of nodes corresponding to arbitrary elements and edges connecting the nodes. The training data 122 is data used in a learning process using a neural network. The weight data 123 is information that manages weights that are the processing results of the learning process of the neural network.

The graph data 121 and the training data 122 are stored in, for example, the storage system 102, and the processor 110 acquires the graph data 121 and the training data 122 from the storage system 102, and obtains the acquired graph data 121 and the training data 122. Load into memory 111.

[CNN on the grid]
FIG. 3 shows an example of a convolutional neural network (CNN). The computer system 100 of the present disclosure executes a CNN on a graph. A typical CNN300 on a grid will be described with reference to FIG. The CNN on the grid processes matrix-like (grid-like) input data such as image data.

In the example of FIG. 3, a 28x28 grayscale handwritten numeric image 301 is input. The CNN300 applies a 5x5 convolution filter and a ReLU function (302) to generate six 24x24 feature maps 303. The CNN 300 applies a 2x2 maximum value pooling to these feature maps (304) to generate a feature map 305 with the image size reduced in half.

The CNN300 also performs a 5x5 convolution filter and a ReLU function (306) to generate 10 8x8 feature maps 307. The CNN300 applies a 2x2 maximum value pooling to these feature maps (308) to generate a feature map 309 with the image size reduced in half. After that, the perceptron 310 consisting of two fully connected layers finally outputs a 10-dimensional vector. The output 10-dimensional vector is converted into a one-hot vector representation by the SoftMax function, and represents the plausibility of which number from 0 to 9 in the input image.

[CNN (forward propagation) on the graph]
Hereinafter, the CNN on the graph of the present disclosure will be described. The computer system 100 executes the CNN on the graph. FIG. 4 shows a configuration example 400 of CNN on the graph. FIG. 4 shows forward propagation (analytical processing) in the CNN 400, which is executed by the data processing unit 120. The CNN 400 on the graph processes the data on a common graph structure in each layer.

The graph structure is composed of a plurality of nodes and a plurality of edges connecting the two nodes. The grid structure is one of the graph structures, and each node is connected to each of the adjacent nodes by an edge. The graph structure in the present disclosure is composed of an arbitrary plurality of nodes and an arbitrary number of edges. A node is connected to any number of edges, one or more.

The graph data is data having a graph structure, and a state value is assigned to each node. The state value can be represented by a vector having one or more dimensions. As shown in FIG. 4, the graph data 121 is input to the CNN 400 on the graph. As described above, each node of the graph data 121 has a state value, and an edge between the nodes is defined.

The CNN 400 of FIG. 4 includes two convolutional layers and two pooling layers, such as the CNN 300 on the grid described with reference to FIG. Specifically, the CNN 400 includes a first convolution layer 411, a first pooling layer 412, a second convolution layer 413, a second pooling layer 414 and a global average polling (GAP) layer 415. The first convolution layer 411 performs a convolution operation of the input graph data 121 on the graph structure of the graph data 121. As a result, two feature maps 402 are generated.

On the graph structure of the first pooling layer 412 and the graph data 121, a pooling process is executed for each of the feature maps 402 to convert them into feature maps 403 with reduced freedom of expression. The second convolution layer 413 performs a convolution operation of the feature map 403 on the graph structure of the graph data 121. As a result, three feature maps 404 are generated.

On the graph structure of the second pooling layer 414 and the graph data 121, a pooling process is executed for each of the feature maps 404 to convert them into a feature map 405 with reduced freedom of expression. The feature map 405 is input to the global average pooling layer 415. The global average pooling layer 415 takes the average value of all neuron values for each channel.

The details of the processing of the convolution layer and the pooling layer will be described later, but each of the convolution layer and the pooling layer updates the state value of each node of the graph data. Therefore, the graph structure of the data input to each layer is maintained in the output data. Specifically, the graph structure of the input graph data 121 and the feature maps 402 to 405 is common. In each layer, there is a neuron corresponding to each node, and the neuron outputs a new value of the corresponding node. In another example, the graph structure of the input data in one or more layers may be different from the graph structure of the output data.

FIG. 5 shows an example of graph data 121. The graph data 121 shown in FIG. 5 includes edge structure information 500 and node structure information 510. The edge structure information 500 includes an edge ID 501, a node ID 502, and an edge attribute 503. The edge ID 501 indicates the ID of each edge of the graph data. The node ID 502 indicates the ID of each node in the graph data. The edge attribute 503 indicates the attribute information given to each edge. The edge attribute 503 includes information on the orientation and type of the edge.

The node structure information 510 includes a node ID 511, an edge ID (OUT) 512, an edge ID (IN) 513, and a node attribute 514. The node ID 511 is the same as the node ID 502. The edge ID (OUT) 512 indicates the ID of the edge flowing out from the node. The node is the start point node of the edge indicated by the edge ID (OUT) 512. The edge ID (OUT) 513 indicates the ID of the edge flowing into the node indicated by the node ID 511. The node is the end point of the edge indicated by the edge ID (IN) 513. The node attribute 514 indicates the attribute information of the node. The node attribute 514 contains information on the node type.

Various types of information can be represented by graph data, and for example, map information, molecular structure information, social network information, transportation network information, and the like can be represented by graph data. For example, map locations are represented by nodes and roads are represented by edges. The attribute information of the node representing the land includes the real estate value, the usage pattern of the land such as a station or a park, and the like. Further, the attribute information of the edge representing the road includes the route name, the road width, the attribute of the connected land (node), the direction, and the like. Attribute types are defined for each land and road according to the attribute information of the land and road. For example, by learning the graph CNN from the attribute information representing the existing real estate value as teacher data, the real estate value of the given land can be estimated.

Convolutional operations on graphs can be broadly divided into a method using a graph Fourier transform and a more direct method called a Regional Graph Convolution Network (R-GCN). Of these, the method using the graph Fourier transform has the advantage that the theoretical background is clear, but has the disadvantage that it can be applied only to a simple undirected graph without overlapping edges.

On the other hand, R-GCN is an intuitive extension of the convolution operation on the grid, and can be defined naturally even when attributes are given to directed graph edges, or more generally nodes and edges. be. Although the features of the present disclosure can be applied to any method, an example of R-GCN will be described below.

[Example of R-GCN]
A known example of R-GCN will be described with reference to FIGS. 6, 7 and 8. The convolution operation updates the state value of a node based on the state value of the node and other nodes around it. In the following, an example of the process of updating the state value of one node will be described. FIG. 6 shows one node 600 and nodes 611 to 614 and 621 to 623 connected to the node 600. The edge connecting the node 600 and the nodes 611 to 614 each has the attribute R1. The edge connecting the node 600 and the nodes 621 to 623 each has the attribute R2.

FIG. 7 is a part of the weight data 123 and shows the value related to the convolution of the node 600. The weight data 123 includes the edge attribute type 601 and the weight 602. The edge attribute type 601 indicates an identifier of the edge attribute type. In this example, the edge attribute type 601 is different from the type shown by the edge attribute 503 of the graph data 121. These may be the same. The weight 602 indicates the weight of each edge attribute type. The value indicated by the weight 602 is updated in the learning (training) of the CNN 400. The bias of the convolution operation of the state value of the node 600 in this example is set to 0.

As shown in FIGS. 6 and 7, the edge attribute type is defined by the edge attribute and the direction of the edge with respect to the node 600. Specifically, the edge attribute type of the edge connecting the nodes 612 to 614 and the node 600 is R1 (IN). The edge attribute type of the edge connecting the node 611 and the node 600 is R1 (OUT). The edge attribute type of the edge connecting the nodes 622 and 623 and the node 600 is R2 (IN). The edge attribute type of the edge connecting the node 621 and the node 600 is R2 (OUT). The start point and end point of the edge 631 are self-loops of the node 600, and the edge attribute type thereof is SELF.

FIG. 8 schematically shows the update of the state value of the node 600. In the convolution operation, first, the sum of the state values of the nodes of the same edge attribute type is calculated. Specifically, it is as follows. The edge attribute type of the nodes 612 to 614 is R1 (IN). The convolution operation calculates the sum 811 of the state values 801 of the nodes 612 to 614. The edge attribute type of the node 611 is R1 (OUT). The convolution operation calculates the sum 812 of the state values 802 of the node 611.

The edge attribute type of the nodes 622 and 623 is R2 (IN). The convolution operation calculates the sum of 813 of the state values 803 of the nodes 622 and 623. The edge attribute type of the node 621 is R2 (OUT). The convolution operation calculates the sum 814 of the state values 804 of the node 621. The convolution operation calculates the total sum 815 of the state values 805 of the node 600 for the self-loop edge 631.

Next, the convolution operation calculates the sum of products of the weights corresponding to the sum of the edge attribute types. The weights of the edge attribute type are shown in the weight data 123 of FIG. That is, in the convolution operation, the sum of the edge attribute types is multiplied by the weight indicated by the weight data 123, and the sum of them is calculated (821). The convolution operation outputs a state value 831 obtained by applying the ReLU function 822 to this sum of products.

In the example of CNN on the graph described with reference to FIGS. 6 to 8, the update value of the state value of the target node is determined from the state value of the target node and the node state value adjacent to the target node. The adjacent node is a node one hop from the target node. The number of hops represents the distance between the nodes and the number of edges between the nodes. If there are multiple routes from one node to another, the number of hops for each route is defined.

[Processing of each layer of CNN on the graph]
FIG. 9 shows the details of the first convolution layer 411, the first pooling layer 412, the second convolution layer 413, and the second pooling layer 414 of the CNN 400 on the graph. The first convolution layer 411 calculates the updated value of the state value of each node from the state value of the node and the node in one hop by using the ReLU function. The first pooling layer 412 executes the pooling process within one hop. The details of the pooling process will be described later.

The second convolution layer 413 calculates the updated value of the state value of each node from the state value of the node and the node within 2 hops by using the ReLU function. The second pooling layer 414 executes the pooling process within two hops. Instead of executing the pooling process within 2 hops, 1 hop pooling may be executed twice.

As shown in FIG. 9, the range of the convolution calculation of the second convolution layer 413 is wider than the range of the convolution calculation of the first convolution layer 411. Further, the first pooling layer 412 executes the pooling process within one hop, whereas the second pooling layer 414 executes the pooling process within two hops. This means that the range of the pooling treatment of the second pooling layer 414 is wider than the range of the pooling treatment of the first pooling layer 412.

As described with reference to FIG. 3, the CNN300 on the conventional grid performs a pooling process (usually a maximum value pooling process) after a convolution operation (convolution operation + application of a nonlinear activation function). Reduce the image size. On the other hand, the pooling layers 412 and 414 of the CNN 400 on the graph of this example execute the pooling process without reducing the graph structure. As will be described later, the pooling process in the CNN 400 on the graph substantially performs pooling by incorporating the regularization of bringing the values closer to each other while keeping the size of the graph constant.

The second convolution layer 413 immediately after the first pooling layer 412 looks at a wide range, that is, widens the kernel size of the convolution filter because the values of neighboring nodes approach each other in the pooling layer. This makes the convolution operation after the pooling process more effective. Since the kernel size of the convolution filter of the second convolution layer 413 is expanded in the pooling layer 414 after the second convolution layer 413, a more effective pooling process is realized by expanding the pooling range.

In the example shown in FIG. 9, the range of the first convolution layer 411 is within one hop, but the range of the first convolution layer 411 may be wider. As long as the range of the second convolution layer 413 is wider than the range of the first convolution layer 411, their relationship is not limited. For example, the range of the second convolution layer 413 may be 3 hops or more.

Examples of CNN400 on the graph can include three or more convolutional layers and pooling layers, respectively. For example, the range (number of hops) of the convolution operation of the convolution layer is, for example, a constant multiple of the previous convolution layer. In the example of doubling the number of hops, the range of the convolution operation of the convolution layer increases to 1 hop, 2 hops, 4 hops, and 8 hops. In the example of 1.5 times the number of hops, the range of the convolution operation of the convolution layer increases to 1 hop, 2 hops, 3 hops, 4 hops, and 6 hops. The range (number of hops) of the pooling process of the pooling layer is matched with, for example, the number of hops of the immediately preceding convolution layer.

The following is an example of processing of CNN400 on the graph. FIG. 10 shows an example of graph data 121 input to the CNN 400 on the graph. The graph data 121 includes nodes N1 to N13 and a plurality of edges connecting the nodes, respectively. Although not explicitly shown in FIG. 10, a state value is defined for each of the nodes N1 to N13, and an edge attribute type is defined for each of the edges.

[1st convolution layer]
FIG. 11 shows an example of the convolution operation of the first convolution layer 411. FIG. 11 shows an example of updating the state value of the node N1. The first convolution layer 411 executes the convolution operation within one hop. The nodes within one hop from the node N1, that is, the nodes connected to the node N1 at one edge are the nodes N2, N6, N9, N10 and N12. The first convolution layer 411 collects the state values of the nodes N2, N6, N9, N10 and N12, and calculates the update state value of the node N1 from the state values of these and the state value of the node N1.

FIG. 12 shows an example of the attributes of the graph data 121. An edge attribute type is defined for each edge of the graph data, and is indicated by a code consisting of P and a number. For example, the edge attribute type of the edge between the node N1 and the node N12 is R1, the edge attribute type of the edge between the node N1 and the node N2 is R2, and the edge attribute type of the edge between the node N1 and the node N6. The type is R3. The edge of the edge attribute type R1 is represented by a solid line, the edge of the edge attribute type R2 is represented by a broken line, and the edge of the edge attribute type R3 is represented by a two-dot chain line.

FIG. 13 shows a part of the weight data 123 of this example. The weight data 123 defines the weight W0 of the current state value of the target node and the weight of the edge attribute type of 1 hop, and also defines the weight of the edge attribute type of 2 hops. The edge attribute type of one hop matches the edge attribute types R1, R2, and R3 of each edge. The two-hop edge attribute type is represented by two Ps and two numbers, and indicates a set of two edge edge attribute types.

More specifically, the two-hop edge attribute type indicates an edge attribute type of an edge close to the target node and an edge attribute type of an edge far from the target node. For example, in the edge attribute type R1R2, the edge attribute type of the edge between the target node and the adjacent node is R1, and the edge attribute type of the edge between the adjacent node of the target node and the node two hops away is R2. Represents that.

FIG. 14 shows the update of the state value of the node N1 by the first convolution layer 411. The first convolution layer 411 has the state value V1 of the target node N1 and the state values V2, V6, V9, V10, V12 of the nodes within one hop of the target node N1, that is, the adjacent nodes N2, N6, N9, N10, N12. To get. The edge attribute types of the edges between the target node N1 and the adjacent nodes N2, N6, N9, N10, and N12 are R2, R3, R1, R2, and R1.

As shown by the weight data 123 in FIG. 13, the weight of the state value of the own node is W0, and the weights of the edge attribute types R1, R2 and R3 are W1, W2 and W3, respectively. In the example of the convolution operation, different weights for each edge attribute type of the edge are multiplied by the state value of the node and added, and the sum of all the edge attribute types is calculated. That is, the convolution operation for the node N1 is expressed by the following mathematical formula. The bias was set to 0.
V1'= W0 x V1 + W1 * (V9 + V12) + W2 * (V2 + V10) + W3 * V6

The first convolution layer 411 inputs the state value V1'as a result of the convolution operation to ReLU, which is a nonlinear function, and holds the output as a new state value V1 of the node N1. A non-linear function different from ReLU (for example, a sigmoid function) may be used. The first convolution layer 411 performs the above-mentioned processing for all the nodes.

[First pooling layer]
Next, the processing of the first pooling layer 412 will be described. In the conventional CNN on the grid, the image size is generally reduced by a pooling process after the convolution operation (+ application of the nonlinear activation function). The pooling layer of the CNN 400 on the graph of the present disclosure executes a process corresponding to the conventional pooling process, that is, a pooling process on the graph, without changing the graph structure of the input graph data 121.

The pooling process is a process of bringing values closer to each other at neighboring nodes, thereby reducing the degree of freedom of expression. An example of pooling processing on some graphs will be described below. Although several methods are known for the pooling process, the commonly used methods are mean value pooling and maximum value pooling. Therefore, in the following, the mean value pooling and the maximum value pooling on the graph will be described.

Similar to the convolution operation, the pooling process updates the state value of each node while maintaining the graph structure. The pooling process determines a new state value of the target node from the state values of the target node and the nodes in the vicinity of the target node. FIG. 15 schematically shows the pooling process by the first pooling layer 412. In FIG. 15, two nodes N1 and N2 are of interest. The first pooling layer 412 updates the state value of the node N1 so as to approach the state values of the adjacent nodes N2, N6, N9, N10, N12. Similarly, the first pooling layer 412 updates the state value of the node N2 so as to approach the state value of the adjacent nodes N1, N3, N4, N5.

In the example of average value pooling on the graph, based on the state values of the target node and the adjacent node, the new state value of the target node becomes the average value of the new state values of the target node and the adjacent node. Determine the new state value of the target node. For example, the first pooling layer 412 acquires the current (pre-update) state values of the node N1 and its adjacent nodes N2, N6, N9, N10, N12 and calculates the average value thereof. The first pooling layer 412 determines the average value as a new state value of the node N1. The first pooling layer 412 may calculate the average value of the values weighted to the state values of the nodes according to the edge attribute type.

Another example of mean pooling on the graph determines a new state value for the target node according to the following formula. N indicates a set of nodes in the graph data 121, and E indicates a set of edges. pi is the output of the pooling process of the node _i , and vi is the input of the pooling process of the node _i . _vi is the output (state value) of the immediately preceding convolution operation. β is a hyperparameter, I is an identity matrix, and L is a Laplacian matrix. D is the total number of edges of the graph data 121. The reason why the second term on the right side is divided by D is to make the strength of the constraint per edge uniform, but a configuration that does not divide by D (D = 1) is also conceivable.

The output p _i of the first pooling layer 412 of the node i is a value closer to the input v _i than the first term, and is also a value closer to the output value p _j of the adjacent node than the second term. As β becomes larger, the contribution of the second term becomes larger, so that the values approach in a wider range on the graph. That is, β is a parameter corresponding to the kernel size of the pooling process. The above equation can be used for more general purpose and appropriate average value pooling.

FIG. 16 shows an example of an applicable Laplacian row example L in the graph data 121 having the graph structure shown in FIG. The above equation for pooling and the Laplacian line example L shown in FIG. 16 are used when all edges are uniformly pooled, ignoring the edge attribute type. The pooling process may be executed ignoring the edge attribute type.

Unlike this, the pooling process may be executed based on the edge attribute type. As a result, a more appropriate pooling process can be executed. The first pooling layer 412 executes a pooling process for each edge attribute type. Specifically, an example of pooling processing based on the edge attribute type follows the following formula.

E ₁ , E ₂ , and E ₃ indicate a set of edges of edge attribute types R1, R2, and R3, respectively. β ₁ , β ₂ , and β ₃ are hyperparameters (pooling kernel size) of edge attribute types R1, R2, and R3, respectively. L ₁ , L ₂ , and L ₃ are Laplacian matrices of pooling of edge attribute types R1, R2, and R3, respectively.

Next, an example of maximum value pooling on the graph will be described. The example of maximum value pooling on the graph selects the maximum value of the state value of the node in the pooling range. For example, the first pooling layer 412 acquires the current (pre-update) state values of the node N2 and its adjacent nodes N1, N3, N4, N5 and selects the maximum value among them. The first pooling layer 412 determines the maximum value as a new state value of the node N2. The first pooling layer 412 may select a node to give an update value based on the weight according to the edge attribute type.

Another example of maximum value pooling on the graph is to apply Math Maximum to the input value (the current state value of the node) and then perform average value pooling. Some functions are known as Smooth Maximum, but LogMeanExp can be used, for example. Another example of maximum value pooling on the graph determines a new state value for the target node according to the following formula.

ρ is a parameter representing the strength of Smooth Maximum, and at the limit of ∞, LogMeanExp matches the maximum value function. The other variables are as described in Mean Pooling. The maximum value pooling represented by [Equation 3] is used when all edges are uniformly pooled, ignoring the edge attribute type. The pooling process may be executed ignoring the edge attribute type.

Similar to the description of the average value pooling, the first pooling layer 412 may execute the pooling process based on the edge attribute type. The first pooling layer 412 uses [Equation 2] instead of [Equation 1]. As a result, a more appropriate pooling process can be executed. The first pooling layer 412 executes a pooling process for each edge attribute type.

[Second convolution layer]
Next, the processing of the second convolution layer 413 will be described. As described above, the first pooling layer 412 reduces the degree of freedom of expression of data by adopting the regularization that the values are brought close to each other while keeping the size of the graph structure constant. The second convolution layer 413 refers to a wide range because the values of neighboring nodes approach each other in the first pooling layer 412, that is, the kernel size of the convolution filter (the range of the nodes referred to in the convolution operation) is expanded.

FIG. 17 schematically shows the convolution operation for the node N1 by the second convolution layer 413. A similar convolution operation is performed on all nodes. The second convolution layer 413 acquires the current (pre-update) state value of the node within the range of 2 hops from the node N1. The node within the 2-hop range is composed of a node of 1 hop and a node of 2 hops from the target node. It should be noted that instead of using the nodes within the 2-hop range, that is, the 1-hop node and the 2-hop node, it is conceivable to use only the 2-hop node.

In the example shown in FIG. 17, the nodes N2, N6, N9, N10, and N12 are nodes one hop from the node N1. Nodes N3 to N5 and N7 to N13 are two-hop nodes from node N1. Nodes N9, N10, and N12 are one-hop nodes from node N1 and are also two-hop nodes. Nodes N9, N10, and N12 may be defined as one-hop nodes from node N1 and may be excluded from two-hop nodes from node N1.

The second convolution layer 413 is defined by the state value of the target node N1, the state value of the node 1 hop from the target node N1, the state value of the node 2 hops from the target node N1, and the weight data 123 for each edge attribute type. Calculate the sum of products of the weights. As described with reference to FIG. 13, the weight data 123 defines the weight of the own node, the weight of the edge attribute type of one edge, and the weight of the edge attribute type of the pair of two edges. Although not shown, the weight data 123 further defines the weights of the edge attribute types of the set of three or more types of edges. Similar to the first convolution layer 411, the state value of the target node obtained by the convolution operation is input to ReLU, and its output is held as a new state value of the target node. The convolution operation of the second convolution layer 413 can include a bias.

[Second pooling layer]
As shown in FIG. 9, the second pooling layer 414 executes the pooling process within a two-hop range. For example, the second pooling layer 414 executes the pooling process executed by the first pooling layer 412 twice. As a result, the pooling process can be performed according to the process range of the second convolution layer 413.

[Overall average pooling layer]
Next, the treatment of the whole area average pooling layer 415 will be described. The CNN400 on the graph handles graph data having various graph structures having different numbers of nodes and edges. The fully connected layer in the conventional CNN is based on the premise that the number of neurons in the input layer is fixed, and cannot be directly applied to the CNN 400 on the graph. The CNN400 on the graph of the present disclosure uses global average pooling. If the graph structure of the input graph data is constant, instead of the global average pooling layer, the values of all neurons are collected in one place and affine arithmetic is performed to calculate the final output value. May be used.

The global mean pooling layer 415 takes the mean of the values of all neurons for each channel of its input. Therefore, the output of the global average pooling layer 415 is a vector of the dimensions of the number of input channels. For example, an affine transformation layer (fully connected layer) is arranged after the global average pooling layer 415 to finally obtain an output of the required dimension. Although the global average pooling layer 415 has a significantly smaller number of parameters than the fully connected layer, it can exhibit performance equal to or higher than that of the fully connected layer.

[Learning by error back propagation (training)]
In the following, parameter learning by error back propagation of CNN400 on the graph will be described. The learning process is executed by the data processing unit 120. Error backpropagation follows the nodes of the graph in forward propagation in reverse. As mentioned above, the second convolution layer 413 increases the kernel size of the convolution filter. However, since the values of neighboring nodes are approaching in the first pooling layer 412, simply increasing the kernel size increases the degree of freedom in parameter setting too much, causing overfitting or a defective setting problem and parameters. There is a possibility that you cannot learn at all.

Therefore, the kernel size of the convolution filter is increased with respect to the second convolution layer 413 after pooling, and at the same time, the weights of the neighbors on the convolution filter are brought closer to each other as in the case of the first pooling layer 412 immediately before. By doing so, the degrees of freedom of the first pooling layer 412 and the second convolutional layer 413 immediately after it become the same, and the CNN on the so-called deep learning graph, which stacks multiple layers of convolution operations and pooling processes, can be obtained. Can be configured appropriately.

Here, the learning (error back propagation) of the parameters of the CNN 400 on the graph optimizes a plurality of weights in the convolutional layer by an equation including the same regularization term as the previous pooling layer. More specifically, the regularization term of the convolution layer is included in the error function E (w) of the entire CNN400.

The above equation shows the error function E (w) of the configuration including three or more convolution layers. E ₀ (w) is a normal error function that does not include the regularization term of the convolution layer. The second term on the right side is the regularization term of the second convolutional layer, and the third term is the regularization term of the third convolutional layer. w _{2, i} , w _{2, j} indicate adjacent nodes (joined at one edge) at a node two hops away from a node. w3 _{, i} , w3 _{, j} indicate adjacent nodes in a node 3 hops away from a certain node. β is the same as β used in the pooling layer. The error function E (w) includes the regularization term of the convolution layer after the second convolution layer.

As described above, the kernel size of the convolution filter is increased toward the second convolution layer, the third convolution layer, and the subsequent stages. By increasing the degree of β in the second and third terms on the right side of the above equation to increase the effect of the regularization term, the degree of freedom in substantially parameter setting in each convolution layer is kept constant.

As described with reference to FIG. 13, the weight data 123 includes the weight of the own node, the weight of the edge attribute type of one edge, and the weight of the edge attribute type of the pair of two edges, and the weight of three or more types of edges. Defines the weight of the set edge attribute type. At this time, for example, the second term on the right side, that is, the regularization term of the second convolution layer, calculates the sum of the squares of the weight differences between the adjacent nodes in the two-hop node from each of all the nodes.

The second term on the right side will be described by taking node N1 as an example. The two-hop nodes from node N1 are nodes N3 to N5 and N7 to N13. For w _{2 and i} , all the weights of these nodes are assigned respectively. For w _{2 and j} , all the weights of all the nodes adjacent to the nodes of w _{2 and i} are assigned. The calculation of the second term of node N1 (β / 2 omitted) is expressed as follows.

An example of node N3 will be described. Looking at node N3 from node N1, the weights of the edge attribute types of the pair of two edges are W12, W23 (via node N2), and W23 (via node N10). Therefore, w2 _{and i} of the node N3 are W12, W23 and W23. In the node of 2 hops from the node N1, the nodes adjacent to the node N3 are the nodes N4, N10 and N12. Therefore, w _{2 and j} are assigned the weights of the nodes N1 as seen from these nodes.

For example, looking at node N4 from node N1, the weight of the edge attribute type of the pair of two edges is W22. Looking at the node N10 from the node N1, the weights of the edge attribute types of the pair of two edges are W11 and W12. Looking at node N12 from node N1, the weight of the edge attribute type of the pair of two edges is W22.

The second term calculates the same processing for the node N1 as described above for the node 3 for the nodes N4, N5, and N7 to N13. Further, the same calculation is performed for the nodes other than the node N1. The third term, like the second term, calculates for a node of 3 hops from each of all the nodes.

In another example, the regularization term may perform calculations based on the relationship of edge attribute types instead of the graph structure. For example, in the second term, the calculation may be performed so that the weights having the same edge attribute type of the edge close to the target node approach each other. For example, in each of the weight groups (W11, W12, W13), (W21, W22, W23), (W31, W32, W33), the second term is defined so that the weights approach each other. The third and subsequent terms are similarly defined according to the edge attribute type of 3 hops or more.

The gradient value of the second convolution layer is obtained by differentiating the error function E (w) with w as in the following mathematical formula.

The first term on the right side in [Equation 6] indicates an error signal (normal error back propagation) from the upper layer. The second term on the right-hand side shows the derivative of the normalization term. The gradient values of other convolutional layers can be expressed by the same equation. The learning process updates the weight w of the convolution layer according to this gradient value. For example, known methods such as SGD (stochastic gradient descent) and ADAM can be used. In this way, in the learning by error back propagation, the weight w is updated with the value obtained by adding the differential term (βLw _{, n} ) of the regularization term to the normal error signal back propagated from the upper layer as a gradient. This makes it possible to impose restrictions on the convolutional layer (filter) according to the immediately preceding pooling.

Next, the entire learning process will be described. FIG. 18 is a flowchart illustrating one example of the learning process executed by the computer 101. The processor 110 executes a neural network construction process (S201). The neural network construction process will be described later.

Next, the processor 110 acquires the learning data 122 from the storage system 102 and stores it in the memory 111 (S202). The training data 122 includes a plurality of samples. Next, the processor 110 starts the loop processing of the error back propagation processing (S203). In this loop processing, the processor 110 repeatedly executes the processing of steps S203 to S207. For example, when the error calculated by the error back propagation processing executed in this loop processing becomes equal to or less than a preset threshold value, or when the error back propagation processing is executed a predetermined number of times, the error back propagation processing is executed. The loop processing ends.

The processor 110 starts loop processing of the sample (step S705). In the sample loop processing, the processing inside the loop is executed for each sample for the plurality of samples acquired in step S202. Further, the processor 110 executes an error back propagation process for one sample (S205).

In the error back propagation process, the processor 110 takes a sample as an input and compares the output result of the CNN 400 on the graph with respect to the input with the teacher data corresponding to the sample, so as to reduce the error between the two data. The weights are updated in order from the upper layer of CNN400.

In the sample loop processing, the processor 110 executes the above-mentioned processing for each of the plurality of samples. In the error back propagation process in step S205, the weight is updated only once for any sample data. On the other hand, in the loop processing of the error back propagation processing, the error back propagation processing is executed a plurality of times in order to minimize the error. Specifically, the processor 110 repeatedly executes an error back propagation process for each of the plurality of samples until a predetermined condition is satisfied. Further, in the sample loop processing, the weights are updated for a plurality of samples.

Next, the processor 110 determines whether or not the error back propagation process has been executed for all the sample data (S206). If it is determined that the error back propagation process has not been executed for all the sample data, the processor 110 returns to step S204 and executes the same process. When it is determined that the error back propagation process has been executed for all the sample data, the processor 110 determines whether or not the predetermined condition is satisfied (S207).

If it is determined that the predetermined condition is not satisfied, the processor 110 returns to step S203 and executes the same process. When it is determined that the predetermined condition is satisfied, the processor 110 stores the learning result in the memory 111 or the storage system 102 (step S208). The learning result includes information showing the structure of the constructed neural network, information on weights obtained by learning, and the like. Weights are obtained for each edge attribute type and are used as weight data 123 in the analysis process.

[Construction of CNN on the graph]
The CNN400 on the graph handles graph data having various graph structures having different numbers of nodes and edges. The data processing unit 120 constructs a CNN 400 according to the graph data 121 to be analyzed. The data processing unit 120 constructs each of the convolution layer and the pooling layer according to the graph structure of the graph data 121. The data processing unit 120 acquires weights for each edge attribute type from the learning result to form weight data 123.

As described above, according to this embodiment, by pooling in the CNN on the graph, the expressible degree of freedom (parameter degree of freedom) can be reduced toward the upper layer, and overfitting during parameter learning can be achieved. It can reduce the probability of learning and well-posed problems. In addition, it is possible to increase the number of CNN layers to form a CNN on a graph that performs so-called deep learning.

The dispersed CNN will be described below. In a distributed CNN, a plurality of subsystems (for example, a computer) connected via a network execute the CNN. A subsystem can communicate (exchange information) only with an adjacent subsystem on the network. The network structure connecting the subsystems does not have to be fixed and may change over time. In a distributed CNN, each subsystem estimates, for example, the value of a system state function independently. Due to the distributed CNN, the estimates of all subsystems match the values of the true system state function.

In another example, each subsystem may take action depending on its environmental input and internal state. Each subsystem can find an action that optimizes the value of the state function of the whole system. For example, factories (computers) connected via a network communicate only with adjacent factories and determine the power consumption of the factories so as to minimize the total profit of all factories. A factory or its computer is a subsystem.

In addition, distributed CNN can be applied to vehicle route optimization by vehicle-to-vehicle communication. A car or its calculator (eg, a car navigation system) is a subsystem. Each car decides the route independently from the current position and the destination. As each vehicle communicates with a neighboring vehicle, a large number of vehicles together constitute one distributed CNN. Each vehicle uses the route of its own vehicle as an input and returns (estimates) the average travel time by the distributed CNN.

Each vehicle independently updates the route (= input) in the direction of reducing the output value (= average moving time) of the distributed CNN by error back propagation. At the same time, each vehicle independently updates the parameters (weights) with error backpropagation so as to reduce the error between the true average travel time obtained from the road traffic information and the output value of the distributed CNN. Each car leaves the distributed CNN once it reaches its destination.

[System configuration]
FIG. 19 shows an example of a computer system that implements a distributed CNN. The computer system includes a plurality of computers 101A to 101M, which are connected via a network. Adjacent computers that can communicate are connected at the edge. The configuration of the computer is as described in the first embodiment. The network configuration of FIG. 19 is an example, and in another example, the computers arranged in a grid may be able to communicate only with the adjacent computer.

The distributed CNN does not treat the neurons that make up the CNN for each layer (convolution layer, pooling layer, etc.), but collects neurons at the same position in multiple layers (hereinafter referred to as columns) into one sub. Calculate in the system. As a result, the entire system, which is a collection of many subsystems, operates as a single CNN.

FIG. 20 shows examples of dispersed CNN layers and columns. The dispersed CNN includes layers 701 to 704 such as a convolution layer and a pooling layer. FIG. 20 shows one column 711 as an example. One column is executed by one subsystem. In the computer system of FIG. 19, any one computer calculates the neurons of the column.

As can be understood from the above description, the CNN on the graph described in Example 1 can be applied to the dispersed CNN. The network structure of the computer system shown in FIG. 19 is consistent with the graph structure of the graph data 121 described in the first embodiment. The computers 101A to 101M perform arithmetic processing of the corresponding node in the graph data 121 in each layer (convolution layer and pooling layer) constituting the CNN, respectively.

FIG. 21 shows a conceptual configuration example of a distributed CNN. FIG. 21 shows three computers (subsystems) 101B, 101C, and 101D as an example. The computer 101D is adjacent to the computer 101B and the computer 101C and can communicate with each other. The computers 101B, 101C, and 101D execute columns 711B, 711C, and 711D, respectively. The squares that make up the column indicate neurons.

Each subsystem holds as an internal state a vector that summarizes the output values of neurons on the column that the device is in charge of. Each subsystem inputs its own internal state vector and the internal state vector of the adjacent subsystem (value of the neuron on the column at the adjacent position) obtained by communicating with the adjacent subsystem, and performs the calculation of each layer to perform its own operation. Update the internal state vector of.

In a computer system that executes distributed CNN, there is no central control system that controls the entire system, and there is no global synchronization mechanism for the entire system. Each subsystem communicates with its neighbors and updates the subsystem's internal state vector asynchronously with the other subsystems. The subsystem updates the internal state vector of its own subsystem based on the internal state vector of other subsystems and the internal state vector of its own subsystem.

Communication with a neighboring subsystem and updating of the internal state vector of a subsystem are not always performed at the same frequency, and one subsystem communicates frequently with the neighboring subsystem, while another subsystem communicates frequently. The frequency of updating the internal state vector may be high. When a subsystem has multiple neighboring subsystems, the frequency of communication with each neighboring subsystem does not have to be the same.

The computer system that executes the distributed CNN performs communication between adjacent subsystems and update processing of the internal state vector asynchronously. Therefore, when performing an update of the internal state vector, the subsystem cannot obtain the value of the neuron on the adjacent column (the column executed by the adjacent subsystem) required for the convolution operation.

Therefore, the subsystem stores the value of the neuron on the adjacent column (internal state vector) obtained during the previous communication with the adjacent subsystem in the memory, and the value stored when the internal state vector is updated. Performs a convolution operation using. This configuration makes it possible to perform error backpropagation asynchronously, as will be described later.

[Convolution]
The treatment of each layer of the dispersed CNN will be described below. The computer system can execute the processing of each layer in the CNN 400 on the graph described in the first embodiment. First, the processing of the convolutional layer will be described. Each subsystem that executes the distributed CNN performs the convolution operation of the column in charge of its own system. Each subsystem performs the processing of the convolutional layer described in Example 1 in the column in charge.

As described in the first embodiment, the convolution operation uses the value of the target node (neuron) and the value of the node having a predetermined number of hops from the target node. In a distributed CNN, each subsystem collects the values of nodes other than the target node (neuron) from other subsystems.

As described above, each subsystem stores the value of the neuron on the adjacent column obtained when communicating with the adjacent subsystem, and when updating the internal state vector, the convolution operation is performed using the stored value. I do. The weight of the convolution operation (kernel coefficient) is common to all subsystems. For example, subsystems can share the same weight data by communicating with each other.

The subsystem requires the values of all columns within the kernel range (number of hops) of the convolution operation. As described in Example 1, the kernel size in the convolution chapter after the pooling layer is increased. For a kernel-sized convolution operation of two hops or more, such as the processing of the second convolution layer 413, the subsystem passes the column values (internal state vector) of the non-adjacent subsystem through the adjacent subsystem. Acquired by multi-hop relay. N-hop information transmission requires n communications.

[Pooling]
Next, the pooling process will be described. In a distributed CNN, each subsystem treats neurons at the same position from the lower layer to the upper layer of the CNN as columns, so the number of neurons is constant in all layers. This is the same as the CNN on the graph described in Example 1. In the distributed CNN, each subsystem performs a pooling process that reduces the degree of freedom of expression without reducing the number of neurons (number of nodes), similar to the pooling process described in the first embodiment.

Both the mean value pooling and the maximum value pooling described in Example 1 can be applied to the pooling treatment in the dispersed CNN. As described in the first embodiment, the pooling process requires the state value of the adjacent node. In a distributed CNN, the subsystem calculates a new internal state vector of its own system from the internal state vector of its own system and the internal state vector acquired from the adjacent subsystem.

[Full combination]
Next, the full connection will be described. The centralized CNN calculates the final output value by repeating the convolution operation and pooling several times, and then collecting the values of all neurons in one place by the fully connected layer and performing an affine operation. On the other hand, in the distributed CNN, each subsystem operates autonomously, and it is not possible to collect the values of all the subsystems in one place.

Therefore, the computer system of the present disclosure uses a known algorithm for obtaining an average value by variance. The algorithm for obtaining the mean value by the variance is, for example, a variance ADMM (Alternating Direction Methods of Multipliers) or an average consensus algorithm.

Specifically, each subsystem first multiplies the output value of the neuron on its column by the weighting factor of the corresponding fully connected layer. Next, the output value of the fully connected layer can be obtained by calculating the average value of the multiplication result values of all the subsystems by dispersion using the algorithm described later. In this method, the value of the neuron is multiplied by the weighting factor, and the average value is output instead of the sum. Since the weighting factor is learned from the training data (teacher data), this difference does not matter. When the weighting coefficient learned in the conventional centralized CNN is used in the distributed CNN, the value of the weighting coefficient held in each subsystem may be multiplied by the number of all the subsystems in advance.

In the following, two examples of the algorithm for calculating the mean value by variance will be described. The first example is an average value calculation algorithm using a distributed ADMM. By solving the following optimization problem using the variance ADMM, the mean value ν of N values x _i can be calculated by the variance.

Since the method using the distributed ADMM always optimizes to satisfy the above equation, even if the value xi or the network topology changes dynamically during the operation, it follows the change and averages at that time. The value ν can be output. Therefore, it is possible to calculate the average value for the value at any specified time point.

The second example is the average consensus algorithm. The average consensus algorithm distributes the value x _i among the subsystems while keeping the sum of the values x _i held by each subsystem constant, and finally the same value for all the subsystems x _i . Converge to. Since the sum of the values x _i is always constant, the convergence value is the average value of the initial values of each value x _i . Of the several types of averaging algorithms, the simplest averaging algorithm updates, for example, the value x _i and the value x _j according to the following equation during communication between the subsystem i and the subsystem j.

As a result, each time communication between neighbors is performed, the variance of the value x _i is reduced while keeping the sum of the values x _i of the entire system constant. Therefore, all the values x _i are the same value, that is, It will converge to the average value of the initial values of each value x _i .

In the average agreement algorithm, if the value of the value x _i or the network topology changes dynamically during operation, the sum of the values x _i in the entire system changes, so the average value of the initial values of each value x _i is calculated. I can't calculate correctly. On the other hand, since the convergence value of the average consensus algorithm is always the average value of the system-wide value x _i at that time, it is useful for learning the weighting coefficient of the convolution filter by error back propagation.

[Learning by error back propagation (training)]
As described above, in the distributed CNN, the subsystem stores the value of the neuron on the adjacent column obtained at the time of the previous communication with the adjacent subsystem, and the convolution operation etc. is performed using the stored value. Update the internal state vector by performing the forward propagation calculation of. By using the value used at the time of forward propagation at the time of error back propagation calculation as it is, the subsystem can update the parameters by error back propagation without synchronization with other subsystems.

The calculation of error back propagation is performed by tracing the nodes (neurons) of the forward propagation graph in reverse. Figures 22 to 25 show examples of propagation rules for each type of node on the graph. Specifically, FIGS. 22 to 25 show the application node 771, the countable node 772, the multiplication node 773, and the variable duplication node (branch node) 774 of the function f, respectively. The propagation law of forward propagation is shown by a solid line, and the propagation law of back propagation is shown by a broken line. The forward propagation inputs are x and y. In the distributed CNN, it is necessary to devise the realization of the variable duplication node 774.

The variable replication node 774 corresponds to using the same variable multiple times when calculating forward propagation. In the distributed CNN, since a plurality of subsystems are distributed and calculated, in the back propagation rule of the variable replication node 774 in FIG. 25, among the error signals dx1 and dx2 from the upper layer, the column in charge of the subsystem other than the own subsystem is in charge. You can't get a signal from.

Therefore, in the error back propagation of the variable replication node 774, each subsystem independently calculates the parameter minute coefficient and updates the parameter value using only the error signal obtained inside the column in charge of its own subsystem. do. Each subsystem then calculates the average of the parameter values held by all the subsystems using the above average consensus algorithm. In the mean consensus algorithm, since the convergence value is always the average value of the values possessed by all the subsystems at that time, continuous learning, that is, updating of the parameter value can be continued.

In addition, while the mathematically correct minute coefficient is the sum of all the error signals from the upper layer, this method updates the parameter by considering the average value of all the error signals from the upper layer as the minute coefficient. Therefore, the learning rate (Learning Rate), which is a hyperparameter of the learning algorithm used for parameter update, is multiplied by the number of all subsystems in advance.

If there is another node after the branch node in reverse error propagation (previous stage in forward propagation), similarly, in the branch node, each subsystem sends an error signal using only the information in the column that its own subsystem is in charge of. It is back-propagated and the parameter update in the subsequent stage is executed independently, and then the average value of the parameter values is calculated by the average agreement algorithm.

[Configuration example of distributed CNN]
Hereinafter, a configuration example of the dispersed CNN of the present disclosure will be described. FIG. 26 shows a configuration example 900 of the distributed CNN and its processing. In the following, the processing executed by one subsystem will be described. First, forward propagation will be described. In forward propagation, the first convolution layer 901 performs a convolution operation and a ReLU in the same manner as the first convolution layer 411 of the first embodiment. The first convolution layer 901 acquires the value of the target neuron from its own internal state vector, and also acquires the value of the adjacent neuron from the internal state vector of the adjacent subsystem held in advance. The first convolution layer 901 executes a convolution operation and ReLU of these values, and updates its own internal state vector.

The first pooling layer 902 performs a pooling process in the same manner as the first convolution layer 411 of the first embodiment. The first pooling layer 902 acquires the value of the target neuron updated by the first convolution layer 901 from the self-internal state vector, and also obtains the value of the target neuron updated by the first convolution layer 901 from the internal state vector of the adjacent subsystem held in advance, and the first of the adjacent neurons. The output value of the convolution layer 901 is acquired. The first pooling layer 902 performs pooling of these values, maximum value pooling in this example, and updates its own internal state vector.

The second convolution layer 904 executes the convolution operation and ReLU in the same manner as the second convolution layer 413 of the first embodiment. The second convolution layer 904 acquires the value of the target neuron updated to the first pooling layer 902 from the self-internal state vector, and also obtains the value of the neighboring neuron from the pre-held internal state vector of the subsystem in the two hops. The output value of the first pooling layer of is acquired. Data from subsystems two hops apart are transferred via neighboring subsystems. The second convolution layer 904 executes a convolution operation and ReLU of these values, and updates its own internal state vector.

The second pooling layer 906 performs a pooling process in the same manner as in the second pooling layer 414 of the first embodiment. The second pooling layer 906 acquires the value of the target neuron updated to the second convolution layer 904 from the self-internal state vector, and also obtains the value of the adjacent neuron from the internal state vector of the adjacent subsystem held in advance. get. The second pooling layer 906 performs pooling of these values, maximum value pooling in this example, and updates its own internal state vector.

The second pooling layer 906 further acquires the value of the neuron updated in the previous pooling process from its own internal state vector, and acquires the value of the adjacent neuron from the internal state vector of the adjacent subsystem held in advance. .. The second pooling layer 906 performs pooling of these values, maximum value pooling in this example, and updates its own internal state vector.

For the fully connected layer 907, the average value is calculated by the distributed ADMM. The fully connected layer 907 calculates the mean value by the distributed ADMM from the value of the target neuron of the self-internal state vector and the value of the target neuron of the internal state vector of another subsystem held in advance.

Next, error back propagation will be described. In error backpropagation, each subsystem is given teacher data. Each subsystem uses the data it holds internally to update its weighted data (including bias) independently of the other subsystems. Each subsystem calculates the mean value of the updated weight data by the mean consensus algorithm. As a result, the weight data of all the subsystems is standardized.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the configurations described. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above-mentioned configurations, functions, processing units and the like may be realized by hardware, for example, by designing a part or all of them by an integrated circuit or the like. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card. In addition, the control lines and information lines indicate what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

101 computer, 110 processor, 111 memory, 123 weight data, 400 CNN, 411 first convolution layer, 412 first convolution layer, 413 second convolution layer, 414 second pooling layer, 900 distributed CNN.

Claims (11)

A computer system that executes a convolutional neural network on a graph.
With one or more processors
Including one or more storage devices,
The convolutional neural network on the graph
With one or more convolutional layers,
Including one or more pooling layers,
The one or more storage devices store the kernel weight data of the one or more convolution layers.
The above-mentioned one or more processors
In each convolution layer, the value of each node is updated by a kernel-based convolution operation with a predetermined number of hops.
In each pooling layer, the value of each node is updated by the pooling process based on the value of each node and the value of the node within the pooling range of a predetermined number of hops from each node.
A computer system in which the size of the kernel of the convolution layer behind the pooling layer is larger than the size of the kernel of the convolution layer before the pooling layer . The computer system according to claim 1.
A computer system in which the pooling range of the pooling layer after the convolution layer is wider than the pooling range of the pooling layer before the convolution layer . The computer system according to claim 1.
The computer system in which the one or more processors perform the pooling process of the pooling range of the predetermined number of hops by repeating the pooling process of the one hop range as many times as the number of times matching the predetermined number of hops. The computer system according to claim 1.
The one or more processors are computer systems that include the regularization term of the one or more convolutional layers in the error function in the error backpropagation learning of the convolutional neural network on the graph. The computer system according to claim 1.
A computer system in which one or more processors perform average value pooling after applying a smooth maximum to an input value in the pooling process. The computer system according to claim 1.
A computer system in which the one or more processors perform global average pooling in the subsequent stages of the one or more convolutional layers and the one or more pooling layers. The computer system according to claim 1.
The computer system includes a plurality of subsystems connected by a network.
Each of the plurality of subsystems communicates only with an adjacent subsystem in the plurality of subsystems .
Each of the plurality of subsystems is a computer system that calculates a column composed of neurons at the same position in the convolutional neural network on the graph. The computer system according to claim 7.
Each of the above-mentioned multiple subsystems
Holds an internal state vector showing the value of the column,
Obtaining the internal state vector from other subsystems, including neighboring subsystems,
A computer system that calculates the column using its own internal state vector and the internal state vector of the other subsystem. The computer system according to claim 7.
The plurality of subsystems are computer systems that calculate the average value of the values of the columns by dispersion in the subsequent stages of the one or more convolutional layers and the one or more pooling layers. The computer system according to claim 7.
Each of the above-mentioned multiple subsystems
Independent of other subsystems, the weights of the one or more convolution layers are updated by learning by error backpropagation.
A computer system that calculates the average value of the updated weights by communicating with the other subsystems. A way in which a computer system performs a convolutional neural network on a graph,
The computer system is
With one or more processors
Including one or more storage devices,
The convolutional neural network on the graph
With one or more convolutional layers,
Including one or more pooling layers,
The one or more storage devices store the kernel weight data of the one or more convolution layers.
The method is
In each convolution layer, the value of each node is updated by a kernel-based convolution operation with a predetermined number of hops.
In each pooling layer, the value of each node is updated by the pooling process based on the value of each node and the value of the node within the pooling range of a predetermined number of hops from each node.
A method in which the size of the kernel of the convolution layer behind the pooling layer is larger than the size of the kernel of the convolution layer before the pooling layer .

Images