CN114818515A - Multidimensional time sequence prediction method based on self-attention mechanism and graph convolution network - Google Patents

Abstract

The invention belongs to the technical field of data processing, and discloses a multi-dimensional time series prediction method based on a self-attention mechanism and a graph convolution network. Completion; then segment the completed time series data into training set, validation set and test set; use the obtained training set to train a pre-built multi-dimensional time series prediction model based on self-attention mechanism and graph convolutional network, Use the validation set to verify the training effect and adjust the network parameters. When the multi-dimensional time series prediction model is trained to meet the preset convergence conditions, use the test set to test the model prediction effect, and take the model with the best test effect as the final multi-dimensional time series. Prediction model: Input the historical data of the industrial time series to be predicted into the obtained multi-dimensional time series prediction model, and predict the change trend of the time series data, thereby predicting the equipment operation status and abnormal risk.

Description

A Multidimensional Time Series Prediction Method Based on Self-Attention Mechanism and Graph Convolutional Networks

technical field

The invention belongs to the technical field of data processing, and in particular relates to a multi-dimensional time series prediction method based on a self-attention mechanism and a graph convolution network.

Background technique

Time series analysis and prediction of industrial big data can provide security for automated and intelligent production processes, and is of great significance to the condition monitoring, risk prevention and process improvement of automated production. Due to the continuous improvement of production requirements and the popularization of smart devices, there are more and more types and quantities of sensors used for equipment and environmental monitoring. The time series data recorded by various types of sensors will form multi-dimensional time series, and the potential correlation between dimensions leads to Industrial multidimensional time series data has strong spatial and temporal complexity. In time, the future value of data in each dimension of the time series is related to the historical value; in space, each dimension of the time series does not exist independently but depends on each other. The spatiotemporal dependence of industrial multidimensional time series data brings challenges to time series data prediction.

In response to the problems and challenges mentioned above, there have been many methods and researches on time series forecasting in recent years. Time series forecasting based on statistical models is generally not ideal for long-term forecasting, and it is mostly suitable for univariate and it is difficult to mine the interdependence between complex structural sequence combinations. In the time series prediction based on the data model, the BP neural network training takes a long time, and it is easy to fall into the local minimum; in the prediction method based on SVM, the selection of the kernel function has the most direct impact on the result, and the difficulty is high; the LSTM-based prediction method Although the prediction method solves the problem of RNN gradient explosion or disappearance to a certain extent, it is theoretically possible to fall into a local optimal solution, and the prediction has a lag. For multi-dimensional industrial time series data prediction in industrial big data scenarios, not only temporal but also spatial dependencies must be considered, that is, there are complex dependencies between data dimensions, and existing methods have not effectively utilized this Space-time dependent properties.

SUMMARY OF THE INVENTION

In view of the deficiencies in the prior art, the present invention provides a multi-dimensional time series prediction method based on a self-attention mechanism and a graph convolution network, which takes the large-scale and complex multi-dimensional time series data in the industrial big data scenario as the research object, The self-attention mechanism and graph convolutional network are used to establish a time series prediction model, and the spatiotemporal correlation information between multi-dimensional time series data is mined to achieve more accurate and efficient industrial time series prediction. Through time series prediction, it can perceive potential hidden problems in the industrial production process, so as to provide decision support for risk prevention or abnormal early warning.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

A multi-dimensional time series prediction method based on self-attention mechanism and graph convolutional network, including the following steps:

Step 1. Preprocess the multi-dimensional time series data recorded by each sensor in the industrial production scenario: complete the missing data in time series;

Step 2. Data set division: divide the completed time series data into training set, validation set and test set;

Step 3. Construction and training of multi-dimensional time series prediction model: Use the training set obtained in step 2 to train the pre-built multi-dimensional time series prediction model based on the self-attention mechanism and graph convolution network, and use the verification set to verify the training effect and adjust the network parameters. When the multi-dimensional time series prediction model is trained to meet the preset convergence conditions, use the test set to test the model prediction effect, and take the model with the best test effect as the final multi-dimensional time series prediction model;

Among them, the neural network of each layer includes an input module, an output module, a graph learning module, a graph convolution module, a temporal convolution module and a self-attention mechanism. The graph structure does not need to be defined in advance. The data adaptive learning graph structure automatically extracts the adjacency matrix of the graph according to different data sources. This adjacency matrix will be input to the graph convolution module in each layer to capture the hidden relationship between the dimensions of the time series; the graph The convolution module uses convolution to extract features to capture the spatial dependencies of multi-dimensional time series data; the temporal convolution module captures the temporal dependencies associated with the multi-dimensional time series data, and the self-attention mechanism learns different time series features after the time convolution module. The local correlation between the adjacent time steps and the time-series characteristics of the interaction and influence degree between adjacent time steps are obtained;

In order to avoid the problem of gradient disappearance, a residual connection is added to the output of the previous layer of network and the output of the graph convolution module. Add skip connection; the output module converts the input channel dimension into the desired output channel dimension;

Step 4: Input the historical data of the industrial time series to be predicted into the multi-dimensional time series prediction model obtained in step 3, and predict the change trend of the time series data, thereby predicting the equipment operation status and abnormal risk.

Further, in the process of learning the graph structure, the graph learning module can not only learn stable and interpretable node relationships when training the model on the training set, but also learn from the test set and other real data sets after the model training is completed. The obtained graph adjacency matrix can be adaptively changed, and the model parameters of the graph learning module will be continuously updated according to the new input data, and then the corresponding graph structure will be learned.

、

，公式如下：Further, when the graph learning module learns the graph structure, it first adopts a random sampling method. In each iteration, the nodes are randomly divided into several groups, and the subgraph structure is learned based on the sampling nodes, so that each node in the graph is divided into several groups. A node may be in a group with any other node except itself, so that the pairwise similarity score between a node and other nodes can be calculated and updated; secondly, the graph learning module learns to extract one-way relationships

,

, the formula is as follows:

、

代表随机初始化的两个节点嵌入，

、

分别是图学习模块学习提取的随机初始化的两个节点的单向关系，

、

是模型训练参数，tanh是激活函数，在迭代训练中可以学习，

是用于控制激活函数饱和状态的一个超参数；为了让得到的图邻接矩阵A成为非对称的，采用如公式（3）所示的操作，其中的

减法操作和ReLU激活函数是为将邻接矩阵正则化，in,

,

represents the randomly initialized two node embeddings,

,

are the one-way relationship between two randomly initialized nodes learned and extracted by the graph learning module,

,

is the model training parameter, tanh is the activation function, which can be learned in iterative training,

is a hyperparameter used to control the saturation state of the activation function; in order to make the obtained graph adjacency matrix A asymmetric, the operation shown in formula (3) is adopted, where the

The subtraction operation and the ReLU activation function are used to regularize the adjacency matrix,

In order to reduce the computational cost of the graph convolution module, formula (4) and formula (5) are used to make the adjacency matrix of the graph more sparse, and at the same time for each node id , its closest topk nodes are selected as its neighbor nodes idx ; while retaining the weight of connected nodes, set the weight of nodes without connectivity to zero, where the argtopk () function is used to calculate the largest topk values in the vector,

。

.

Further, the temporal convolution module includes two extended initial layers, which are respectively composed of multiple one-dimensional convolution filters, but these two extended initial layers are not exactly the same, and one of them is followed by tanh activation. function, works as a filter, and another is followed by a sigmoid activation function, which acts as a gate to control the amount of information passed to the next module; the expanded initial layer contains multiple filters of different sizes, and uses a dilated convolution, with For operating long-term sequences, for a one-dimensional convolution with a convolution kernel size of b and the number of f , the size of the receptive field of the convolution is shown in the formula:

In this temporal convolution module, dilated convolution is used to expand the receptive field, and the expansion factor of each layer of the network is set to h , h > 1, the size of the receptive field of the network increases exponentially with the increase of the hidden layer, and the feeling after the expansion Domains are represented as:

。

.

Further, after the time convolution module, the self-attention mechanism analyzes the feature information extracted by the time convolution to learn its internal relationship, and weights the time series features of different time steps to obtain the mutual relationship between adjacent time steps. The timing characteristics of the role and the degree of influence are as follows:

The structure of the self-attention mechanism consists of three matrices, namely the query vector Q , the key vector K and the value vector V , where K is the representation of the feature vector, Q is task-related, and V aggregates feature information into features according to the attention matrix On the vector; the calculation formula of the correlation matrix in the attention mechanism is defined as follows:

和

指的是时间序列的特征向量，

和

分别指的是特征维度为

和

、时间段长度均为L的时间序列数据矩阵，

指的是

标准化后的值，

、

和

均为可学习权重参数，则计算注意力矩阵E的公式定义为：in,

and

refers to the eigenvectors of the time series,

and

respectively refer to the feature dimension of

and

, a time series data matrix whose time period length is L ,

Refers

normalized value,

,

and

are all learnable weight parameters, then the formula for calculating the attention matrix E is defined as:

指的是矩阵

的转置，

指的是特征维度和时间段长度均为L的时间序列数据矩阵，这里使用Softmax函数将时间注意力矩阵按列归一化，得到的矩阵中每一列的概率之和为1，矩阵E中第i行第j列的元素

表示时间步

对时间步

的影响程度；将第j列的所有概率权重与对应时刻的特征信息相乘并累加到

时间步，那么

时间步的特征信息就可以捕获到其它时间步的动态影响；in

refers to the matrix

transpose of ,

It refers to the time series data matrix with both feature dimension and time period length L. Here, the Softmax function is used to normalize the time attention matrix by column, and the sum of the probabilities of each column in the obtained matrix is 1. element at row i and column j

represents a time step

pair time step

The degree of influence of

time step, then

The characteristic information of the time step can capture the dynamic influence of other time steps;

Therefore, the feature information of different time steps is aggregated to any time step and quantified to obtain the time series feature vector of the interaction and influence degree between adjacent time steps. The calculation formula is defined as:

指的是特征维度为U、时间段长度为L的时间序列数据矩阵。where E refers to the calculated attention matrix, U refers to the total feature dimension,

It refers to the time series data matrix with feature dimension U and time period length L.

Further, for a node in the graph, the graph convolution module exchanges and aggregates the information of the node itself and all of its neighbor nodes, uses the neighbor node information to update the state of its own node, and recursively analyzes the graph. All nodes in the graph perform the same operation to capture the spatial dependencies in the graph structure. The graph convolution module includes two mixed-hop propagation layers, which are divided into three steps: message passing, message selection, and state update, where , the formula for message passing is defined as follows:

是指的信息传播的层数，

是指前一层输出，即当前层要输入的隐藏状态，

是一个超参数，选择不同大小的值就可以控制根节点原始状态保留下来的比例，变量

的定义如下：in

refers to the number of layers of information dissemination,

refers to the output of the previous layer, that is, the hidden state to be input by the current layer,

It is a hyperparameter. By choosing values of different sizes, you can control the proportion of the original state of the root node retained. Variables

is defined as follows:

指的是增广矩阵，I是单位矩阵；where A refers to the adjacency matrix of the graph, D is the degree matrix of the graph,

refers to the augmented matrix, I is the identity matrix;

代表当前层要输出的隐藏状态，

是指用来做特征选择的参数矩阵，G指的是当前信息传播的总层数，The formula definition for message selection is as follows,

represents the hidden state to be output by the current layer,

refers to the parameter matrix used for feature selection, G refers to the total number of layers of current information dissemination,

The interdependence between nodes in the graph is extracted through the graph convolution module, that is, the relationship between dimensions in multi-dimensional time series data. The output of this module will be sent to the next layer of network as the input of the time convolution module. .

Compared with the prior art, the advantages of the present invention are:

(1) The present invention is different from the previous multi-dimensional time series prediction methods. In view of the problem of low accuracy of multi-dimensional time series prediction, the improved graph convolution network and the introduction of self-attention mechanism are used for more accurate multi-dimensional time series prediction. The present invention uses graph convolution. The network establishes a prediction model for multi-dimensional time series data. In the graph convolution network, the spatiotemporal characteristics of multi-dimensional time series data are captured through temporal convolution and graph convolution, respectively, while considering its longitudinal temporal correlation and horizontal spatial dependence. The resulting prediction model has higher interpretability and prediction accuracy.

(2) When learning the dependencies between dimensions, the graph structure, that is, the adjacency matrix of the graph, does not need to be defined in advance, and the graph structure given in advance is not completely accurate, but can be learned adaptively through the graph learning module. , automatically extract the adjacency matrix of the graph according to different data sources, and send it to the subsequent graph convolution module; and in the process of generating the graph structure, the model parameters of the graph learning module will be continuously updated according to the new input data, and learn continuously to the corresponding stable and interpretable node relationships.

(3) A self-attention mechanism is introduced after the time convolution module in the prediction network in the present invention, and the feature information extracted by the time convolution is analyzed to learn the internal relationship between them, and assign the time series features of different time steps. The weights allow the model to focus on several time steps that have a greater impact on the prediction results, reduce the impact of irrelevant features on subsequent spatial feature extraction and the final prediction results, and improve the final prediction accuracy.

(4) Data preprocessing is an important step before time series prediction. In the present invention, the completion method of missing data adopts a completion method that considers spatiotemporal features, which can better learn the true and comprehensive feature distribution of data, which is a prediction model. Training establishes a more complete and objective data foundation, which helps to improve the final prediction accuracy.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

Fig. 1 is the flow chart of the present invention;

2 is a frame diagram of a multi-dimensional time series prediction model of the present invention;

Figure 3 is a schematic diagram of a time convolution module;

Figure 4 is a schematic diagram of the self-attention mechanism;

Figure 5 is a schematic diagram of a graph convolution module.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

In order to make up for the deficiency that the existing methods do not comprehensively consider the dependencies between time and space when performing multi-dimensional time series prediction, this embodiment proposes a multi-dimensional time series prediction method based on a self-attention mechanism and a graph convolution network, which is combined with Figure 1 The flow chart shown includes the following steps:

Step 1. Preprocess the multi-dimensional time series data recorded by each sensor in the industrial production scenario: due to problems such as the acquisition equipment or transmission line failure during the data collection process, the time series data will be missing, and the missing part will lead to deviations in the data distribution characteristics. It will affect the modeling of time series data. Here, the deep convolution generative adversarial network training model is used to complete the missing data in time series.

Step 2. Data set division: Divide the time series data after completion, and divide the training set, validation set and test set. The data proportion is 60%, 20% and 20% respectively. The data set division method adopts the current method. It is enough to have the technology, so I won't go into details here.

Step 3. Construction and training of multi-dimensional time series prediction model: Use the training set obtained in step 2 to train the pre-built multi-dimensional time series prediction model based on the self-attention mechanism and graph convolution network, and use the verification set to verify the training effect and adjust the network parameters. The network parameters can be adjusted by existing methods such as increasing the number of training iterations or adjusting the prediction step size. When the multi-dimensional time series prediction model is trained to meet the preset convergence conditions (the model prediction error is smaller than the preset accuracy control parameters) , use the test set to test the model prediction effect, and take the model with the best test effect as the final multi-dimensional time series prediction model.

Combined with the model frame diagram shown in Figure 2, the neural network of each layer includes an input module, an output module, a graph learning module, a graph convolution module, a temporal convolution module and a self-attention mechanism. The graph structure does not need to be defined in advance. The module adaptively learns the graph structure according to the input multi-dimensional time series data, and automatically extracts the adjacency matrix of the graph according to different data sources. hidden relationship; the graph convolution module uses convolution extraction features to capture the spatial dependencies of multi-dimensional time series data; the time convolution module captures the time dependencies associated with the multi-dimensional time series data, and the self-attention mechanism is in the time convolution module. After that, the local correlation between different temporal features is learned, and the temporal features of the interaction and influence degree between adjacent time steps are obtained.

In order to avoid the problem of gradient disappearance, a residual connection is added to the output of the previous layer of network and the output of the graph convolution module. Add skip connections; the output module converts the input channel dimension into the desired output channel dimension, that is, the output size is adjusted according to different prediction steps.

Step 4. Input the historical data of the industrial time series to be predicted into the multi-dimensional time series prediction model obtained in step 3, and predict the change trend of the time series data, so as to predict the equipment operation status and abnormal risk, which is the judgment and the equipment operation status. The perception of abnormal risk provides preliminary decision support.

In the process of learning the graph structure, the graph learning module can not only learn stable and interpretable node relationships when training the model on the training set, but also learn graphs on the test set and other real data sets after the model training is completed. The adjacency matrix can be adaptively changed, and the model parameters of the graph learning module will be continuously updated according to the new input data, and then the corresponding graph structure will be learned.

Each module is described below.

(1) Graph Learning Module

The main task of the graph learning module is to adaptively learn the graph adjacency matrix according to the input multi-dimensional time series data, and the extracted sparse adjacency matrix will be input to the subsequent graph convolution module to capture the hidden relationship between the various dimensions of the time series.

，这就显然限制了模型处理节点数量较多的大图形的能力。为了解决这个问题，图学习模块在学习图结构时，首先采用一种随机抽样的方法，在每次迭代中，把节点随机分成几组，基于采样节点去学习子图结构。这样图中的每一个节点都有可能和除本身之外其他任意一个节点在一个分组中，从而可以计算和更新一个节点与其余节点之间的两两相似性得分。并且，如果把节点分割到s个组中，每次迭代过程中图学习模块的时间复杂度和空间复杂度就会从

降低到

。经过多次迭代之后，图上所有的节点都经过了良好的训练，因此就可以构造出一个全局图来充分表示和利用空间依赖性。得益于在预测之前可以先进行并行预先计算，所以在计算上也不会占用太多资源。In existing research, in order to construct a graph, some distance is usually used to measure the similarity between a pair of nodes, such as dot product and Euclidean distance. Although these methods can be effectively measured, their time complexity and space complexity are both

, which obviously limits the model's ability to handle large graphs with a large number of nodes. In order to solve this problem, the graph learning module first adopts a random sampling method when learning the graph structure. In each iteration, the nodes are randomly divided into several groups, and the subgraph structure is learned based on the sampling nodes. In this way, each node in the graph may be in a group with any other node except itself, so that the pairwise similarity score between one node and other nodes can be calculated and updated. And, if the nodes are divided into s groups, the time complexity and space complexity of the graph learning module in each iteration process will be changed from

Reduce to

. After many iterations, all nodes on the graph are well trained, so a global graph can be constructed to fully represent and exploit spatial dependencies. Thanks to the parallel pre-computation that can be performed before the prediction, it does not take up too many computational resources.

、

，公式如下：On the other hand, existing distance metrics are usually symmetric or bidirectional. In industrial multi-dimensional time series forecasting, it is believed that a change in one node will lead to a change in another related node. For example, an increase in the speed of a motor may lead to an increase in its temperature. Therefore, the relationship learned by the graph learning module should be one-way, and the one-way relationship is specially extracted.

,

, the formula is as follows:

、

代表随机初始化的两个节点嵌入，

、

分别是图学习模块学习提取的随机初始化的两个节点的单向关系，

、

是模型训练参数，tanh是激活函数，在迭代训练中可以学习，

是用于控制激活函数饱和状态的一个超参数；为了让得到的图邻接矩阵A成为非对称的，采用如公式（3）所示的操作，其中的

减法操作和ReLU激活函数是为将邻接矩阵正则化，in,

,

represents the randomly initialized two node embeddings,

,

are the one-way relationship between two randomly initialized nodes learned and extracted by the graph learning module,

,

is the model training parameter, tanh is the activation function, which can be learned in iterative training,

is a hyperparameter used to control the saturation state of the activation function; in order to make the obtained graph adjacency matrix A asymmetric, the operation shown in formula (3) is adopted, where the

The subtraction operation and the ReLU activation function are used to regularize the adjacency matrix,

In order to reduce the computational cost of the graph convolution module, formula (4) and formula (5) are used to make the adjacency matrix of the graph more sparse, and at the same time for each node id , its closest topk nodes are selected as its neighbor nodes idx ; while retaining the weight of connected nodes, set the weight of nodes without connectivity to zero, where the argtopk () function is used to calculate the largest topk values in the vector,

。

.

The graph structure learned by the graph learning module, that is, the adjacency matrix of the graph does not need to be manually defined in advance, but can be learned automatically. In the process of learning the graph structure, not only can stable and interpretable node relationships be learned when the model is trained on the training set, but also the graph adjacency matrix learned on the test set or other real data sets after the model training is completed. Adaptively changing, the model parameters of the graph learning module will be continuously updated according to the new input data, and then the corresponding graph structure will be learned.

(2) Time convolution module

The main task of the time convolution module is to capture the time dependency of the time series data before and after. In order to obtain medium and long-term dependencies, this module is divided into two parts, as shown in Figure 3, including two extended initial layers, each consisting of multiple layers. One-dimensional convolutional filters, but the two extended initial layers are not exactly the same, one is followed by a tanh activation function, which works as a filter, and the other is followed by a sigmoid activation function, which acts as a gate to control the transfer The amount of information to give to the next module.

The dilated inception layer contains a variety of filters of different sizes, mainly to be able to discover temporal patterns in different ranges, and to use dilated convolutions, mainly to be able to operate on long time series.

First of all, the selection of a correct and appropriate size filter for the convolutional neural network is directly related to the performance of the network. When the kernel of the filter is too large, short-term temporal patterns cannot be represented in detail, and when the kernel is too small, long-term temporal patterns cannot be easily discovered. In image processing, there is a general initialization method that aggregates the outputs of 3 sizes of 1*1, 3*3 and 5*5. Mapping to the field of industrial time series data, the time series value generally changes in several fixed cycles, so in this module, a filter with sizes of 1*2, 1*3, 1*5 and 1*10 is used. The devices are combined into an expanded initial layer. For example, when the change period is 15, you can let the input pass 1*10 first, and then pass 1*5, so that the timing pattern of the corresponding time range can be captured.

Secondly, the receptive fields of the convolution determine the length of the time series that it can operate on. For a one-dimensional convolution with a convolution kernel size b and the number f, the size of the convolution receptive field is as shown in the formula Show:

So if you want to operate on long-term sequences, either the number of convolution kernels is large enough, or the convolution kernels are large enough, which will lead to a very complex structure of the network model. In order to solve this problem, dilated convolution is used in this temporal convolution module to expand the receptive field, so that the network with simple structure can also process long-term sequences. Set the expansion factor of each layer of the network to h , h > 1, the size of the receptive field of the network increases exponentially with the increase of the hidden layer, and the receptive field after the expansion is expressed as:

。

.

(3) Self-attention mechanism

For a certain dimension of industrial multi-dimensional time series data, its data values at different time steps have a high degree of dynamic correlation. The time convolution mentioned above is to capture this time dependency. In order to explore the local correlation between different time series features more clearly, the self-attention mechanism adopted in this module is shown in Figure 4. The structure of the self-attention mechanism consists of three matrices, namely the query vector Q , the key vector K and the value vector V , where K is the representation of the feature vector, Q is task-related, and V aggregates feature information into features according to the attention matrix On the vector; the calculation formula of the correlation matrix in the attention mechanism is defined as follows:

和

指的是时间序列的特征向量，

和

分别指的是特征维度为

和

、时间段长度均为L的时间序列数据矩阵，

指的是

标准化后的值，

、

和

均为可学习权重参数，则计算注意力矩阵E的公式定义为：in,

and

refers to the eigenvectors of the time series,

and

respectively refer to the feature dimension of

and

, a time series data matrix whose time period length is L ,

Refers

normalized value,

,

and

are all learnable weight parameters, then the formula for calculating the attention matrix E is defined as:

指的是矩阵

的转置，

指的是特征维度和时间段长度均为L的时间序列数据矩阵，这里使用Softmax函数将时间注意力矩阵按列归一化，得到的矩阵中每一列的概率之和为1，矩阵E中第i行第j列的元素

表示时间步

对时间步

的影响程度；将第j列的所有概率权重与对应时刻的特征信息相乘并累加到

时间步，那么

时间步的特征信息就可以捕获到其它时间步的动态影响。in

refers to the matrix

transpose of ,

It refers to the time series data matrix with both feature dimension and time period length L. Here, the Softmax function is used to normalize the time attention matrix by column, and the sum of the probabilities of each column in the obtained matrix is 1. element at row i and column j

represents a time step

pair time step

The degree of influence of

time step, then

The characteristic information of the time step can capture the dynamic influence of other time steps.

Therefore, the feature information of different time steps is aggregated to any time step and quantified to obtain the time series feature vector of the interaction and influence degree between adjacent time steps. The calculation formula is defined as:

指的是特征维度为U、时间段长度为L的时间序列数据矩阵。为了减少时间注意力矩阵计算复杂度，使用卷积核大小为1的一维卷积网络来代替矩阵乘法作为编码函数，即

、

和

注意力相关矩阵计算函数在本文的时间注意矩阵计算过程中均为一维卷积操作。where E refers to the calculated attention matrix, U refers to the total feature dimension,

It refers to the time series data matrix with feature dimension U and time period length L. In order to reduce the computational complexity of the temporal attention matrix, a one-dimensional convolutional network with a kernel size of 1 is used instead of matrix multiplication as the encoding function, i.e.

,

and

The attention correlation matrix calculation functions are all one-dimensional convolution operations in the time attention matrix calculation process in this paper.

The self-attention mechanism introduced by this method analyzes the feature information extracted by time convolution to learn its internal connections, and weights the time series features of different time steps to obtain the time series features of the interaction and influence degree between adjacent time steps. , reducing the influence of irrelevant features on subsequent spatial feature extraction and final prediction results, thereby improving the final prediction accuracy and achieving a prediction accuracy of more than 99%.

(4) Graph convolution module

指的是矩阵的转置，包括两个混合跳跃的传播层，MLP是指多层感知机。The main task of the graph convolution module is to exchange and aggregate the information of the node itself and all its neighbor nodes for a node in the graph, use the neighbor node information to update the state of its own node, and recursively analyze the graph in the graph. All nodes perform the same operation, thereby capturing spatial dependencies in the graph structure. The module is mainly divided into three steps: message delivery, message selection and status update. The structure is shown in Figure 5, where A refers to the adjacency matrix of the graph,

Refers to the transpose of the matrix, including two mixed-hop propagation layers, MLP refers to Multilayer Perceptron.

Among them, the formula of message passing is defined as follows:

是指的信息传播的层数，

是指前一层输出，即当前层要输入的隐藏状态，

是一个超参数，选择不同大小的值就可以控制根节点原始状态保留下来的比例，变量

的定义如下：in

refers to the number of layers of information dissemination,

refers to the output of the previous layer, that is, the hidden state to be input by the current layer,

It is a hyperparameter. By choosing values of different sizes, you can control the proportion of the original state of the root node retained. Variables

is defined as follows:

指的是增广矩阵，I是单位矩阵；where A refers to the adjacency matrix of the graph, D is the degree matrix of the graph,

refers to the augmented matrix, I is the identity matrix;

代表当前层要输出的隐藏状态，

是指用来做特征选择的参数矩阵，G指的是当前信息传播的总层数，The formula definition for message selection is as follows,

represents the hidden state to be output by the current layer,

refers to the parameter matrix used for feature selection, G refers to the total number of layers of current information dissemination,

做m次拉普拉斯平滑，相当于叠加m层图卷积网络，这个图信号将最终收敛到同一个值，节点本身的信息则全部丢失。Figure 5 shows the information propagation step and the information selection step in the hybrid hop propagation layer, which firstly propagates the information horizontally and then selects the information vertically. In graph convolutional networks, Laplacian smoothing can cause over-smoothing problems in graph convolutional networks. Assuming that a graph is connected and not a bipartite graph, for a graph signal above it

Doing m times of Laplacian smoothing is equivalent to stacking m layers of graph convolutional networks. The graph signal will eventually converge to the same value, and all the information of the node itself will be lost.

就是发挥这样的作用，这样传播的节点状态既可以保持局部性，又可以保证全局性。但如果只使用公式（14）进行消息选择，一些节点信息就会丢失。假设某个节点不存在与其他节点之间的空间依赖关系，把其邻居节点的信息与当前节点信息进行聚合，这不仅没起到信息融合的作用，反而给当前节点增加了与本身毫不相关的噪音。因此，引入公式（15）信息选择步骤来关注每一次传播时重要的信息，假设当给定的图结构不包含空间依赖性时，可以把所有

的

调整为0来保留原始节点自身的信息，这样就避免的信息的无用传播。In order to solve this problem, it is necessary to retain a certain proportion of the original state of the node during the propagation process, the equation (13) in

It is to play such a role, so that the propagated node state can maintain both locality and globality. But if only Equation (14) is used for message selection, some node information will be lost. Assuming that a node does not have a spatial dependency relationship with other nodes, the information of its neighbor nodes is aggregated with the current node information, which not only does not play the role of information fusion, but adds to the current node that has nothing to do with itself. noise. Therefore, the information selection step of Equation (15) is introduced to focus on the important information at each propagation, assuming that when the given graph structure does not contain spatial dependencies, all

of

Adjusted to 0 to retain the original node's own information, so as to avoid useless propagation of information.

The interdependence between nodes in the graph is extracted through the graph convolution module, that is, the relationship between dimensions in multi-dimensional time series data. The output of this module will be sent to the next layer of network as the input of the time convolution module. .

The method of the present invention can be used for prediction of various events, such as equipment operation state prediction, industrial equipment failure prediction, power load prediction and the like. Different from the previous multi-dimensional time series prediction methods, the present invention proposes to use an improved graph convolution network and introduce a self-attention mechanism for more accurate multi-dimensional time series prediction. The invention processes multi-dimensional time series data into graph data, the nodes in the graph represent dimensions, and the presence or absence of edges between nodes represents the presence or absence of the influence relationship between dimensions, so that the graph convolution network can be used to directly process the graph after processing. data for modeling. This not only considers the serial correlation before and after the original time of the time series data, but also considers the dependency between dimensions, which improves the prediction accuracy of the model. The invention also introduces a self-attention mechanism to learn the internal relationship of the time series features, so that the model focuses on the key time step that has a greater impact on the prediction result, reduces the influence of irrelevant features, and simultaneously improves the accuracy of the prediction result.

In addition, in addition to the differences in the prediction model itself, different processing methods are also used in data preprocessing. For the processing of missing data, methods such as fitting or simple interpolation are generally used to fill in missing values. These filling methods are too simple and do not consider the spatiotemporal dependence of multidimensional time series data. Therefore, the data preprocessing of this prediction method proposes a time-series missing data completion model that considers spatiotemporal features to establish a data foundation for subsequent time-series prediction.

Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the above examples. Those of ordinary skill in the art, within the scope of the present invention, make changes, modifications, additions or substitutions. It belongs to the protection scope of the present invention.

Claims (6)

1. a multi-dimensional time series prediction method based on self-attention mechanism and graph convolutional network, is characterized in that, comprises the following steps: Step 1. Preprocess the multi-dimensional time series data recorded by each sensor in the industrial production scenario: complete the missing data in time series; Step 2. Data set division: divide the completed time series data into training set, validation set and test set; Step 3. Construction and training of multi-dimensional time series prediction model: Use the training set obtained in step 2 to train the pre-built multi-dimensional time series prediction model based on the self-attention mechanism and graph convolution network, and use the verification set to verify the training effect and adjust the network parameters. When the multi-dimensional time series prediction model is trained to meet the preset convergence conditions, use the test set to test the model prediction effect, and take the model with the best test effect as the final multi-dimensional time series prediction model; Among them, the neural network of each layer includes an input module, an output module, a graph learning module, a graph convolution module, a temporal convolution module and a self-attention mechanism. The graph structure does not need to be defined in advance. The data adaptive learning graph structure automatically extracts the adjacency matrix of the graph according to different data sources. This adjacency matrix will be input to the graph convolution module in each layer to capture the hidden relationship between the dimensions of the time series; the graph The convolution module uses convolution to extract features to capture the spatial dependencies of multi-dimensional time series data; the temporal convolution module captures the temporal dependencies associated with the multi-dimensional time series data, and the self-attention mechanism learns different time series features after the time convolution module. The local correlation between the adjacent time steps and the time-series characteristics of the interaction and influence degree between adjacent time steps are obtained; In order to avoid the problem of gradient disappearance, a residual connection is added to the output of the previous layer of network and the output of the graph convolution module. Add skip connection; the output module converts the input channel dimension into the desired output channel dimension; Step 4: Input the historical data of the industrial time series to be predicted into the multi-dimensional time series prediction model obtained in step 3, and predict the change trend of the time series data, thereby predicting the equipment operation status and abnormal risk. 2. The multi-dimensional time series prediction method based on self-attention mechanism and graph convolutional network according to claim 1, is characterized in that, in the process of learning graph structure, described graph learning module can not only train the model in the training set. Stable and interpretable node relationships are learned, and after the model training is completed, the graph adjacency matrices learned on the test set and other real data sets can be adaptively changed, and the model parameters of the graph learning module will continue to be based on new models. The input data is updated to learn the corresponding graph structure. 3. The multi-dimensional time series prediction method based on self-attention mechanism and graph convolutional network according to claim 2, is characterized in that, when described graph learning module learns graph structure, first adopts a kind of method of random sampling, in In each iteration, the nodes are randomly divided into several groups, and the subgraph structure is learned based on the sampling nodes, so that each node in the graph may be in a group with any other node except itself, so that it can be calculated and updated. Pairwise similarity scores between one node and the rest of the nodes; secondly, the graph learning module learns to extract one-way relationships

,

, the formula is as follows:

in,

,

represents randomly initialized two node embeddings,

,

are the one-way relationship between two randomly initialized nodes learned and extracted by the graph learning module,

,

is the model training parameter, tanh is the activation function, which can be learned in iterative training,

is a hyperparameter used to control the saturation state of the activation function; in order to make the obtained graph adjacency matrix A asymmetric, the operation shown in formula (3) is adopted, where the

The subtraction operation and the ReLU activation function are used to regularize the adjacency matrix,

In order to reduce the computational cost of the graph convolution module, formula (4) and formula (5) are used to make the adjacency matrix of the graph more sparse, and at the same time for each node id , its closest topk nodes are selected as its neighbor nodes idx ; while retaining the weight of connected nodes, set the weight of nodes without connectivity to zero, where the argtopk () function is used to calculate the largest topk values in the vector,

. 4. The multi-dimensional time series prediction method based on a self-attention mechanism and a graph convolutional network according to claim 1, wherein the temporal convolution module comprises two extended initial layers, which are respectively composed of a plurality of one-dimensional volumes. However, the two extended initial layers are not exactly the same, one is followed by a tanh activation function, which works as a filter, and the other is followed by a sigmoid activation function, which acts as a gate to control the transmission to the next The amount of information in the module; the expanded initial layer contains a variety of filters of different sizes, and uses expanded convolution to operate on long-term sequences, for a one-dimensional convolution with a convolution kernel size b and the number f , the size of the receptive field of its convolution is shown in the formula:

In this temporal convolution module, dilated convolution is used to expand the receptive field, and the expansion factor of each layer of the network is set to h , h > 1, and the size of the receptive field of the network increases exponentially with the increase of the hidden layer. The receptive field is expressed as:

. 5. The multi-dimensional time series prediction method based on self-attention mechanism and graph convolutional network according to claim 1, is characterized in that, described self-attention mechanism is after the time convolution module, to the feature information extracted by time convolution Perform analysis to learn its internal connections, assign weights to the time series features of different time steps, and obtain the time series features of the interaction and influence degree between adjacent time steps, as follows: The structure of the self-attention mechanism consists of three matrices, namely the query vector Q , the key vector K and the value vector V , where K is the representation of the feature vector, Q is task-related, and V aggregates feature information into features according to the attention matrix On the vector; the calculation formula of the correlation matrix in the attention mechanism is defined as follows:

in,

and

refers to the eigenvectors of the time series,

and

respectively refer to the feature dimension of

and

, a time series data matrix whose time period length is L ,

Refers

normalized value,

,

and

are all learnable weight parameters, then the formula for calculating the attention matrix E is defined as:

in

refers to the matrix

transpose of ,

It refers to the time series data matrix with both feature dimension and time period length L. Here, the Softmax function is used to normalize the time attention matrix by column, and the sum of the probabilities of each column in the obtained matrix is 1. element at row i and column j

represents a time step

pair time step

The degree of influence of

time step, then

The characteristic information of the time step can capture the dynamic influence of other time steps; Therefore, the feature information of different time steps is aggregated to any time step and quantified to obtain the time series feature vector of the interaction and influence degree between adjacent time steps. The calculation formula is defined as:

where E refers to the calculated attention matrix, U refers to the total feature dimension,

It refers to the time series data matrix with feature dimension U and time period length L. 6. The multi-dimensional time series prediction method based on self-attention mechanism and graph convolution network according to claim 1, is characterized in that, described graph convolution module is aimed at a node in the graph, the information of this node itself and its The information of all neighbor nodes is exchanged and aggregated, and the information of neighbor nodes is used to update the state of its own node, and the same operation is performed on all nodes in the graph in a recursive manner, thereby capturing the spatial dependencies in the graph structure. The graph volume The product module includes two mixed-hop propagation layers, which are divided into three steps: message passing, message selection, and state update, where the formula for message passing is defined as follows:

in

refers to the number of layers of information dissemination,

refers to the output of the previous layer, that is, the hidden state to be input by the current layer,

It is a hyperparameter. By choosing values of different sizes, you can control the proportion of the original state of the root node retained. Variables

is defined as follows:

where A refers to the adjacency matrix of the graph, D is the degree matrix of the graph,

refers to the augmented matrix, I is the identity matrix; The formula definition for message selection is as follows,

represents the hidden state to be output by the current layer,

refers to the parameter matrix used for feature selection, G refers to the total number of layers of current information dissemination,

The interdependence between nodes in the graph is extracted through the graph convolution module, that is, the relationship between dimensions in multi-dimensional time series data. The output of this module will be sent to the next layer of network as the input of the time convolution module. .

Images