CN116842398B - A topic-aware information forwarding prediction method and system based on shielded self-attention network - Google Patents
A topic-aware information forwarding prediction method and system based on shielded self-attention network Download PDFInfo
- Publication number
- CN116842398B CN116842398B CN202310765614.8A CN202310765614A CN116842398B CN 116842398 B CN116842398 B CN 116842398B CN 202310765614 A CN202310765614 A CN 202310765614A CN 116842398 B CN116842398 B CN 116842398B
- Authority
- CN
- China
- Prior art keywords
- user
- topic
- representation
- attention
- context
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0499—Feedforward networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种基于屏蔽自注意力网络的话题感知信息转发预测方法及系统，涉及信息转发预测技术领域。本发明的技术要点包括：提取消息中的话题表示，将话题表示与用户嵌入进行融合表示，获得用户‑话题依赖表示；构建包含上下文编码器、掩码生成器、掩码编码器的屏蔽自注意力网络，将用户‑话题依赖表示输入屏蔽自注意力网络，获得上下文用户依赖感知的用户表示；利用注意力机制计算目标用户与历史感染用户列表的相似度，以对用户表示进行优化重建；通过计算对应优化重建后用户表示的级联表示与下一个用户之间的似然关系来获取下一个用户的激活概率。本发明可为舆情研判、个性化推荐、热点话题识别等应用提供有价值的决策辅助支撑。
The present invention discloses a topic-aware information forwarding prediction method and system based on a shielded self-attention network, and relates to the technical field of information forwarding prediction. The technical points of the present invention include: extracting topic representations in messages, fusing topic representations with user embeddings to obtain user-topic dependency representations; constructing a shielded self-attention network including a context encoder, a mask generator, and a mask encoder, inputting the user-topic dependency representation into the shielded self-attention network, and obtaining contextual user dependency-aware user representations; using an attention mechanism to calculate the similarity between a target user and a list of historically infected users to optimize and reconstruct the user representation; and obtaining the activation probability of the next user by calculating the likelihood relationship between the cascade representation of the corresponding optimized and reconstructed user representation and the next user. The present invention can provide valuable decision-making support for applications such as public opinion analysis, personalized recommendations, and hot topic identification.
Description
Technical Field
The invention relates to the technical field of information forwarding prediction, in particular to a topic perception information forwarding prediction method and system based on a shielding self-attention network.
Background
With the enhancement of online social media (such as microblog, netbook, facebook, etc.) functions and the increasing growth of mass users, large-scale digital information is produced explosively. The user is placed in the online information and participates in the information transmission and diffusion. When one piece of information is forwarded and propagated one by one among the users, an information cascading phenomenon is formed, namely, the information propagation behavior of the current user can drive other users to generate the same or further behavior. Wherein the node users participating in the information cascade are called "infected users". The next infected user is predicted based on the currently known information cascade environment, i.e. for the current message, it is predicted whether the target user will perform a forwarding operation. Predicting information forwarding behavior in a social network can effectively provide convenience for a plurality of services, such as community detection, epidemic prevention, advertisement delivery, product recommendation and the like.
Information propagation forwarding prediction is generally classified into three categories from the study method: (1) Model-based methods, which generally assume that information propagates in a static network and assign a certain propagation probability or threshold to each user, have strong assumability and limited model prediction performance. (2) Based on the propagation path method, a diffusion sequence is established in order of time stamp precedence relationship, and the influence among people is explained through the observed diffusion sequence. Such methods model local information of the entire propagation cascade. (3) The method based on the network topology graph describes information cascading by the topology graph, nodes represent participating users, and edges represent the relationship among the users. Interactions between users are typically modeled internally by means of a graph neural network. Such methods model global information of the entire propagation cascade.
The existing approach assumes that information is spread evenly among users, ignoring the user's preference issues under different trending topics. In reality, however, the user will not point to topics that are not of interest at all, and will not participate in forwarding and discussion. In addition, for a propagation path of a message, users on the path are arranged in order of activation time, and the existing method only considers the influence from a direct forwarder on the path for a target user, so that all users on the whole path are ignored to have a certain influence on the direct forwarder.
Disclosure of Invention
Therefore, the invention provides a topic perception information forwarding prediction method and a topic perception information forwarding prediction system based on a shielding self-attention network, aiming at predicting user forwarding behaviors in a social network by fusing hot topics in information propagation, user propagation context perception and user propagation history perception.
According to an aspect of the present invention, there is provided a topic perception information forwarding prediction method based on a masked self-attention network, the method including the steps of:
Step one, modeling user-topic dependence, comprising: extracting topic representations in the message, and fusing the topic representations with user embedding to obtain user-topic dependent representations;
Step two, user-context dependent modeling, comprising: constructing a masked self-attention network comprising a context encoder, a mask generator, and a mask encoder, inputting the user-topic dependent representation into the masked self-attention network, obtaining a user representation of context user dependent perception;
step three, modeling the user-propagation history dependency, which comprises the following steps: calculating the similarity between the target user and the historical infected user list by using an attention mechanism so as to optimize and reconstruct the user representation of the context user dependent perception;
Step four, predicting the next user activation state, which comprises the following steps: acquiring the activation probability of the next user by calculating a likelihood relation between the cascade representation C i corresponding to the optimized and reconstructed user representation and the next user; wherein the cascade representation C i＝{(u1,t1),(u2,t2),…,(um,tm) represents the observed affected user sequence.
Further, the topic in step one is denoted as Dt i, and the user is embedded asUser-topic dependency representationThe method is calculated according to the following formula:
In the method, in the process of the invention, Representing a weighted representation of a user under an h-th topic, the weighted representation embedded by computing the userSimilarity to topic representation Dt i.
Further, the context encoder in step two is configured to encode the user-topic dependent representation, which includes a stack of multiple identical layers, each layer consisting of a fully connected feed forward network and a multi-headed self-care layer; the multi-head attention layer is obtained by splicing a plurality of point power-up attention mechanisms passing through a linear space; the mask generator is used for generating a mask matrix according to the context user list output by the context encoder and the target user; the mask encoder is configured to multiply each layer of input according to the mask matrix to generate a context-dependent perceived user representation.
Further, in the second step, the generation mask matrix according to the context user list and the target user output by the context encoder is expressed as follows:
Wherein σ represents a sigmoid activation function; w 1 represents a trainable weight matrix; The user list except the user u j is represented and consists of a predecessor user and a successor user of the user u j; d represents the dimension of the vector representation.
Further, the specific steps of the third step include: calculating the attention score of the context user dependence perception user corresponding to the target user and the historical infected user; obtaining the attention weight of the two according to the attention score calculationIntroducing a time decay factor, dividing the continuous time into discrete time intervals, and assigning a learnable weight to each time intervalWeighting the attention weightAnd the learnable weightSumming to obtain historical infected user weightsObtaining the optimized reconstructed user representation according to the historical infection user weight as follows:
In the method, in the process of the invention, Representing contextual user-dependent perceived user representations; Representing that the contextual user infects the user in dependence on perceived history.
Further, the contextual user corresponding to the target user relies on perceived user representations and attention scores of historically infected usersThe calculation formula of (2) is as follows:
In the method, in the process of the invention, User representations that respectively represent contextual user-dependent awarenessHistorical infected users with context user dependent awarenessIs a non-linear mapping of (2);
the calculation formula for obtaining the attention weights of the two according to the attention score calculation is as follows:
In the method, in the process of the invention, Representing the attention score of user u j corresponding to any of the historically infected users u 1,u2…,uj-1, n representing the count used to traverse.
Further, the specific steps of the fourth step include:
Using a location feed forward network, using a ReLU as an activation function, a subject cascade representation is obtained And generating a multi-topic cascade representationThe final goal is to calculate the next userThe probability of forwarding message m i is obtained by maximizing the similarity training of the user and the cascade embedding:
Where Θ represents a set of parameters that can be learned, the target is further translated into a negative log likelihood for the above formula minimization formula:
The motivation of a user by a particular topic is considered in the training of the objective function, thus maximizing the similarity between user embedding and topic embedding:
Thus, the additional training objective translates into the above-described minimization likelihood problem:
the overall goal of model training is:
where ζ is the equilibrium coefficient of the model.
According to another aspect of the present invention, there is provided a topic awareness information forwarding prediction system based on a masked self-attention network, the system comprising:
The user-topic dependence modeling module is configured to extract topic representations in the message, and fuse the topic representations with user embedments to obtain user-topic dependence representations;
A user-context dependent modeling module configured to construct a masked self-attention network comprising a context encoder, a mask generator, a mask encoder, input the user-topic dependent representation into the masked self-attention network, obtain a contextual user-dependent perceived user representation;
A user-propagation history dependency modeling module configured to calculate a similarity of a target user to a list of historically infected users using an attention mechanism to optimally reconstruct a representation of users that are perceived as contextual users;
an activation state prediction module configured to obtain an activation probability for a next user by calculating a likelihood relationship between a cascade representation C i corresponding to the optimized reconstructed user representation and the next user; wherein the cascade representation C i＝{(u1,t1),(u2,t2),…,(um,tm) represents the observed affected user sequence.
Further, the topic in the user-topic dependency modeling module is denoted as Dt i, and the user is embedded asUser-topic dependency representationThe method is calculated according to the following formula:
In the method, in the process of the invention, Representing a weighted representation of a user under an h-th topic, the weighted representation embedded by computing the userSimilarity with topic representation Dt i;
The context encoder in the user-context dependent modeling module is configured to encode the user-topic dependent representation, comprising a stack of multiple identical layers, each layer consisting of a fully connected feed forward network and a multi-headed self-care layer; the multi-head attention layer is obtained by splicing a plurality of point power-up attention mechanisms passing through a linear space; the mask generator is used for generating a mask matrix according to the context user list output by the context encoder and the target user; the mask encoder is configured to multiply each layer of input according to the mask matrix to generate a context-dependent perceived user representation.
Further, the specific steps of the user-propagation history dependency modeling module for performing an optimal reconstruction of the contextual user-dependent perceived user representation include: calculating the attention score of the context user dependence perception user corresponding to the target user and the historical infected user; obtaining the attention weight of the two according to the attention score calculationIntroducing a time decay factor, dividing the continuous time into discrete time intervals, and assigning a learnable weight to each time intervalWeighting the attention weightAnd the learnable weightSumming to obtain historical infected user weightsObtaining the optimized reconstructed user representation according to the historical infection user weight as follows:
In the method, in the process of the invention, Representing contextual user-dependent perceived user representations; Representing that the contextual user infects the user in dependence on perceived history.
The beneficial technical effects of the invention are as follows:
The invention provides a topic perception information forwarding prediction method and a topic perception information forwarding prediction system based on a shielding self-attention network, which comprise an end-to-end topic perception forwarding prediction model based on the shielding self-attention network, and are used for fully expressing and establishing a cascade structure in information propagation and diffusion so as to solve the problem that whether a given target information prediction target user can execute forwarding operation or not. The invention uses BERT to encode hot topic semantic information in the current propagation information, obtains the similarity of topic text embedding and target user embedding to capture user-topic perception dependence; a masked self-attention network is designed to extract the effects of predecessor node users and successor node users while emphasizing the importance of historically infected users, which are modeled to reconstruct and enhance the final user embedded characterization.
Experiments are carried out on three real application scenes, and the model performance is compared with the existing model, so that the invention is obviously superior to the existing information forwarding behavior prediction model in two indexes of MRR and A@k. The invention can provide valuable decision-making auxiliary support for public opinion research, personalized recommendation, hot topic identification and other applications.
Drawings
The invention may be better understood by reference to the following description taken in conjunction with the accompanying drawings, which are included to provide a further illustration of the preferred embodiments of the invention and to explain the principles and advantages of the invention, together with the detailed description below.
FIG. 1 is a block diagram of a topic awareness information forwarding prediction method based on a masked self-attention network according to an embodiment of the present invention;
FIG. 2 is a graph showing the impact of the number of stack layers and the number of stack layers on the user-propagation history based on a model dataset on the performance of a model in an embodiment of the present invention; wherein (a) the corresponding user-context dependency represents the number of stacked layers; (b) the number of corresponding user-propagation history dependent stack layers;
FIG. 3 is a diagram of the effect of user-context dependent representation of the number of stacked layers and user-propagation history dependent number of stacked layers on model performance based on DBLP datasets in an embodiment of the invention; wherein (a) the corresponding user-context dependency represents the number of stacked layers; (b) the number of corresponding user-propagation history dependent stack layers;
FIG. 4 is a diagram of the impact of user-context dependent representation of the number of stacked layers and user-propagation history dependent number of stacked layers on model performance based on Douban datasets in an embodiment of the invention; wherein (a) the corresponding user-context dependency represents the number of stacked layers; (b) the number of stacking layers is dependent on the corresponding user-propagation history.
Detailed Description
In order that those skilled in the art will better understand the present invention, exemplary embodiments or examples of the present invention will be described below with reference to the accompanying drawings. It is apparent that the described embodiments or examples are only implementations or examples of a part of the invention, not all. All other embodiments or examples, which may be made by one of ordinary skill in the art without undue burden, are intended to be within the scope of the present invention based on the embodiments or examples herein.
Formalization of the information forwarding behavior prediction problem is defined as: assuming that the ith message m i in the online social platform is spread at time t 0, the formed cascade is c i＝{(u1,t1),(u2,t2),…,(um,tm), and the text content, namely the topic, is expressed as Dt i. Where the doublet (u j,tj) represents the propagation that user u j is activated to participate in the cascade at time t j. The size of the cascade is |c i |=m, representing the total number of users participating in this piece of information propagation. The task of information propagation forwarding behavior prediction is: knowing the user list U, the concatenation information c i, for each concatenation c i, the next affected user U m+1 is predicted by a conditional probability p (U m+1|ci) from the observed affected user sequence { { (U 1,t1),(u2,t2)…(um,tm) }, wherein U i epsilon U. The invention provides an end-to-end framework based on a neural network, which predicts the next infected user U m+1 through given c i, U and Dt i, and the main idea is to integrate topic information, contextual environment user information and historical infected user information deep modeling information cascade embedded representation, The similarity between the user embedding and the cascade embedding is used for predicting whether the next user is activated.
The embodiment of the invention provides a topic perception information forwarding prediction method based on a shielding self-attention network, which mainly comprises four parts as shown in fig. 1: (1) user-topic dependency modeling; (2) user-context dependent modeling; (3) user-propagation history dependency modeling; (4) next user activation state prediction.
(1) User-topic dependency modeling: extracting topic representations in message m i Using topic representation Dt i with user embeddingFusion representation to get user-topic dependency representationTo perceive the user's dependence on topics in the currently propagating text.
According to the embodiment of the invention, topic semantic messages in the current propagation message are added for acquiring the dependence between the current user and the specific topic. First, the currently propagated text information is modeled using BERT, obtaining a topic representation Dt i. For user embedding in cascade c i For the h topic, by calculating the userSimilarity with a specific topic Dt i to perceive the dependency relationship between the two, and the weight of the user under the h topic is expressed as follows:
The user representation is pooled with the weighting addition to obtain a user-topic dependency representation
(2) User-context dependent modeling: providing a masked self-attention network of context encoder, mask generator, mask encoder to generate a user representation of context user dependent awarenessTo capture the dependency between the user and the predecessor and successor activation nodes in the cascade.
Given a message m i, concatenated to c i＝{(u1,t1),(u2,t2),…,(um,tm), user u j forms a propagation path with its predecessor { u 1,u2,…,uj-1 } and successor { u j+1,uj+2,…,um }, in accordance with an embodiment of the present invention. When user u j browses his own homepage, the content published by the following user of the predecessor user appears, and the content contains the current hot topics or the preferences of users. These contents will have an impact on the decision of user u j on whether or not message m i is forwarded. On the other hand, user u j forwards message m i to other people, including his subsequent users, to consider the tastes and preferences of catering to these users. Thus, a masked self-attention network is proposed to capture the interactions between user u j and its predecessor and successor users. The shielded self-attention network consists of three parts: 1) a context encoder, 2) a mask generator, 3) a mask encoder.
1) Context encoder: a stack of several identical layers, each layer consisting of a fully connected feed forward network and a multi-headed self-care layer. The multi-head attention layer is formed by splicing H point power-up attention mechanisms passing through a linear space:
Wherein the method comprises the steps of D < d' represents a parameter matrix,Representing the variance vector. ReLU is a nonlinear activation function. The context encoder layer i is expressed as:
Wherein the method comprises the steps of Is initialized by the predecessor node and successor node of user u j.
2) Mask generator: the mask is a list of contextual usersAnd target user u j to generate a Mask (Mask) matrix M a, in particular by computingAnd each userWherein sigma represents a sigmoid activation function,Each user representation is derived from the output of the last layer of the generic encoder.
3) Mask encoder: the composition of the mask encoder is the same as that of the generic encoder, except that the inputs of each layer of the mask encoder are multiplied by a mask matrix to perceive trending topics:
finally, a context user dependent perceived user representation is generated:
(3) User-propagation history dependency modeling: the final representation of the node is reconstructed and optimized using an attention mechanism to calculate the similarity relationship of the user to the list of historically infected users. The history infected user list is a list formed from the initial time t 1 to the time t j,u1,u2…,uj-1 of participation of the target user.
In general, not all users in the cascade will have an impact on the target user u j, but only a subset will have an impact on the users, according to an embodiment of the invention. In the flooding sequence, these historical forwarding user lists that have forwarded message m i can influence the perspective of user u j on message m i, and thus whether it is forwarded or not. The correlation between the two is modeled using the attention mechanism. Specifically, the user is first calculatedAnd its precursor nodeIs a fraction of the attention of (2):
Wherein, Respectively usersAnd a precursor nodeIs used for the non-linear mapping of (a). The attention weight between the two is:
In the method, in the process of the invention, The attention score corresponding to any one of the user u j and the history infected user u 1,u2…,uj-1 is represented, and n represents the count used for traversal.
In addition, these historical user-to-userIs attenuated over time, thus introducing a time attenuation factor. For a given user cascade sequenceThe continuous time between t 1 and t m is divided into discrete time intervals:
Assigning a learnable weight to each time interval And further obtaining the total historical infection user weight:
thus, the user after the historical infection user correlation fusion is finally expressed as:
(4) The next user activation state prediction: the activation probability of u m+1 is obtained by calculating the likelihood relationship between the finally obtained user perceived cascade representation C i and the next user u m+1.
According to an embodiment of the invention, a topic cascade representation is obtained using a location feed forward network (Position Feedforward Network, PFN), using a ReLU as an activation functionAnd generating a multi-topic cascade representationThe final goal is to calculate the next userThe probability of forwarding message m i is obtained by maximizing the similarity training of the user and the cascade embedding:
Where Θ represents a set of parameters that can be learned, the target is further translated into a negative log likelihood for the above formula minimization formula:
Furthermore, the motivation of a user by a particular topic is considered in the training of the objective function, thus maximizing the similarity between user embedding and topic embedding:
Thus, the additional training objective translates into the above-described minimization likelihood problem:
the overall goal of model training is:
where ζ is the equilibrium coefficient of the model.
The pseudo code of the topic aware forwarding prediction method based on the masked self-attention network is as shown in algorithm 1:
another embodiment of the present invention provides a topic awareness information forwarding prediction system based on a masked self-attention network, the system comprising:
The user-topic dependence modeling module is configured to extract topic representations in the message, and fuse the topic representations with user embedments to obtain user-topic dependence representations;
A user-context dependent modeling module configured to construct a masked self-attention network comprising a context encoder, a mask generator, a mask encoder, input the user-topic dependent representation into the masked self-attention network, obtain a contextual user-dependent perceived user representation;
A user-propagation history dependency modeling module configured to calculate a similarity of a target user to a list of historically infected users using an attention mechanism to optimally reconstruct a representation of users that are perceived as contextual users;
an activation state prediction module configured to obtain an activation probability for a next user by calculating a likelihood relationship between a cascade representation C i corresponding to the optimized reconstructed user representation and the next user; wherein the cascade representation C i＝{(u1,t1),(u2,t2),…,(um,tm) represents the observed affected user sequence.
In this embodiment, preferably, the topic in the user-topic dependency modeling module is represented as Dt i, and the user is embedded asUser-topic dependency representationThe method is calculated according to the following formula:
In the method, in the process of the invention, Representing a weighted representation of a user under an h-th topic, the weighted representation embedded by computing the userSimilarity with topic representation Dt i;
In this embodiment, preferably, the context encoder in the user-context dependent modeling module is configured to encode the user-topic dependent representation, and includes a stack of multiple identical layers, each layer being composed of a fully connected feed-forward network and a multi-headed self-care layer; the multi-head attention layer is obtained by splicing a plurality of point power-up attention mechanisms passing through a linear space; the mask generator is used for generating a mask matrix according to the context user list output by the context encoder and the target user; the mask encoder is configured to multiply each layer of input according to the mask matrix to generate a context-dependent perceived user representation.
In this embodiment, preferably, the specific step of performing, in the user-propagation history dependency modeling module, optimal reconstruction of the context user-dependent perceived user representation includes: calculating the attention score of the context user dependence perception user corresponding to the target user and the historical infected user; obtaining the attention weight of the two according to the attention score calculationIntroducing a time decay factor, dividing the continuous time into discrete time intervals, and assigning a learnable weight to each time intervalWeighting the attention weightAnd the learnable weightSumming to obtain historical infected user weightsObtaining the optimized reconstructed user representation according to the historical infection user weight as follows:
In the method, in the process of the invention, Representing contextual user-dependent perceived user representations; Representing that the contextual user infects the user in dependence on perceived history.
In this embodiment, preferably, the specific process of obtaining the activation probability of the next user in the activation state prediction module includes: using a location feed forward network, using a ReLU as an activation function, a subject cascade representation is obtainedAnd generating a multi-topic cascade representationThe final goal is to calculate the next userThe probability of forwarding message m i is obtained by maximizing the similarity training of the user and the cascade embedding:
Where Θ represents a set of parameters that can be learned, the target is further translated into a negative log likelihood for the above formula minimization formula:
The motivation of a user by a particular topic is considered in the training of the objective function, thus maximizing the similarity between user embedding and topic embedding:
Thus, the additional training objective translates into the above-described minimization likelihood problem:
the overall goal of model training is:
where ζ is the equilibrium coefficient of the model.
The function of the topic sensing information forwarding prediction system based on the masked self-attention network according to the embodiment of the present invention may be described by the topic sensing information forwarding prediction method based on the masked self-attention network, so that the system embodiment is not described in detail, and reference may be made to the above method embodiment, which is not described herein.
Further experiments prove the technical effect of the invention.
For model training, the batch size is 32, the learning rate is 0.001, and the user embedding dimension is 60. The topic number h is set to 5, and the specific topic embedding dimension is d=60/h=12. For the user-context aware modulo module, the context encoder and mask encoder stack numbers are both set to 6. For the user-history infection dependency module, 6 layers are also stacked. All experiments were performed on a single NVIDIAGeForce GTX 3080Ti graphic card. To verify the versatility and effectiveness of the model, experiments were performed in three different application scenarios, model factors (MEMETRACKER), DBLP, and Douban, respectively. The statistics of the dataset are summarized in table 1.
Table 1 data set statistics
MEMETRACKER: popular web popularity and phrases are used as model factors (Meme), including 9600 tens of thousands of model factors on news and web blogs. Each meme is considered a piece of information and the URL of a blog or news is considered an active user. And acquiring the user and time information thereof. The URLs with the former 5000 module numbers are picked out. The concatenation sequence is obtained from all the modular genes of the 5000 URLs and the concatenation sequence with the length smaller than 3 is eliminated. 80% of the modular genes are classified as training sets, 10% as validation sets and the remainder as test sets.
DBLP (DataBase SYSTEMS AND Logic Programming): is a database system of computer English literature. The dataset contains information of the title, author, year, etc. of the paper. The data set can be used for paper influence study, paper influence sorting, paper topic modeling analysis and the like. The method and the device treat authors of documents as nodes in cascade propagation sequences, treat one citation of papers as one activation to establish cascade sequences, and extract topics by taking titles of the papers as text information of the cascade sequences. The invention firstly builds a directed graph taking the user as a node, and if the paper of the user A refers to the paper of the user B, a directed edge from B to A is formed. And randomly extracting a cascade sequence with the length of 3 or more from the cascade graph to construct a data set. As with MEMETRACKER dataset, 80 was taken as training set, 10% as validation set and 10% as test set.
Douban: is a platform for users to share their personal experiences with accessed movies, books, music, etc. The bean cotyledon user can pay attention to others, and can be paid attention to by others, so that the cascade scale can be increased. Taking a movie as an example, a movie is regarded as one type of information. When the user joins the movie, he/she is activated in the movie cascade. Similar to the two data sets described above, 80% of the bean data was used as the training set, and the remainder was split equally to the test set and the validation set.
Experiments and evaluations were performed on three datasets, MEMETRACKER, DBLP and Douban, of the ICP-TMAN algorithm proposed by the present invention. For information propagation transfer issue to be a predictive problem, the invention uses four indexes of MRR, A@10, A@50 and A@100. Wherein A@k denotes the accuracy of the first k predictors.
Comparing ICP-TMAN with the following classical information transmission forwarding prediction methods, the ICP-TMAN is mainly distributed in three types of prediction methods: (1) a propagation model-based method, (2) a propagation path-based method, and (3) a propagation topology-based method. Specifically, the following is described.
Propagation model-based method: netRate is a method based on an independent cascading model that uses a propagation model of a continuous timing structure to predict the edges between two nodes and predict the dynamics of the underlying network structure on which it depends.
Propagation path-based methods: CYAN-RNN is a cyclic neural network based approach. The model takes into account the time steps of the cascading sequence, using the attention mechanism to model when users will be activated and the dependencies between users. This approach predicts not only who the next infected user is, but also when it will be infected; deepDiffuse is a model based on LSTM, which considers the order of user-specific infections in the information propagation cascade sequence, and uses the attention mechanism to obtain the dependency relationship between these users and the next infected user; DNM is a deep learning technique that uses convolutional neural networks and multi-headed attention mechanisms to obtain the dependencies between users. Using a self-attention mechanism to acquire an active user and predict the next infected user; TAN is a technique that uses deep learning, taking into account cascading sequence information and specific topic information. The method uses an attention mechanism to fuse the user and topics to acquire the dependency relationship between the user and the topics, so as to predict the next activated user; hiDAN is to design a non-sequential framework to model the cascading tree, use two layers of attention mechanisms to capture dependencies between users without direct correlation, and predict the interdependence of this node in the future.
The method based on the propagation topological graph comprises the following steps: topo-LSTM is a model based on LSTM, and modeling is performed by using Lattice-LSTM in consideration of structural information in a social network, and the situation that cascading sequences are not adjacent but adjacent in the social network structure is considered mainly, so that the next user participating in propagation can be effectively predicted.
Table 2 results of experiments comparing the Baseline method at MEMETRACKER, DBLP and Douban
Table 2 summarizes performance of the ICP-TMAN model with the seven baseline models on three datasets MEMETRACKER, DBLP and Douban. Where sub-optimal results are underlined. As can be seen from the table, the ICP-TMAN of the present invention was significantly higher than the seven baseline models in the performance of the three data sets on MRR, A@10, A@50, A@100. In particular, ICP-TMAN models were 15.98%,14.01,8.87%,5.95% improved over sub-optimal results in MRR, A@10, A@50, A@100, respectively. The ICP-TMAN model fully excavates user-topics, user-propagation context and deep embedded representation of user-infection history, and the fusion of the three effectively improves the performance of information forwarding behavior prediction. Furthermore, it can be seen from the table that model NetRate does not perform well because the traditional independent cascade-based approach relies heavily on static models of the underlying assumptions and, at the same time, gives nodes a probability of activation of the assumptions, with a certain gap from the true propagation structure. The suboptimal model HiDAN considers non-sequence dependency in the propagation cascade sequence, proves that influence exists between two users without direct information transmission, and also laterally verifies the effectiveness of modeling the influence of each preceding node and subsequent nodes on a target node in the invention, so that the performance of information propagation forwarding prediction is greatly improved, but topics are not considered, and a shielding self-attention network is not adopted to sense the dependence between users, so that the prediction performance is inferior to that of an ICP-TMAN model.
To verify the effectiveness of each module in the ICP-TMAN model, the present invention further conducted an ablation experiment.
ICP-TMAN -Inf: and removing text and topic information in the information propagation text to verify the improvement degree of the propagation mode under a specific topic and the traditional uniform propagation mode on the model prediction performance.
ICP-TMAN -Tim: the importance of the timing factor is verified by removing the module that the influence in information propagation is attenuated by the increase of time.
ICP-TMAN -Mas: and modeling the dependency relationship between the target user and the context user is removed, so that the importance degree of the factor of influence of the precursor node and the subsequent node of the setting user is observed.
Table 3 experimental results comparing variant methods on MEMETRACKER, DBLP and Douban datasets
The results of the ablation experiments on the microblog dataset and the APS dataset are shown in table 3, from which it can be seen that ICP-TMAN performed optimally compared to the other variants. The predictive performance is reduced to different extents after the three modules are removed. In particular, when the user and user context dependent representation of the masked attention network module is removed (i.e., ICP-TMAN -Mas), MRR, A@10, A@50, A@100 drop sharply across both MEMETRACKER, DBLP and Douban datasets. Fully described, in the cascade propagation list of modeling information of the model, the indispensable influence of the predecessor user and the successor user of the target user is included, so that the significant contribution of the user-context dependent representation to the model is described. Meanwhile, the topic information module and the time sequence attenuation module (namely ICP-TMAN -Inf and ICP-TMAN -Tim) are removed, ICP-TMAN prediction performance is obviously reduced on three data sets, prediction of next user activation in information cascade is illustrated, and the superiority of information transmission under specific topics in consideration of only uniform transmission in comparison with the traditional method is considered. While also verifying the necessity of taking into account time factors.
A comparison experiment is also performed on the selection of part of the parameters, and the comparison analysis is performed on the user-context dependent representation module stacking layers (Mas-layers) and the user-transmission history dependent stacking layers (His-layers). Fig. 2-4 show the variation of MRR, a@10, a@50, a@100 as the number of stacked layers increases. For MEMATRACKER datasets, as shown in FIG. 2, the four metrics reach their highest values when the number of user context-dependent stack layers, user propagation history-dependent stack layers, is 4 and 5, respectively. For DBLP datasets, as shown in FIG. 3, the predictive performance of the model is best when the number of user context-dependent stack layers and the number of user propagation history-dependent stack layers are both 8. For DouBan datasets, as shown in fig. 4, the ICP-TMAN model forwarding prediction performance is optimal when the number of user context dependent stack layers and the number of user propagation history dependent stack layers are 3 and 5, respectively.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments are contemplated within the scope of the invention as described herein. The disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is defined by the appended claims.
Claims (5)
1. The topic perception information forwarding prediction method based on the shielding self-attention network is characterized by comprising the following steps of:
Step one, modeling user-topic dependence, comprising: extracting topic representations in the message, and fusing the topic representations with user embedding to obtain user-topic dependent representations; the topic is denoted as Dt i, and the user embedding is denoted as User-topic dependency representationThe method is calculated according to the following formula:
In the method, in the process of the invention, Representing a weighted representation of a user under an h-th topic, the weighted representation embedded by computing the userSimilarity with topic representation Dt i;
Step two, user-context dependent modeling, comprising: constructing a masked self-attention network comprising a context encoder, a mask generator, and a mask encoder, inputting the user-topic dependent representation into the masked self-attention network, obtaining a user representation of context user dependent perception; the context encoder is used for encoding the user-topic dependent representation and comprises a stack formed by a plurality of identical layers, wherein each layer is formed by a fully-connected feed-forward network and a multi-head self-attention layer; the multi-head attention layer is obtained by splicing a plurality of point power-up attention mechanisms passing through a linear space; the mask generator is used for generating a mask matrix according to the context user list output by the context encoder and the target user; the mask encoder is used for multiplying each layer of input according to the mask matrix to generate a user representation of context user dependent perception;
Step three, modeling the user-propagation history dependency, which comprises the following steps: calculating the similarity between the target user and the historical infected user list by using an attention mechanism so as to optimize and reconstruct the user representation of the context user dependent perception; the method comprises the following specific steps: calculating the attention score of the context user dependence perception user corresponding to the target user and the historical infected user; obtaining the attention weight of the two according to the attention score calculation Introducing a time decay factor, dividing the continuous time into discrete time intervals, and assigning a learnable weight to each time intervalWeighting the attention weightAnd the learnable weightSumming to obtain historical infected user weightsObtaining the optimized reconstructed user representation according to the historical infection user weight as follows:
In the method, in the process of the invention, Representing contextual user-dependent perceived user representations; representing a history of infected users of the contextual users dependent on awareness;
Step four, predicting the next user activation state, which comprises the following steps: acquiring the activation probability of the next user by calculating a likelihood relation between the cascade representation C i corresponding to the optimized and reconstructed user representation and the next user; wherein the cascade representation C i＝{(u1,t1),(u2,t2),…,(ym,ym) represents the observed affected user sequence.
2. The method of claim 1, wherein generating a mask matrix from the list of contextual users and the target user output by the context encoder in step two is expressed as:
Wherein σ represents a sigmoid activation function; w 1 represents a trainable weight matrix; The user list except the user u j is represented and consists of a predecessor user and a successor user of the user u j; d represents the dimension of the vector representation.
3. The method for forwarding and predicting topic awareness information based on a masked self-attention network of claim 2, wherein the contextual user corresponding to the target user depends on the perceived user representation and the attention score of the historically infected userThe calculation formula of (2) is as follows:
In the method, in the process of the invention, User representations that respectively represent contextual user-dependent awarenessHistorical infected users with context user dependent awarenessIs a non-linear mapping of (2);
the calculation formula for obtaining the attention weights of the two according to the attention score calculation is as follows:
In the method, in the process of the invention, The attention score corresponding to any one of the user u j and the history infected user u 1,u2…,uj-1 is represented, and n represents the count used for traversal.
4. A method for predicting topic awareness information forwarding based on a masked self-attention network as in claim 3 wherein the specific steps of step four include:
Using a location feed forward network, using a ReLU as an activation function, a subject cascade representation is obtained And generating a multi-topic cascade representationThe final goal is to calculate the next userThe probability of forwarding message m i is obtained by maximizing the similarity training of the user and the cascade embedding:
Where Θ represents a set of parameters that can be learned, the target is further translated into a negative log likelihood for the above formula minimization formula:
The motivation of a user by a particular topic is considered in the training of the objective function, thus maximizing the similarity between user embedding and topic embedding:
Thus, the additional training objective translates into the above-described minimization likelihood problem:
the overall goal of model training is:
where ζ is the equilibrium coefficient of the model.
5. A topic awareness information forwarding prediction system based on a masked self-attention network, comprising:
The user-topic dependence modeling module is configured to extract topic representations in the message, and fuse the topic representations with user embedments to obtain user-topic dependence representations; the topics in the user-topic dependency modeling module are denoted as Dt i, and the users are embedded as User-topic dependency representationThe method is calculated according to the following formula:
In the method, in the process of the invention, Representing a weighted representation of a user under an h-th topic, the weighted representation embedded by computing the userSimilarity with topic representation Dt i;
A user-context dependent modeling module configured to construct a masked self-attention network comprising a context encoder, a mask generator, a mask encoder, input the user-topic dependent representation into the masked self-attention network, obtain a contextual user-dependent perceived user representation; the context encoder is configured to encode the user-topic dependent representation, and includes a stack of identical layers, each layer consisting of a fully-connected feed-forward network and a multi-headed self-attention layer; the multi-head attention layer is obtained by splicing a plurality of point power-up attention mechanisms passing through a linear space; the mask generator is used for generating a mask matrix according to the context user list output by the context encoder and the target user; the mask encoder is used for multiplying each layer of input according to the mask matrix to generate a user representation of context user dependent perception;
A user-propagation history dependency modeling module configured to calculate a similarity of a target user to a list of historically infected users using an attention mechanism to optimally reconstruct a representation of users that are perceived as contextual users; the method comprises the following specific steps: calculating the attention score of the context user dependence perception user corresponding to the target user and the historical infected user; obtaining the attention weight of the two according to the attention score calculation Introducing a time decay factor, dividing the continuous time into discrete time intervals, and assigning a learnable weight to each time intervalWeighting the attention weightAnd the learnable weightSumming to obtain historical infected user weightsObtaining the optimized reconstructed user representation according to the historical infection user weight as follows:
In the method, in the process of the invention, Representing contextual user-dependent perceived user representations; representing a history of infected users of the contextual users dependent on awareness;
an activation state prediction module configured to obtain an activation probability for a next user by calculating a likelihood relationship between a cascade representation C i corresponding to the optimized reconstructed user representation and the next user; wherein the cascade representation C i＝{(u1,t1),(u2,t2),…,(um,tm) represents the observed affected user sequence.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310765614.8A CN116842398B (en) | 2023-06-27 | 2023-06-27 | A topic-aware information forwarding prediction method and system based on shielded self-attention network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310765614.8A CN116842398B (en) | 2023-06-27 | 2023-06-27 | A topic-aware information forwarding prediction method and system based on shielded self-attention network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116842398A CN116842398A (en) | 2023-10-03 |
| CN116842398B true CN116842398B (en) | 2024-06-28 |
Family
ID=88169992
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310765614.8A Active CN116842398B (en) | 2023-06-27 | 2023-06-27 | A topic-aware information forwarding prediction method and system based on shielded self-attention network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116842398B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113868474A (en) * | 2021-09-02 | 2021-12-31 | 子亥科技(成都)有限公司 | An information cascade prediction method based on self-attention mechanism and dynamic graph |
| CN114936723A (en) * | 2022-07-21 | 2022-08-23 | 中国电子科技集团公司第三十研究所 | Social network user attribute prediction method and system based on data enhancement |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11586880B2 (en) * | 2018-08-28 | 2023-02-21 | Beijing Jingdong Shangke Information Technology Co., Ltd. | System and method for multi-horizon time series forecasting with dynamic temporal context learning |
| CN113254803B (en) * | 2021-06-24 | 2021-10-22 | 暨南大学 | A social recommendation method based on multi-feature heterogeneous graph neural network |
| CN113688600B (en) * | 2021-09-08 | 2023-07-28 | 北京邮电大学 | Information propagation prediction method based on topic perception attention network |
| CN113850446B (en) * | 2021-10-22 | 2022-08-12 | 重庆理工大学 | Information Diffusion Prediction Method Fusion Spatiotemporal Attention and Heterogeneous Graph Convolutional Networks |
| CN115409155B (en) * | 2022-08-17 | 2026-01-09 | 电子科技大学 | Information Cascade Prediction System and Method Based on Transformer Enhanced Hawkes Process |
| CN115660147B (en) * | 2022-09-26 | 2024-10-15 | 哈尔滨工业大学 | A method and system for predicting information propagation based on influence modeling between and within propagation paths |
| CN116051304B (en) * | 2022-12-19 | 2026-01-02 | 福州大学 | Information propagation prediction system based on cascading relationships |
| CN115878907B (en) * | 2022-12-28 | 2023-08-11 | 人民网股份有限公司 | Method and device for predicting social network forwarding behavior based on user dependency |
| CN115759482A (en) * | 2022-12-29 | 2023-03-07 | 人民网股份有限公司 | Social media content propagation prediction method and device |
-
2023
- 2023-06-27 CN CN202310765614.8A patent/CN116842398B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113868474A (en) * | 2021-09-02 | 2021-12-31 | 子亥科技(成都)有限公司 | An information cascade prediction method based on self-attention mechanism and dynamic graph |
| CN114936723A (en) * | 2022-07-21 | 2022-08-23 | 中国电子科技集团公司第三十研究所 | Social network user attribute prediction method and system based on data enhancement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116842398A (en) | 2023-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lu et al. | RevCore: Review-augmented conversational recommendation | |
| US9535897B2 (en) | Content recommendation system using a neural network language model | |
| Wu et al. | Neural demographic prediction using search query | |
| Fan et al. | Adversarial attacks for black-box recommender systems via copying transferable cross-domain user profiles | |
| WO2018226960A1 (en) | Key-value memory networks | |
| Tsinganos et al. | Utilizing convolutional neural networks and word embeddings for early-stage recognition of persuasion in chat-based social engineering attacks | |
| Yang et al. | Recommender system-based diffusion inferring for open social networks | |
| Chen et al. | Meta-path based neighbors for behavioral target generalization in sequential recommendation | |
| Liao et al. | Coronavirus pandemic analysis through tripartite graph clustering in online social networks | |
| CN117494036A (en) | End-to-end virtual social robot detection method based on multi-mode information fusion | |
| Fazelnia et al. | Variational user modeling with slow and fast features | |
| Dong et al. | Improving sequential recommendation with attribute-augmented graph neural networks | |
| Hashemi et al. | Neural instant search for music and podcast | |
| Yao et al. | Recommender transformers with behavior pathways | |
| Samra et al. | Cross-domain latent factors sharing via implicit matrix factorization | |
| Ting et al. | Applying social network embedding and word embedding for socialbots detection | |
| CN111815028A (en) | Method and device for predicting propagation path of sudden hot spot event | |
| CN116842398B (en) | A topic-aware information forwarding prediction method and system based on shielded self-attention network | |
| Fang et al. | Multimodal recommender system based on multi-channel counterfactual learning networks | |
| Zhu et al. | Feature Interaction Dual Self-attention network for sequential recommendation | |
| Boryczka et al. | Speed up Differential Evolution for ranking of items in recommendation systems | |
| Wu | Advances in collaborative filtering and ranking | |
| Syed | Abstractive summarization of social media posts: a case study using deep learning | |
| Zheng et al. | Routing User-Interest Markov Tree for Scalable Personalized Knowledge-Aware Recommendation | |
| Sangeetha et al. | Predicting personalized recommendations using GNN |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |