CN105608228A - High-efficiency distributed RDF data storage method - Google Patents

Abstract

The invention discloses an efficient distributed RDF data storage method. The method is as follows: 1) the user selects a named graph or sets a new named graph for each triple to be uploaded; and sets an effective predicate and triple for the triple according to business requirements; 2 ) The data control system parses each triple in the RDF data uploaded by the user, extracts the predicate of the triple and the valid predicate of the named graph of the triple; then according to the valid predicate, the triple split into two triples with the same unique identifier: a triple of complete predicates of the same subject and a triple of valid predicates of the same subject; valid predicates are part of the complete predicates; 3) the data control system will get The triplet data of complete predicates of the same subject and the triplet data of valid predicates of the same subject are stored in different database clusters respectively. The invention improves the high availability of data.

Description

An Efficient Distributed RDF Data Storage Method

technical field

The invention relates to the technical field of RDF data storage, in particular to an efficient distributed RDF data storage method, which belongs to the field of computer software.

Background technique

With the rapid development of Internet technology, the application scope of the Internet has become more and more extensive, and a huge knowledge network database has been formed, but it has also brought many challenges. In order to connect different forms of knowledge network databases, so that computers can understand The connection between data and data puts forward the concept of Semantic Web. The goal of the Semantic Web is to make the information resources on the network understandable by machines, so as to realize the automatic processing of network information resources and adapt to the rapid growth of network information resources.

The Semantic Web defines a resource description framework, RDF, to describe information resources on the Web. RDF is a data model of network resource objects and their relationships. It provides a general data model to support the description of network resources. RDF uses triples (subject, predicate and object) to describe various resources on the network and their relationships. relationship between. From the perspective of the graph, the model is composed of nodes and edges between nodes. Nodes represent subjects and objects, and edges represent predicates. In this way, nodes can be used to represent resources, and edges represent resource attributes.

Currently, stand-alone RDF database management systems are commonly used for RDF data storage, such as GraphDB, stardog, and allegrograph. This RDF storage method can manage a large amount of triple data, but with the rapid growth of Internet information resources, the storage capacity of a single machine is limited, which can no longer meet the current demand for massive triple data storage. For the storage of massive triplet data, some scholars have proposed a variety of schemes, but they are all in the research stage. For example, using Hadoop or Hbase distributed clusters to store triplet data, because Hadoop or Hbase both naturally have the ability to store and manage massive data, and use mapreduce to simulate data query; but because of this storage method, the triplet data of the same subject Storage is decentralized, that is, triples of the same subject may be stored on multiple machines; coupled with the complexity of RDF data association relationships, there may be associations between each triplet, and mapreduce simulates and implements data query schemes. When querying data, a large amount of data association screening is required. The current storage solution cannot realize high-speed query of data, and the query performance is relatively low. Especially in the case of a large amount of data, a simple query may take more than ten seconds to execute , which cannot meet the actual business query requirements.

Contents of the invention

Aiming at the problems encountered in the RDF data storage mentioned above, the present invention proposes an efficient distributed RDF data storage method, which solves the problems of limited data storage capacity and scattered triplet data in the existing RDF data storage methods .

In order to solve the above problems, the present invention proposes a highly efficient distributed RDF data storage method, which mainly includes the following implementation steps:

1) The data parser parses the RDF data uploaded by the user, and parses each triplet data into a triplet object in a unified format; processes the parsed data, parses and extracts the predicates in the triplets, and extracts Name the effective predicates of the graph. The effective predicates are defined by the user's business requirements. The user determines the currently used predicate triples according to the specific business requirements, that is, the triples that constitute the effective predicates. According to the effective predicates of the naming graph, the triplet data of the same subject is split into two parts, namely the triplet data of the complete predicate of the same subject and the triplet data of the effective predicate of the same subject; the complete predicate of the same subject The triplet data of the same subject is the complete triplet data of the same subject, the triplet data of effective predicates of the same subject is the triplet data of partial predicates of the same subject, so the triplet data of effective predicates of the same subject Data is a subset of triplet data of full predicates of the same subject. According to the effective predicates of the naming graph, the triplet data of the same subject is split into two parts, that is, the triplet data of the complete predicate of the same subject and the triplet data of the effective predicate of the same subject; and generate a unique ID, The triplet that uniquely identifies the subject, each triplet of the subject will generate the unique ID, which is used to uniquely identify the triplet of the subject, the triplet data of the complete predicate of the same subject and the effective predicate of the same subject Triple data share this unique ID.

2) The data is divided into two parts for storage management, that is, triple data of complete predicates of the same subject and triple data of valid predicates of the same subject are stored separately. Use an open source distributed NoSQL database cluster to store the complete predicate triplet data of the same subject. In order to ensure the integrity of the data, when the predicate requirements change in the future, the effective predicate triplet data will be expanded or reduced. Use RDF database clusters to store the triple data of valid predicates of the same subject. The triple data of valid predicates of the same subject is a subset of the triple data of complete predicates of the same subject. In the case of unchanged storage capacity, improve The storage capacity and management capabilities of triplet data in the system reduce the amount of triplet data and improve the efficiency of data query; the RDF database cluster is composed of data nodes, routing nodes and configuration nodes.

3) Effective predicate triples data in RDF database clusters can be expanded dynamically. The RDF database cluster only stores the triplet data of valid predicates of the same subject, and the valid predicates can be changed dynamically. When the valid predicates change, the user first submits the predicate update task, and the predicate update task monitoring module of the system monitors the predicate update task submitted by the user. When the user submits the predicate update task, the monitoring module starts the predicate update task in the background to detect which predicates have changed, and the triples stored in the RDF database cluster also need to change accordingly. The data management module is responsible for valid predicates. The NoSQL database cluster stores the triple data of the complete predicate of the same subject, and the triple data of the changed predicate is imported into the RDF database cluster to ensure the integrity of the stored triple data.

Further, the basic structure of triples and named graphs (graphname) in RDF data is a collection of multiple triples, each triple consists of a subject, a predicate and an object, and the predicate represents the subject and Each triple can also be understood as consisting of a subject, a predicate and an object. A series of such triples is called an RDF graph. The name of the defined RDF graph is named graph (graphname). The named graph is the space for data storage, which is equivalent to the concept of database in relational database. According to the definition of business requirements, you can select an existing named map, or add a new named map.

Further, for the complete predicate and the effective predicate, the present invention divides the predicates of the triples of the same subject into two parts, that is, the complete predicate and the effective predicate; the complete predicate: all the predicates contained in a named graph, and the effective predicate: the user Customized according to business requirements, that is, the predicates that users will use in a naming diagram; divide the triples of the same subject into two parts according to the predicate information, that is, the triples of complete predicates and the triples of valid predicates .

Further, the triple data of complete predicates of the same subject and the triple data of effective predicates of the same subject are stored and managed separately. Since there are generally multiple triple predicates for the same subject, and most of the predicates are redundant data in actual requirements, they will not be used in existing business requirements, but may be used when future requirements change In order to ensure the integrity of the data, this part of the data cannot be lost, so this mode is used to divide and manage the data, that is, the triple data of the complete predicate and the triple data of the effective predicate are stored separately, and Use a unique ID to associate these two parts of data, use an open source distributed NoSQL database cluster to store triple data of complete predicates, and use an RDF database cluster to store triples of valid predicates.

Further, the RDF database cluster is composed of data nodes, routing nodes and configuration nodes; the data nodes mainly store data and are composed of multiple open source stand-alone RDF databases; the routing (router) nodes control the data nodes, including data update , data node selection, data fragmentation and data synchronization, etc.; the configuration node (config) manages the configuration information of the data node, including the IP and port of each data node, name, naming map, predicate information, storage triplet data volume, Information such as the maximum filling factor and the master-slave library label.

Further, when the data fragmentation and data node selection are used to store triplet data, in order to solve the problem of data dispersion, the triplet data of the same subject is stored in the same data node, and the data of the same named graph is stored in the data node The maximum storage capacity is stored in the same data node, so that the distributed query of data reduces the calculation amount of data query and the data transmission between different nodes, and improves the query speed. When performing data sharding, the triplet data of the same subject is regarded as an atomic data, and the corresponding data node is selected according to the current data storage capacity, storage capacity, maximum filling factor, and graph distribution of each data node, and the triplet is stored data.

Compared with prior art, positive effect of the present invention is:

Aiming at the storage of large-scale RDF data, the present invention proposes a new distributed RDF data storage scheme, divides the data into two parts for storage management, and separates storage modes for triplet data of complete predicates and triplet data of valid predicates . Improve the storage capacity of RDF data, so that it can manage massive RDF data; improve high availability of data, RDF database cluster has data fragmentation and backup data, when a data node fails, it can ensure the normal and uninterrupted operation of the system; The sharding strategy uses the triplet data of the same subject as an atomic data, performs data sharding and data node selection according to the naming graph and subject, reduces the dispersion of triplet data in each data node, and reduces the complexity of data query And the data transmission volume between different nodes, while improving the query efficiency of data.

Description of drawings

The accompanying drawing is a system architecture diagram of an efficient distributed RDF data storage method of the present invention.

detailed description

In order to express the method of the present invention more clearly and intuitively, the present invention will be further described in detail below in conjunction with the accompanying drawings. The efficient distributed RDF data storage method of the present invention comprises the following steps:

1) Data access, responsible for providing a unified data access interface to the outside world, and accessing data through the provided interface. It mainly includes interfaces such as data upload, data update, data query, predicate expansion and predicate information query.

2) Data control, providing data control and processing functions mainly including data management, predicate management and data storage management.

Data management provides RDF data management functions, including RDF data upload, update and query control; RDF upload data control, including RDF data parser, RDF data segmentation module and unique ID generation. When data is uploaded, first, the RDF data parser performs RDF data analysis, supports the analysis of RDF data in various formats, including RDF data in xml, json, and nt formats, and parses the data into a unified format according to the data format uploaded by the user. RDF data object; then, the RDF data segmentation module splits the RDF data object in a unified format generated by parsing, and the user defines the named map name of the RDF data, which is used to determine which named map the uploaded data is saved to. According to the named map of the RDF data Obtain its valid predicate list, divide the data into two parts according to the valid predicate list, that is, the triplet object of the complete predicate of the same subject, and the triplet object of the valid predicate of the same subject; finally, generate a unique ID for unique identification The triplet of the subject, which is associated with the triplet of the complete predicate of the subject and the triplet of the valid predicate of the same subject, is generated using the ID auto-increment strategy, and the unique ID generator is used to obtain the data. Name the self-incrementing ID of the graph, and generate a triple containing the ID to be encapsulated into the triple object of the complete predicate of the same subject and the triple object of the effective predicate of the same subject.

Predicate management provides predicate management functions for RDF data, including functions such as predicate expansion, predicate reduction, and predicate information query. The expansion of predicates refers to the expansion of effective predicates. Since the RDF database cluster only stores the triples of some predicates, when the predicate that the user needs to use a certain named graph is not in the effective predicates, the effective predicates need to be extended to expand these in the database. A triplet of predicates. Predicate expansion step: the user submits the predicate of the named graph to be expanded, and the predicate management module obtains the named graph and its extended predicates submitted by the user, compares the valid predicates in the named graph, and verifies that the predicate needs to be extended. In order to ensure that the existing Valid predicates do not include extended predicates submitted by users, which play the role of user input data verification; predicate expansion tasks are submitted through predicate expansion scheduling, and the predicate expansion tasks are executed asynchronously in the background to import data and read corresponding triples from the NoSQL database Data, extract triples of extended predicates, and import them into RDF database clusters.

Data storage management provides data management module and predicate management module to operate the database. All operations on the database are carried out through this module, providing a unified data access interface, realizing the separation of data processing and data storage, including data query and update of the database and upload functions, as well as predicate extended data import, predicate information query, update and upload functions.

3) Data persistence, responsible for the physical storage of data, save the data to disk, divide the data into two parts for persistence, and use NoSQL database cluster and RDF database cluster for data storage. The NoSQL database cluster uses an open-source distributed NoSQL database cluster, which uses its massive data management capabilities to store triple data of complete predicates to ensure data integrity. When valid predicates change, read their corresponding predicates The triplet data is imported into the RDF database cluster. The RDF database cluster consists of multiple data nodes, routing nodes, and configuration nodes; the data nodes mainly store triplet data, and are composed of multiple stand-alone open source RDF databases; the routing nodes control the data nodes, including data update, data node selection, data sharding and data synchronization, etc. The routing node manages the RDF database cluster and is the central node of the cluster, controlling each RDF database data node. The configuration node manages the configuration information of the data nodes, including the IP and port of each data node, name, naming map, predicate information, storage triplet data volume, maximum filling factor, and master-slave library marking and other information. The fill factor refers to the ratio of the data storage capacity to the maximum data capacity, the maximum fill factor refers to the maximum allowable fill factor value, and the current fill factor refers to the ratio of the current data storage capacity to the maximum data capacity. When uploading triplet data, the routing node obtains the data node where the named graph data is located according to the named graph of the triplet and the configuration information of the configuration node. If the named graph data is not stored in any data node, it means that the named If the graph is a new graph, select a data node with the smallest current filling factor from all data nodes to store the uploaded triplet data; if the named graph is stored in some data nodes, select a certain data node from these data nodes A data node with the smallest current filling factor. If the smallest current filling factor value in the data node is greater than or equal to the maximum filling factor value, the named graph data needs to be stored in fragments, and a filling factor minimum is selected from other data nodes. , store the uploaded triplet data, otherwise, directly select the data node with the smallest filling factor to store the uploaded triplet data. After the data is stored in the data node, update the corresponding configuration information, including updating the naming graph information and the storage triplet data volume of the data node and other configuration information.

Implementation case analysis of data upload:

1. Prepare the triplet data, and define the named graph (graphname) of the triplet data, that is, determine which named graph the data will be uploaded to, and upload the triplet data and its named graph to the data management by calling the data upload interface module.

2. The data management module invokes the data parser, parses the triplet data, and encapsulates the data into triplet data objects in a unified format.

3. The data management module calls the data segmentation module, and queries the effective predicate list of its named graph through the predicate management module. According to the effective predicate list, the uploaded triplet data object is divided into two parts, that is, the complete predicate triplet data object and A valid predicate triplet data object.

4. The data management module uses a unique ID auto-increment generator to generate a unique ID for uploading triplet data, and encapsulates the ID value into a complete predicate triplet data object and a valid predicate triplet data object respectively.

5. Invoke the data storage control module to store the complete predicate triplet data and valid predicate triplet data in the NoSQL database cluster and the RDF database cluster respectively. The complete predicate triple data is directly stored in the NoSQL database cluster. The routing nodes of the RDF database cluster control the storage of valid predicate triple data.

6. The routing node of the RDF database cluster obtains the data node where the named graph is located by calling the configuration node. If the named graph data is not stored in any data node, it means that the named graph is a new graph, and then select one from all data nodes The data node with the smallest filling factor currently stores the uploaded triplet data, and continues to store data according to step 10.

7. If the named graph is stored in some data nodes, select a data node with the smallest current filling factor from these data nodes.

8. If the minimum current fill factor value in the selected data node is greater than or equal to the maximum fill factor value, the named graph data needs to be stored in fragments, and a data node with the smallest current fill factor is selected from other data nodes. Store the uploaded triplet data, and continue to store data according to step 10.

9. If the minimum current filling factor value in the selected data node is smaller than the maximum filling factor value, then directly select the data node with the smallest current filling factor, store the uploaded triplet data, and continue data storage according to step 10.

10. After the data is stored in the data node, update the corresponding configuration information: named map information, the storage triplet data volume of the data node and the current fill factor.

Claims (9)

1. An efficient distributed RDF data storage method, the steps of which are: 1) The user selects a named map or sets a new named map for each triple to be uploaded; and sets an effective predicate and its triple for the triple according to business requirements; 2) The data control system analyzes each triple in the RDF data uploaded by the user, and extracts the predicate of the triple and the valid predicate of the named graph of the triple; then, according to the valid predicate, the triple The group splits into two triples with the same unique identity: a triple of complete predicates of the same subject and a triple of valid predicates of the same subject; where complete predicates are all predicates contained in the named graph of triples , the valid predicate is a part of the complete predicate; 3) The data control system stores the obtained triplet data of complete predicates of the same subject and the triplet data of effective predicates of the same subject into different database clusters respectively. 2. The method according to claim 1, characterized in that, using an open source distributed NoSQL database cluster to store the triple data of the complete predicate of the same subject, and using an RDF database cluster to store the triple data of the effective predicate of the same subject. 3. The method according to claim 2, wherein when the data control system receives the predicate update task, it detects the changed predicate according to the predicate update information in the update task, and then updates the corresponding Predicates in triples. 4. The method according to claim 2 or 3, wherein the RDF database cluster includes data nodes, routing nodes and configuration nodes; wherein, the data nodes are used for data storage; the routing nodes are used for controlling the data nodes , including data update, data node selection, data fragmentation and data synchronization; the configuration node is used to manage the configuration information of the data node, including the IP and port of each data node, name, naming map, predicate information, and storage triplet data quantity, maximum filling factor and master-slave library marking information. 5. The method of claim 4, wherein the data control system stores triplet data of the same subject to the same data node. 6. The method according to claim 5, wherein the data control system stores the data of the same named graph into the same data node within the maximum storage capacity of the data node. 7. The method according to claim 4, wherein the routing node obtains the data node where the data of the naming graph is located according to the configuration information of the naming graph of the triplet and the configuration node; wherein, if the naming graph of the If the data is not stored in any data node, select a data node with the smallest current filling factor from all data nodes to store the uploaded triplet data; Select the data node with the smallest current filling factor in the data node. If the smallest current filling factor value in the data node is greater than or equal to the maximum filling factor value, the data of the named graph will be stored in pieces, and a filling factor will be selected from other data nodes. The node with the smallest filling factor stores the uploaded triplet data; otherwise, select the data node with the smallest filling factor currently to store the uploaded triplet data. 8. The method according to claim 7, wherein after the data node stores a triple, the corresponding configuration information is updated, including named map information, stored triple data volume and current filling factor. 9. The method according to claim 1, wherein the data control system expands the extracted effective predicates: for the predicates of the named graphs submitted by the user, the data control system obtains the named graphs submitted by the user and its Expand the predicate, compare the valid predicates in the named graph, and verify that the predicate is to be expanded.