JP7789079B2 - Asynchronous persistence of replicated data changes in database accelerators - Google Patents

Description

The present invention relates generally to crash recovery for databases, and more particularly to a computer-implemented method for crash recovery for linked databases. The present invention further relates to a linked database system with crash recovery for linked databases, and a computer program product.

Managing large amounts of data remains a continuing challenge for enterprise IT (information technology) organizations. This is due to (i) the ever-increasing amount of data and (ii) the variety of data perspectives. In addition to traditional structured data, enterprise IT systems also store large amounts of semi-structured and so-called unstructured data. Furthermore, to meet increasing industrial requirements for data analysis, specialized database systems optimized for analytical processing have been introduced in addition to traditional transaction-oriented databases. On the other hand, attempts have been made to perform all analytical processing in traditional transaction databases. However, analytical processing has proven to have an excessively strong and unpredictable impact on the performance of online transaction processing.

As a result, sophisticated combinations of transaction-oriented database management systems and analytics-optimized database management systems have been introduced. One example of such a combination product is based on the IBM DB2 Analytics Accelerator (IDAA) architecture.

On one side, a full-fledged row-based database management system (DBMS) may serve as the application endpoint for data manipulation language (DML) operations and query execution. Using heuristic decision criteria and rules, the query optimizer may transparently decide whether a query should be executed in the source DBMS, particularly for online transaction processing, or offloaded to the target DBMS, particularly for online analytical processing.

The target DBMS may be a full-fledged column-based database management system that maintains a shadow copy of a selected set of source DBMS tables. A strategy for creating a shadow copy of the target DBMS may involve migrating data stored in one or more tables in the source DBMS to the target DBMS at a given point in time. However, query data in the target DBMS may become out of date if the load occurred some time ago and the corresponding source tables have been modified in the meantime. Therefore, the contents of tables are typically adapted incrementally as changes are recorded in the corresponding source database tables.

Both database systems implement transaction mechanisms to guarantee the ACID (atomic, consistent, isolation, durable) properties of their respective databases. That is, concurrent modifications are properly isolated by locking techniques, consistency checks may ensure that the database state moves from one consistent state to another, and logging techniques such as write-ahead logging may be implemented to guarantee atomicity and durability of transactional changes.

However, ensuring ACID properties when the source database is modified by the current transaction can incur significant overhead. In particular, logging slows down the processing of insert/update/delete (IUD) statements because change records must be written to persistent storage before the transaction can continue. Compared to the frequently used in-memory processing of changes to the contents of database tables, adding this durability layer using a transaction log is relatively slow. On the other hand, crash recovery is required to restore the latest consistent state of the respective database in the event of a failure.

In the context of a database accelerator such as IDAA, the target accelerator database simply mirrors a snapshot of the source database. That is, the source database may act as the master of the data, or data manipulation via the IUD may be handled by the source database management system, with changes replicated to the target database via update techniques. The accelerator database may therefore act as a cache for storing partitions of the source database that may require fast analytical operations.

Several disclosures already exist in this context. The document US Pat. No. 6,299,499 describes a method for replicating database data in a clean shutdown state and generating a read-only copy of the replicated data. A related system may include a tracking module that monitors first transactions from a database application to a source storage device and generates log entries with at least one marker indicating a known good state of the application. The system further includes a computer coupled to a target storage device containing a database and a log file. The computer processes the transactions based on the log entries to replicate the data to the target storage device and performs a first snapshot to recreate the data stored in the log file in the database.

Additionally, the document in Patent Document 2 describes a distributed database system that provides fast crash recovery. Upon recovery from a database head node failure, a connection may be established with one or more storage nodes of a distributed storage system that store data for the database implemented by that database head node. Upon establishing a connection with a storage node, the database may be made available for access, for example, for various access requests.

However, conventional combined databases that are optimized for transactions on one side and analytical processing on the other can all require the well-known overhead of maintaining a persistent recovery log, which can significantly slow down the processing of the combined database, in addition to the suboptimal use of changes from the source database in the target database. Therefore, to achieve higher performance values for combined transactional/analytical database management systems and at the same time elegantly solve the recovery challenges of the target database, it may be necessary to reduce the required overhead.

U.S. Patent Application Publication No. 2015/0 205 853(A1) US Patent Application Publication No. 2014/0 279 930(A1)

According to one aspect of the present invention, a computer-implemented method for crash recovery for linked databases may be provided. The linked databases may include a source database and an associated target database, and selected queries for a database management system including the source database may be transferred to a database management system including the target database for processing. The method may include synchronizing selected portions of the contents of tables in the source database with respective portions of the contents of tables in the target database; applying changes to the source database to an in-memory target database portion of the database management system including the target database using a recovery log file of the source database during synchronization; and asynchronously storing the persistently applied changes to the in-memory target database portion to persistent target database storage.

The method may include, upon a database crash of the target database system, restoring the in-memory target database portion with the most recent snapshot available in persistent target database storage, and, upon a database crash of the target database system, applying changes from a source database recovery log file having a timestamp later than the most recent snapshot available in persistent target database storage to the in-memory target database portion.

According to another aspect of the present invention, a linked database system having crash recovery for the linked database may be provided. The linked database may include a source database and an associated target database, and selected portions of the contents of tables in the source database may be synchronized with respective portions of the contents of tables in the target database. The linked database system may include a processor and memory communicatively coupled to the processor, the memory storing program code portions that, when executed, enable the processor to synchronize selected portions of the contents of tables in the source database with respective portions of the contents of tables in the target database, apply changes to the source database to an in-memory target database portion of a database management system including the target database using a recovery log file of the source database during synchronization, and asynchronously store the persistently applied changes to the in-memory target database portion in persistent target database storage.

In addition, the memory may store program code portions that, when executed, enable the processor to restore the in-memory target database portion with the most recent snapshot available in the persistent target database storage in the event of a database crash of the target database system, and to apply changes from a source database recovery log file having a later timestamp than the most recent snapshot available in the persistent target database storage to the in-memory target database portion in the event of a database crash of the target database system.

The proposed computer-implemented method for crash recovery for linked databases may provide numerous advantages, technical effects, contributions, or improvements, or combinations thereof.

Updates and modifications to the in-memory portion, as well as queries, may continue without any slowdown due to also durably storing the changes to the target database. Thus, logging and state storage of physical database data structures may be separated from IUD processing within the database transaction. This means that the persistence phase is skipped when changes to the target database, particularly the in-memory portion, are made, and update operations may continue without additional delay. Therefore, records that may be needed for crash recovery of the target database may be written asynchronously by a dedicated asynchronous data persistence service. Changes made by IUD transactions apply only to the in-memory representation of the data, so they may be immediately processed by the target database, i.e., database queries that are offloaded to the accelerator, regardless of whether they have already been stored to a durable medium. This may also be interpreted as lifting the durability constraints of the database's ACID properties.

Because changes related to in-memory data structures are written asynchronously to their persistent counterparts, in the event of a database crash, some changes may be lost in the target database system. Crash recovery may therefore need to synchronize a potentially incomplete snapshot with the source database system by replaying the lost changes. Unlike other solutions, where offloaded tables are not fully reloaded by a bulk load mechanism, the missing portions may be recovered incrementally. That is, the target database state may be recovered from the latest asynchronously persistent snapshot of the target database, with the help of recovery metadata to determine which additional changes may still be lost; that is, which changes from the source database need to be replayed by the target database from the source database system's transaction or recovery log files to ultimately restore the current consistent state of the target database.

This allows incoming queries on the target database to be delayed while the recovery process is active until a consistent update state is re-established in the in-memory portion of the target database. While the delay caused by recovery may be significantly lower compared to known solutions, the processing and storage costs of asynchronously maintaining the database snapshots in the target database system may be slightly higher. However, this is a small price to pay for higher availability and shorter recovery times for the target database system.

Asynchronous persistent storage of changes to the target database may also help perform logging more efficiently, I/O operations may be performed more efficiently, for example by a batch process, or self-cancelling changes may be deleted altogether. That is, target records that have been modified in a way that can be recovered before the next persistent storage step of the target database changes may be deleted in a buffer between the target database and the persistent storage medium.

Further embodiments of the inventive concepts applicable to methods and systems will be described below.

According to one interesting embodiment of the method, synchronizing may involve reading entries from the recovery log file relating to the source database and applying the read entries to the target database. This may be done by the management part of the target database management system. However, other techniques for synchronization may also be used, such as those based on SQL (structured query language) or Q, a query language optimized for column-based databases.

According to one useful embodiment of this method, the source database may be optimized for transactions, i.e., online transaction processing (OLTP), or the source database may be a row-oriented relational DBMS, or both. This may represent and efficiently support the operational backbone of enterprise IT (information technology) operations. A row-oriented database may be optimized for balanced, high-speed read, write, and modify operations on the data within the database. A row-oriented database may also be effective for generating reports; however, this type of database is often not optimal for analytical-type operations.

According to one acceptable embodiment of this method, the target database may be optimized for analytical operations, i.e., online analytical processing (OLAP), or the target database may be a column-oriented database, or both. According to a further embodiment, this type of database may be a column-oriented database, which may support queries with many dependencies and intercorrelations significantly better than OLAP-optimized database systems.

According to an advantageous embodiment, the method may also include, in the event of a target database crash, delaying queries to the target database until recovery of the target database can be completed. Thus, the recovery process may be transparent to the user. The user may ultimately experience a slight delay in their analytical queries. However, since response times for complex analytical queries are relatively long in any case, the user may not notice the delay at all.

Optionally, queries targeting the target database may be executed by the source database during the target database recovery process. This may slow down the source database slightly during the transaction, but this may be an acceptable compromise for overall user satisfaction.

According to a preferred embodiment of this method, the metadata defining the selected tables may be part of the recovery log file. In this way, the general architecture of the in-memory target database may already be defined in the recovery log file of the source database. This may also apply to the data portion of the tables to be mirrored to the source database. Thus, a single source for data definitions may be used so that conflict situations cannot arise.

According to an advanced embodiment of this method, storing the persistently applied changes may include waiting until a predetermined number of changes can be completed in the in-memory target database portion. The predetermined number may be configurable during database setup, or may be changed during operation of the database management system, or both. However, an advantage of this approach may be that it does not slow down analysis operations on the target database. Thus, a group of updates to the target database may be extracted and persistently stored at one time. In addition to using a predetermined number of changes to the target database, it may also be considered to monitor the load on the target database and perform persistent storage of changes to the target database during periods of relatively low analysis load on the target database.

This feature may be implemented in a separate thread or process that may wait until a predetermined number of changes are completed in the in-memory portion. For each change in the set of changes that has completed since the last execution, the changes are persisted to the persistent database portion of the target database. In addition, to enable identification of change replay points during the recovery phase, associated metadata describing the most recently persisted change (e.g., the last corresponding block record sequence number in the source database system) may also be persistently stored. This may enable the time of the last consistently changed set stored. This time may be useful for recovering the in-memory portion of the target database from the persistent portion of the target database and incrementally recovering the remaining IUDs of the in-memory portion of the target database from the source database recovery log file. This additional data is requested from the source database or the incremental update process when recovery of the target database is required. At the end of the recovery process, the query processor may be notified by a "reactive" signal that queries may now be processed from the target database.

According to another advantageous embodiment of the method, restoring a table or partition thereof in the in-memory target database portion may include prioritizing or restoring the table or partition by one selected from the group consisting of data usage, query priority, and data priority.

These options are described in more detail in the following paragraphs, but the general concept is to enable different optimization methods during the recovery process in order to allow queries to be executed even while the target database is "reconstructing", i.e. in recovery mode. These optimization options may be configurable and may generally be combinable in one implementation.

According to one optional embodiment of this method, prioritizing recovery by data usage may include maintaining a counter for each table in the target database, or a counter for its partitions. The counter value may indicate how many queries may be waiting for related tables, with the database table with the highest counter value being restored first. Other tables may then be restored subsequently. This option may enable rapid recovery of tables in the target database that are in high demand. Thus, users may be provisioned with fully restored portions of the database, i.e., tables with high demand, as soon as possible. This option may be referred to as demand or data usage optimized recovery.

Therefore, a list of waiting queries may be taken into account to control the recovery process and minimize recovery time. When a new database query arrives at the target database while recovery is in progress, its data access may be analyzed. If the database query does not access non-recovered data, it may be processed immediately. Otherwise, the query is registered in the list of waiting queries as part of the recovery state. The recovery phase is evaluated by a recovery schedule to derive a set of recovery actions that prioritizes tables and/or table partitions that are actually blocking queries. Upon completion of recovery, the most beneficial recovery action may be scheduled via one of the available recovery strategies, the recovery state may be updated, and waiting queries against recovered tables and/or partitions may be notified when they no longer need to be blocked.

In more detail, crash recovery for a target database can be described as follows: First, a list of tables or table partitions that need to be recovered by using known recovery metadata in the target database. Second, the list of tables or table partitions to be recovered is stored in the recovery state. Third, if the list of what to recover is not empty, the following may be done:
(i) determining the next table and/or table partition to recover from the list;
(ii) scheduling recovery operations via a recovery strategy configured for the target database;
(iii) waiting for the recovery action to complete;
(iv) updating the recovery state, i.e., marking the table/partition to be recovered and removing it from the blocked data lists of all queries waiting for that table and/or table partition; and (v) determining the list of queries whose blocked data lists are now empty and notifying the query processor that the query can now be processed.

Thereby, the list of tables and/or table partitions to be recovered may be implemented by a priority queue, using the algorithm described below to dynamically calculate the next recovery item for the next recovery cycle. To accelerate the recovery process, the steps referred to in (iii) may be performed in parallel for multiple tables and/or table partitions.

The determination of the tables and/or table partitions to recover may be performed as follows.
(i) creating an empty histogram of the shut-off table and/or table partition when the crash recovery process is initiated;
(ii) when a blocking query is registered in the recovery state, incrementing a usage counter for each table and/or table partition in the blocking area list of the query by one;
(iii) when it is necessary to determine the next table/table partition to be recovered, selecting the table/table partition with the highest usage counter, and (iv) when the table/table partition recovery operation is completed, removing the corresponding histogram data.

According to another optional embodiment of this method, prioritizing recovery by query priority may include first restoring database tables that receive queries with the highest priority. Such priority values may be assigned to the database system, e.g., production databases versus test databases, or may be assigned based on individual queries. This option may be referred to as query priority or simply data priority optimized recovery.

According to a further optional embodiment of this method, prioritizing recovery by data priority may include maintaining (at least) two groups of database tables, each group relating to a distinct group of users, and first restoring the database tables of the group with a higher constituent group priority. Such a situation may arise in a multi-user/multi-group/multi-tenant environment, where one user, group, or tenant may be assigned a higher priority for queries. For example, one tenant may have a higher guarantee of availability of the database system. In such a case, this tenant may be assigned a higher priority. Such a scenario may work best when the multi-tenant database is operated in a cloud computing data center. This option may be denoted as customer priority optimized recovery.

According to another interesting embodiment, the method may further include determining a data volume to be restored for the next table to be restored and recovering that table(s) using a recovery strategy that depends on the volume to be restored. The recovery strategy is thereby an incremental update strategy or a bulk update strategy. Thus, it may be determined for each database table (or group of database tables) which update strategy to use in order to minimize the total recovery time of the target database. This option may be denoted as time-optimized recovery.

Furthermore, embodiments may take the form of a related computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium may be any apparatus that may include means for storing, communicating, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.

It should be noted that embodiments of the present invention are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will infer from the above and below that, unless otherwise indicated, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matter, in particular any combination of features from a method-type claim with a feature from an apparatus-type claim, is also considered to be disclosed in this document.

The above-defined aspects and further aspects of the present invention will be apparent from and will be explained with reference to the exemplary embodiments described hereinafter, but the present invention is not limited thereto.

Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings:

1 is a block diagram illustrating an embodiment of the computer-implemented method of the present invention for crash recovery for linked databases. FIG. 1 is a block diagram illustrating an embodiment of a linked database. FIG. 2 is a block diagram illustrating an embodiment of how target databases may be synchronized. FIG. 1 is a block diagram showing an embodiment of the proposed concept in a form closer to implementation. 1 is a block diagram illustrating an embodiment of a linked database system including components for a customer priority optimization recovery strategy. 1 is a block diagram illustrating an embodiment of a linked database system including components for a volume-optimized recovery strategy. 1 is a block diagram illustrating an embodiment of a linked database system for crash recovery for linked databases. 1 illustrates an embodiment of a computer system including a linked database system.

In the context of this description, the following conventions, terms, or expressions, or combinations thereof, may be used:

The term "crash recovery" may refer to the process of reconstructing the state of a database before a crash occurred. When a crash occurs, data may be unavailable or inconsistent.

The term "linked database" may refer to at least two databases that are closely related to each other. In the context of this document, linked databases may be referred to as databases that may store at least part of the same data. On the other hand, the primary database in such a pair may be optimized for a different task than the secondary database.

The terms "source database" or "primary database" may refer to a database that is optimized for, for example, high-speed transactions, i.e., online transaction processing. However, a database optimized in this manner, i.e., for high-speed read, write, and update operations on data, may be slow to execute complex queries involving many tables or large amounts of data, such as in the case of online analytical processing. In addition, online analytical processing may slow down online transaction processing. Thus, highly optimized database management systems for databases of the types just mentioned may function successfully in tandem.

The terms "target database" or "secondary database" may refer to the second database in such a tandem of databases optimized for different tasks. In the context of the concepts described herein, the target database may be optimized for online analytical processing. The target database may store at least some of the tables and some of the table data of the source database. In addition, the target database may include two parts: an in-memory part for fast execution of complex multidimensional queries, and a persistent part that may store the tables and data of the in-memory portion of the target database in more long-term storage, such as a hard disk or flash memory. In this way, the target database may be able to recover most of its contents from persistent storage in the event of a target database crash.

The term "selected portion of the table contents" may refer to the just-mentioned portion or partition of data in the portion of the table in the primary database that may be synchronously copied and stored in the target database.

The term "selected query" may refer to a particular type of query directed to a linked database that, based on the nature of the query, is better executed by one of the two databases, particularly the target database. For example, when the query type relates to online analytical processing, the query may be forwarded to the target database and not executed by the source database.

The term "database management system" may refer to a combination of administrative/management systems, typically implemented in a combination of hardware and software, and at least one associated database that stores data.

The term "in-memory target database portion" may refer to a portion of a target database that may hold substantially all of its data in the main memory of a computer system. A database management system for a target database may include an in-memory portion and a persistent portion of the target database, where the persistent portion may be a persistently stored copy of the in-memory portion apart from the most recent changes made to it in memory.

The term "persistent target database storage" may describe that portion of a target database management system that allows target database data to be stored persistently, i.e., using a hard disk or flash memory instead of in-memory storage.

The term "latest snapshot" may refer to the last consistently stored state of the target database.

The term "later timestamp", and in particular changes having later timestamps, may refer to, for example, records in the source database recovery log file that contain a time indicator that may have been created later in time than the most recent snapshot stored in the persistent portion of the target database.

The term "recovery log file" may refer to the set of file protocol operations performed on a database, specifically all operations that modify data, i.e., insert, update, and delete operations. The recovery log file may be designed to allow a complete reconstruction of the database. Therefore, the table definitions of the database may also be part of the recovery log file.

The term "metadata" may refer to data about data, particularly the definitions of data in tables in a database, and potentially the relationships between them.

Below, a detailed description of the drawings will be provided. All instructions in the drawings are schematic. First, a block diagram of an embodiment of the computer-implemented method of the present invention for crash recovery for linked databases will be provided. Thereafter, further embodiments and embodiments of a linked database system with crash recovery for linked databases will be described.

Figure 1 shows a block diagram of a preferred embodiment of a computer-implemented method 100 for crash recovery for linked databases, particularly according to the IDAA architecture, where the linked databases provided (102) include a source or primary database, particularly optimized for transactions, such as a role-based database, and an associated target or secondary database, which may be optimized for online analytical processing operations (OLAP) and advantageously organized on a column basis.

Selected queries to the database management system containing the source database, particularly queries directed to analytical operations, are transferred, or offloaded, to the database management system containing the target database for processing.

Additionally, method 100 includes synchronizing (104) selected portions of the contents of the tables of the source database (in some implementations, only a portion of the data in the portions of the tables, in other implementations, a complete copy) with respective portions of the contents of the tables of the target database by applying (106) changes to the source database during synchronization using the recovery log file of the source database to the in-memory portion of the target database of the database management system that contains the target database.

Additionally, method 100 includes asynchronously storing (108) persistently applied changes to the in-memory target database portion in a persistent target database storage portion; restoring (110) the in-memory target database portion with the most recent snapshot available in the persistent target database storage in the event of a database crash of the target database system; and applying (112) changes from a source database recovery log file having a later timestamp than the most recent snapshot available in the persistent target database storage to the in-memory target database portion in the event of a database crash of the target database system.

Optionally, the persistent portion of the target database may also be updated in parallel. However, this would require a persistent storage process or storage processor to also be active to persistently store changes to the in-memory portion of the target database.

Additionally, it may be advantageous to bulk load the target database from the source database, especially during the initialization process. This may help avoid an equally slower incremental update or synchronization process because the source database may already have a larger number of entries and therefore a larger number of entries in its recovery log file. This initialization may be effective for only selected tables or their partitions.

Figure 2 shows a block diagram of a linked database embodiment 200. A primary or source database 202 receives OLTP queries 210 and OLAP queries 214. Queries identified as OLAP queries 214 are migrated or offloaded 222 to a secondary or target database 206. The source database 202 includes multiple tables 204 and associated stored data. The target database 206 also includes tables in database tables 208 that represent at least a subset of the database tables 204 of the source database 202 and at least a subset of that data.

After the OLAP operation is performed by the target database 206, the data is returned 224 and OLAP output 220 is returned to the requesting program or process. The OLTP query 210 is executed directly in the source database 202 and returned to the requesting program or process as OLTP output 212. Thus, the OLTP or source database 202 may operate at its peak performance without being slowed down by any resource-intensive OLAP queries.

Because the data organization in the target database 206 may be different, for example, column-oriented instead of row-oriented in the source or OLAP database 202, the target database may return OLAP results 224 significantly faster than the source database 202.

Figure 3 shows a block diagram of an embodiment 300 of how target databases can be synchronized. A source database management system 302 controls the operation of the source database 202 and its associated tables 204 (compare Figure 2). The same is true for a target database management system 308, the target database 206, and its associated tables 208.

The source database management system 302 also maintains a recovery log file 306 for the source database 202. A log reader or log file reader 314 reads the recovery log file 306 and provides this data to an apply unit 316, which applies changes (i.e., inserts, updates, deletes) made to the source database to a selected set of selected tables and data in the target database 206. The selected set of selected tables and data may be a subset of related tables and data in the source database 202 as predefined. The apply unit 316 can optimize the application of changes to the target database 206 depending on the OLAP query being executed. A log buffer 318 may be useful for this purpose.

To initialize the target database 206, a bulk load operation 312 may be performed from the source database 202 to the target database 206 for performance reasons.

Note that for reasons of clarity of the inventive concept, Figure 3 does not yet show the division of the target database management system into in-memory and persistent portions of the target database 206 and its tables 208. This will be shown in the next figure.

Note that this synchronization mechanism may represent one of many synchronization techniques. Other synchronization techniques may also be applicable.

Figure 4 shows a block diagram of an embodiment of the proposed concept in a more practical form 400. Elements of the source database management system (primarily the upper part of Figure 4) that are optionally used to synchronize the source database with the target database will not be described again.

The target database management system 308 includes an in-memory portion 402 of the target database and a persistent portion 404 of the target database. A persistence service 406 writes the state of the in-memory target database portion 402 to the persistent database 404 during operation. During these regular operations, for example, after a predetermined number of updates to the in-memory database 402, a query processor 408 directs incoming queries to the in-memory portion 402 of the target database.

However, if the target DBMS is currently undergoing crash recovery, especially if it is performing a demand or data usage optimized recovery, the behavior differs as follows:
(i) the query processor 408, together with the recovery processor 410, analyzes the query data accesses to determine, using the current query analyzing view decomposition techniques, a list of target tables and/or a list of table partitions that need to be made available for the query;
(ii) a blocked data list is determined by looking at all of the tables and/or table partitions in the data access list from the previous step in the recovery state that have been determined not to have been recovered;
(iii) if the list from the previous step is not empty, the query and its blocked data list are added to the waiting queries, and the query process waits until recovery is finished (the blocked data list is empty), and then the query can continue; and (iv) after a completion notification is sent from recovery processor 410 to query processor 408, the query of the target database is processed as normal.

During operations controlled by the recovery processor 410, the latest consistent snapshot available in the persistent database 404 of the relevant tables and/or table partitions is loaded 412 into the in-memory portion 402 of the target database, and entries in the recovery log file 306 of the source database 202 that have a later timestamp than the snapshot in the persistent database 404 are recreated in the in-memory database 402. These are requested from the recovery processor 410 via the query processor 408 and provided from the source DBMS 302, for example from the recovery log file 306, via the log reader 314 and apply unit 316.

Figure 5 shows a block diagram of an embodiment 500 of a linked database system including components for a customer priority optimization recovery strategy. A source DBMS 502 illustratively has a first source database 504 for a first user or tenant (e.g., in a cloud computing environment) and a second source database 506 for a second user or tenant. In addition, separate source database systems for other users or tenants may be available.

Separate data synchronization subsystems 510, 512 are implemented within the data synchronization system 508 to synchronize the source databases 504, 506 with respective in-memory portions 518, 520 of the in-memory target database 516. A query process 522 receives database queries for execution in the in-memory portion of the target database(s) 516. These queries are typically OLAP queries offloaded from the source database(s) 504, 506.

The recovery process or processor 524 of the target DBMS 514 also receives data from the query processor regarding queries to be registered and analyzed in the data access analyzer 526. During the target database recovery process, the query analyzer determines which tables should be recovered first based on the priority of a particular user's queries by determining the queued queries 530 and already recovered table partitions 532 in the recovery state management system 528. This is ultimately determined and decided by the recovery schedule 534. To accomplish this task, the recovery schedule 534 constantly exchanges data with the recovery state management system 528 to check the actual recovery state and receives configuration data from the workload management system configuration storage. The recovery schedule 534 also exchanges data with the data synchronization system 508 to trigger the target database management system recovery database load from the source database management system 502.

In this way, you can ensure that users or customers with higher configured priorities have earlier access to recovered database tables, optimizing recovery times depending on the access usage of a particular table.

In detail, this may be achieved by the following steps: When the crash recovery process for the target database is initialized, an empty histogram of blocked tables/table partitions for each tenant is created. When a new blocked query is registered in the recovery state, the usage counter for each table/partition in that query's blocked data list is incremented by one. Then, when the next table/partition to be recovered needs to be determined, the table's recovery priority is determined based on the blocked query usage counter count and the current tenant's WLM (workload management system) configuration (i.e., its priority or importance), and the recovery item with the highest priority is selected. Finally, when the table/partition recovery operation is completed, the corresponding histogram entry is also removed.

Calculating a user or tenant specific recovery priority can be implemented, for example, as follows: When there is a strict priority between pairs of tenants, e.g., production systems are more important than test systems (which may be specified by simple configuration text), the list of tables/partitions to be recovered is grouped by user/tenant, the list of groups is ordered by decreasing user/tenant priority, and the next table/partition is selected from the first non-empty group by a query usage counter selection algorithm.

However, when there is a relative resource sharing position for each tenant in the WLM configuration (e.g., Tenant 1 50%, Tenant 2 30%, Tenant 3 20%), the priority for each recovery item is calculated as follows: (i) determine the query usage counter q for the recovery item; (ii) then determine the tenant t to which the recovery item belongs; (iii) then determine the resource sharing position r(t) for tenant t; (iv) based thereon, determine the priority as a*q*b*r(t), where a and b are static configuration parameters, particularly in the range [0...1], and to reduce the number of parameters, b can be calculated based on a, e.g., b=1-a. Finally, (v) the list of recovery items can be ordered by priority, e.g., in a priority queue.

Figure 6 shows a block diagram of an embodiment 600 of a linked database system including components for a volume-optimized recovery strategy. Elements already introduced by Figure 5 are indicated with the same reference numerals. On the upper left side, the source DBMS 502 is shown along with the source database 504 and associated recovery log file 604. Initially, the in-memory database portion 516 (persistent portion not shown) of the target DBMS 514 may be bulk loaded via the bulk loader 602.

In addition to the in-memory portion 516 of the target DBMS 514, the focus here may be on other metadata 608 and the recovery process or recovery processor 614. The recovery process includes at least three components: a recovery item selection unit 616, a change estimation unit 618, and a recovery schedule 620. The recovery processor 614 exchanges data with the in-memory database 516 to collect state information about the tables in the in-memory database portion 516 of the target DBMS 514.

As already mentioned above, in the event of a database crash on the target DBMS side, the target database 516 needs to be restored from the source database 504 either via a bulk load mechanism or incrementally. Additionally, the tables/table partitions to be recovered are dynamically selected by the recovery process 614. The proposed concept is extended by a change estimation component 618, which is responsible for estimating or determining the amount of data that needs to be restored during crash recovery. Therefore, this change estimation component 618 evaluates data change statistics to estimate how many data changes have accumulated in the source database since the recovery baseline timestamp. Based on this information, the recovery scheduler 620 selects the most efficient data synchronization method for restoring the data of the tables/table partitions to be recovered. The data change statistics are maintained during regular update processes via incremental or bulk loads of the target database (i.e., the in-memory database portion 516). The change estimation 618 may be triggered by the recovery item selection component 616.

The recovery scheduler 620 also exchanges data with the bulk loader 602 and the details of the incremental update process 606 to manage the recovery process of the in-memory database portion 516. For example, when recovery of the in-memory database portion 516 is completed, a recovery completion notification is received from the bulk loader 602. On the other hand, the recovery scheduler 620 requests the reproduction of changes to specific tables in the in-memory database portion 516. Similar to signals from the bulk loader 602, the recovery scheduler 620 also receives recovery completion notifications from the incremental update process (processor) 606. As already explained in the context of FIG. 3, the incremental update process 606 includes a log reader (not shown) adapted to read recovery log file entries from the recovery log file 604 and a recovery log file application unit (not shown) adapted to incrementally update the memory database portion 516 with respective recovery log file entries from the source database 504. Please refer back to Figure 3 for more details.

To successfully manage the processes of the embodiments described herein, data change statistics should be stored in a persistent metadata catalog included in the metadata 608 of the target database system 514 and should be maintained when the target database system 514 is updated, i.e., via incremental updates or bulk load strategies. The data change statistics may store the following information: (i) the timestamp when the update was processed; (ii) schema information for the database table, e.g., column types, column widths, ...; (iii) the amount of data changes per table/table partition affected by the update, i.e., inserted, deleted, and updated records; and aggregated metrics, e.g., total number of changed records, total data volume applied, total execution time of the update, etc.

Statistical data may be linked to a time interval, such as the last x number of days. Additionally, statistical data may be maintained incrementally as part of each update cycle.

Furthermore, the recovery baseline 610 is also maintained in the persistent metadata 608 (e.g., metadata catalog) of the target database system 514. The recovery baseline 610 is needed for estimation to determine the data volume that needs to be restored. Therefore, it is not necessary to determine the exact timestamp; it is sufficient to estimate its value; for example, it may be maintained as a heartbeat timestamp updated by the target database system at regular time intervals, it may be maintained as a starting point for the recovery process after a crash, or it may be maintained as the time of the last successfully persisted target database snapshot. Thereby, a timestamp may be maintained for each table in the target database.

Crash recovery for the target database can be performed as follows: (i) first, the next table/partition to be recovered is determined; (ii) the data volume that needs to be recovered since the recovery baseline is estimated; (iii) the best recovery strategy is selected based on the estimated data volume, and the recovery time is estimated; (iv) the table recovery according to the selected strategy is then scheduled; and (v) these steps are repeated in a loop until all data is recovered.

Estimating the data volume to be restored may be performed as follows: (i) for the table/partition to be restored, the corresponding data change statistics 612 are looked up; (ii) a recovery baseline for that table/partition is determined; and (iii) the number of changes that need to be replicated via the incremental update process in the interval [recovery baseline, current recovery time] is estimated.

Figure 7 shows a block diagram of an embodiment of a linked database system 700 for crash recovery for linked databases. The linked database 700 includes a source database 706 and an associated target database 708, in which selected portions of the contents of tables in the source database are synchronized with respective portions of the contents of tables in the target database.

The linked database system 700 includes a processor 702 and a memory 704 communicatively coupled to the processor 702, the memory 704 storing program code portions that, when executed, enable the processor to synchronize selected portions of the contents of tables in the source database 706 with respective portions of the contents of tables in the target database 708, for example, using a synchronization unit 714.

When executed, the stored program code portions enable the processor 702 to synchronously apply, using the apply unit 716, changes to the source database 706 to an in-memory target database portion 710 of a database management system that includes the target database, and to asynchronously persistently store, by, for example, the storage processor 718, the applied changes to the in-memory target database portion 712 in persistent target database storage.

Furthermore, when executed, the stored program code portions enable the processor 702 to restore, for example by a restore unit 720, the in-memory target database portion 710 with the latest snapshot available in the permanent target database storage portion 712 in the event of a database crash of the target database system, and to apply, for example by a second apply unit 722, changes from a source database recovery log file having a later timestamp than the latest snapshot available in the permanent target database storage portion 712 to the in-memory target database portion 710 in the event of a database crash of the target database system.

Additionally, all modules and units of the linked database system 700 may be electrically interconnected to exchange signals and/or data. This may apply to the processor 702, memory 704, source database system 706, target database system 708 including in-memory portion 710 and persistent portion 712, synchronization unit 714, application unit 716, storage processor 718, restore unit 720, and second application unit 722. Instead of 1:1 connections between these modules and units, they may be connected to a linked database system internal bus system 724 for signaling and data exchange.

Embodiments of the present invention may be implemented in conjunction with virtually any type of computer, regardless of whether the platform is suitable for storing and/or executing program code. Figure 8 illustrates, by way of example, a computer system 800 suitable for executing program code relating to the proposed method.

Computer system 800 is merely one example of a suitable computer system, and whether computer system 800 is capable of implementing and/or performing any of the functions set forth above is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the present invention described herein. Computer system 800 has components that operate in conjunction with numerous other general-purpose or special-purpose computer system environments or configurations. Examples of well-known computer systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system/server 800 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. Computer system/server 800 may be described in the general context of computer system-executable instructions, such as program modules, executed by computer system 800. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system/server 800 may also be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

As shown in the drawings, computer system/server 800 is shown in the form of a general-purpose computing device. Components of computer system/server 800 may include, but are not limited to, one or more processors or processing units 802, system memory 804, and a bus 806 that couples various system components, including system memory 804, to processor 802. Bus 806 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus. Computer system/server 800 typically includes a variety of computer system-readable media. Such media may be any available media that is accessible by the computer system/server 800 and includes both volatile and non-volatile media, removable and non-removable media.

System memory 804 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 808 and/or cache memory 810. Computer system/server 800 may also include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 812 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to removable, non-volatile magnetic disks (e.g., "floppy disks"), and an optical disk drive may be provided for reading from or writing to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such cases, each may be connected to bus 806 by one or more data media interfaces. As will be further shown and described below, memory 804 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

By way of example and not limitation, a program/utility having a set (at least one) of program modules 816, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 804. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may comprise an implementation of a networking environment. The program modules 816 generally perform the functions and/or methods of embodiments of the present invention described herein.

The computer system/server 800 may also communicate with one or more external devices 818, such as a keyboard, pointing device, or display 820; one or more devices that allow a user to interact with the computer system/server 800; or any device (e.g., a network card, modem, etc.) that allows the computer system/server 800 to communicate with one or more other computer devices; or a combination thereof. Such communication may occur via an input/output (I/O) interface 814. Additionally, the computer system/server 800 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 822. As shown, the network adapter 822 may communicate with other components of the computer system/server 800 via the bus 806. Although not shown, it should be understood that other hardware and/or software components may be used with computer system/server 800. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

In addition, a linked database system 700 with crash recovery for the linked database may be attached to the bus system 806.

The descriptions of various embodiments of the present invention have been provided for illustrative purposes and are not intended to be exhaustive or limiting to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements over technology found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

The present invention may be embodied as a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system for propagation media. Examples of computer-readable media may include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read-only memory (CD-ROM), compact disk-read/write (CD-R/W), DVDs, and Blu-Ray® disks.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: Portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves with recorded instructions, and the like, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing/processing device, or may be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing/processing device for storage.

The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk or C++, and conventional procedural programming languages such as the "C" programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by using state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, cause means for implementing the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored includes instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams.

The flowcharts and/or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. In addition, it will be noted that each block of the block diagrams and/or flowchart diagrams, and combinations of blocks in the block diagrams and/or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," as used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

Structure, material, acts, and equivalents corresponding to all means- or step-plus-function elements in the following claims are intended to include any structure, material, or acts for performing that function in combination with other specifically claimed elements. The description of the present invention has been provided for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The embodiments have been chosen and described to best explain the principles and practical application of the invention, and to enable others skilled in the art to understand the invention in various embodiments with various modifications as may be suitable for the particular uses anticipated.

Briefly, the concept of the present invention may be summarized in the following points:

1. A computer-implemented method for crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected queries to a database management system including the source database are transferred to a database management system including the target database for processing, the method comprising:
- synchronizing selected portions of the contents of tables in a source database with respective portions of the contents of tables in a target database;
During synchronization, applying changes to the source database to an in-memory target database portion of a database management system that contains the target database;
asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage;
In the event of a database crash of the target database system, restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage;
- in the event of a database crash of the target database system, applying changes from a source database recovery log file having a timestamp later than the most recent snapshot available in the persistent target database storage to the in-memory target database portion.

2. The method according to item 1, wherein the synchronizing comprises:
- One that involves reading recovery log file entries relating to the source database and applying the read entries to the target database.

3. A method according to item 1 or 2, wherein the source database is optimized for transactions, or the source database is a row-oriented relational database management system, or both.

4. A method according to any of the preceding items, wherein the target database is optimized for analytical operations, or the target database is a column-oriented database, or both.

5. A method according to any of the preceding items,
In the event of a target database crash, further comprising delaying queries to the target database until recovery of the target database is complete.

6. A method according to any of the preceding items, wherein the metadata defining the selected table is part of the recovery log file.

7. A method according to any of the preceding items, wherein storing the permanently applied changes comprises:
- One that involves waiting until a predetermined number of changes are completed in the in-memory target database portion.

8. A method according to any of the preceding items, wherein restoring tables of the in-memory target database portion includes prioritizing recovery by one selected from the group consisting of data usage, query priority, and data priority.

9. The method according to claim 8, wherein prioritizing recovery by data usage comprises:
- maintaining a counter for each table in the target database, the counter value of the counter indicating how many queries are waiting for the associated table;
- Restoring the database table with the highest counter value first.

10. The method according to item 8 or 9, wherein prioritizing recovery by query priority comprises:
- One that involves first restoring the database table that receives the query with the highest priority.

11. The method according to any of items 8 to 10, wherein prioritizing recovery by data priority comprises:
- maintaining two groups of database tables, each group relating to a distinct group of users;
- Restoring the database tables of groups with higher configuration group priority first.

12. A method according to any of the preceding items, comprising:
determining the data volume to be restored for the next table to be restored;
recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being an incremental update strategy or a bulk update strategy.

13. A linked database system with crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected portions of the contents of tables in the source database are synchronized with respective portions of the contents of tables in the target database, the linked database system comprising:
a processor and a memory communicatively coupled to the processor, the memory storing program code portions that, when executed, cause the processor to: synchronize selected portions of the contents of tables of a source database with respective portions of the contents of tables of a target database;
During synchronization, applying changes to the source database to an in-memory target database portion of a database management system that contains the target database;
asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage;
In the event of a database crash of the target database system, restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage;
- In the event of a database crash of the target database system, it is possible to apply changes from a source database recovery log file having a later timestamp than the most recent snapshot available in the persistent target database storage to the in-memory target database portion.

14. The linked database system according to item 13, wherein the program code portion further enables the processor to: read entries in a recovery log file relating to the source database for synchronization; and apply the read entries to the target database.

15. A linked database system according to item 13 or 14, wherein the source database is optimized for transactions, or the source database is a row-oriented relational database management system, or both.

16. A linked database system according to any of items 13 to 15, wherein the target database is optimized for analytical operations, or the target database is a column-oriented database, or both.

17. A linked database system according to any of items 13 to 16, wherein the program code portion further enables the processor to: in the event of a crash of the target database, delay queries to the target database until recovery of the target database is complete.

18. A linked database system according to any of items 13-17, in which the metadata defining the selected tables is part of the recovery log file.

19. A linked database system according to any of items 13 to 18, wherein the program code portion further enables the processor to: wait until a predetermined number of changes are completed in the in-memory target database portion in order to store the changes to be permanently applied.

20. A linked database system according to any of items 13-19, wherein restoring tables of the in-memory target database portion includes prioritizing recovery by one selected from the group consisting of data usage, query priority, and data priority.

21. A linked database system according to item 20, wherein the program code portion, for prioritizing recovery by data usage, causes the processor to: maintain a counter for each table in the target database, the counter value of the counter indicating how many queries are waiting for the associated table;
- Further allows restoring the database table with the highest counter value first.

22. The linked database system according to item 20 or 21, wherein the program code portion further enables the processor to prioritize recovery by query priority, restoring first the database table that receives the query with the highest priority.

23. A linked database system according to any of items 20-22, wherein the program code portion for prioritizing recovery by data priority causes the processor to: maintain two groups of database tables, each group relating to a distinct group of users;
- Further allowing the database tables of groups with higher configuration group priority to be restored first.

24. A linked database system according to any of items 13 to 15, wherein the program code portion includes a processor: determining a data volume to be restored for the next table to be restored;
- recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being an incremental update strategy or a bulk update strategy.

25. A computer program product for a linked database system with crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected portions of the contents of tables in the source database are synchronized with respective portions of the contents of tables in the target database, said computer program product including a computer readable storage medium having program instructions embodied thereon, said program instructions being executable by one or more computer systems or controllers to cause said one or more computer systems to:
- synchronizing selected portions of the contents of tables in a source database with respective portions of the contents of tables in a target database;
During synchronization, applying changes to the source database to an in-memory target database portion of a database management system that contains the target database;
asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage;
In the event of a database crash of the target database system, restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage;
In the event of a database crash of the target database system, changes from a source database recovery log file having a timestamp later than the most recent snapshot available in the persistent target database storage are applied to the in-memory target database portion.

Claims (25)

1. A computer-implemented method for crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected queries for a database management system including the source database are transferred for processing to a database management system including the target database, the method comprising:
synchronizing selected portions of the contents of the tables of the source database with respective portions of the contents of the tables of the target database;
During said synchronization, applying changes to said source database to an in-memory target database portion of said database management system that includes said target database;
asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage;
In the event of a crash of the target database , restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage;
and upon the crash of the target database , applying changes from a recovery log file of the source database having a timestamp later than the most recent snapshot available in the persistent target database storage to the in-memory target database portion. The synchronizing
2. The method of claim 1, further comprising: reading entries from a recovery log file relating to said source database; and applying said read entries to said target database. The method of claim 1, wherein the source database is optimized for transactions or is a row-oriented relational database management system. The method of claim 1, wherein the target database is optimized for analytical operations or is a column-oriented database. 2. The method of claim 1, further comprising, in the event of the crash of the target database, delaying queries to the target database until recovery of the target database is complete. The method of claim 1, wherein metadata defining the selected table is part of the recovery log file. Storing the permanently applied changes
2. The method of claim 1, further comprising waiting until a predetermined number of modifications are completed in the in-memory target database portion. 2. The method of claim 1, wherein the restoring the tables of the in-memory target database portion includes prioritizing recovery from the crash by one selected from the group consisting of data usage, query priority, and data priority. said prioritizing said recovery by said data usage;
maintaining a counter for each table in the target database, the counter value indicating how many queries are waiting on the associated table;
and restoring the database table with the highest counter value first in order of 1. said prioritizing said recovery by said query priority;
9. The method of claim 8, further comprising first restoring the database table that receives a query with the highest priority. said prioritizing said recovery by said data priority;
maintaining two groups of database tables, each group relating to a distinct group of users;
and restoring the database tables of the groups having higher configuration group priority first. determining a data volume to be restored for the next table to be restored;
10. The method of claim 1, further comprising: recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being an incremental update strategy or a bulk update strategy that recovers via a bulk load mechanism . A linked database system, the linked database including a source database and an associated target database, wherein selected portions of the contents of tables of the source database are synchronized with respective portions of the contents of the tables of the target database;
a processor and a memory communicatively coupled to the processor, wherein the processor synchronizes selected portions of the contents of the table of the source database with respective portions of the contents of a table of the target database;
During said synchronization, applying changes to said source database to an in-memory target database portion of a database management system that includes said target database;
asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage;
In the event of a crash of the target database , restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage;
and upon said crash of said target database, applying to said in-memory target database portion changes from a recovery log file of said source database having a timestamp later than the most recent snapshot available in said persistent target database storage. The synchronizing
14. The linked database system of claim 13, further comprising reading entries in the recovery log file relating to the source database and applying the read entries to the target database. The linked database system of claim 13, wherein the source database is optimized for transactions or is a row-oriented relational database management system. The linked database system of claim 13, wherein the target database is optimized for analytical operations or is a column-oriented database. In the event of the crash of the target database, the processor:
14. The linked database system of claim 13, wherein queries to the target database are delayed until recovery of the target database is complete. The linked database system of claim 13, wherein metadata defining the selected table is part of the recovery log file. Storing the permanently applied changes
14. The linked database system of claim 13, including waiting until a predetermined number of modifications are completed in the in-memory target database portion. 14. The linked database system of claim 13, wherein said restoring tables of said in-memory target database portion includes prioritizing said crash recovery by one selected from the group consisting of data usage, query priority, and data priority. said prioritizing said recovery by said data usage;
maintaining a counter for each table in the target database, the counter value indicating how many queries are waiting on the associated table;
and restoring the database table with the highest counter value first in order of priority. said prioritizing said recovery by said query priority;
21. The linked database system of claim 20, including first restoring the database table that receives a query with the highest priority. said prioritizing said recovery by said data priority;
maintaining two groups of database tables, each group relating to a distinct group of users;
21. The linked database system of claim 20, further configured to: first restore the database tables of the groups having higher constituent group priority. said processor determining a data volume to be restored for the next table to be restored;
14. The linked database system of claim 13, further comprising: recovering the table using a recovery strategy dependent on the volume to be recovered, the recovery strategy being an incremental update strategy or a bulk update strategy recovered via a bulk load mechanism . A computer program for crash recovery for linked databases, the computer program causing a processor to carry out a method according to any one of claims 1 to 12.