JP7704702B2 - Data migration management device, data migration management method and data migration management system - Google Patents

Description

The present disclosure relates to a data migration management device, a data migration management method, and a data migration management system.

A database management system is a tool used by a company or other organization to collect, store, and organize data. Broadly speaking, a database management system refers to a system for storing and managing data in a structured format, providing functionality for manipulating the data in a database and facilitating access to the data for use in various applications.

Online transaction processing systems (OLTP systems) and online analytical processing systems (OLAP systems) are two types of database processing systems that can be used to manage data in a database.

A database using an OLTP system (hereinafter "OLTP database") captures and maintains transaction data in a database. OLTP databases are designed to support fast query processing, real-time updates, high concurrency, and strong consistency. Each transaction requires an individual database record consisting of many fields or columns. Typically, queries only modify a few rows of data at a time. Examples of OLTP database applications include banking and credit card transactions or retail cash register scanning.

On the other hand, databases using OLAP systems (hereinafter "OLAP databases") apply complex queries to large amounts of historical data, including small amounts of transactions, that are aggregated from OLTP databases and other sources for data mining, analysis, and business intelligence projects. Queries to OLAP databases often require one or more columns of data aggregated from many rows. Examples of OLAP database applications include year-over-year financial performance or lead generation trends. OLAP databases and data warehouses give analysts and decision makers the ability to derive insights from data using custom reporting tools.

Typically, because of these differences, OLTP and OLAP queries are processed in separate databases. However, in recent years, the boundaries between transactional and analytical workloads have become blurred, with OLAP systems becoming more transactional in nature and OLTP systems becoming more analytical in nature. For example, to facilitate statistical analysis, it may be necessary to perform short, highly parallel queries on historical data stored in an OLAP database, fulfilling the small range queries typically performed in an OLTP system. Conversely, when performing transactions on the OLTP database for a given application, it may be necessary to perform large scale analyses on historical information stored in the OLAP database.

To facilitate the realization of a hybrid transactional analytical processing system (HTAP) that includes OLTP and OLAP processes, it is necessary to efficiently move data from the OLTP database to the OLAP database using extract-transform-store (ETL) techniques and other integration tools.

Traditionally, technologies have been considered to promote data movement in HTAP systems.

For example, Non-Patent Document 1 discloses that "IBM Db2 Analytics Accelerator (IDAA) is a state-of-the-art hybrid database system that seamlessly extends the high transaction performance of Db2 for z/OS with ultra-fast column store processing in the Db2 warehouse. The accelerator maintains a copy of data from Db2 for z/OS in a back-end database. Using replication techniques, data can be synchronized at a single point in time at the granularity of a table, one or more of its partitions, or incrementally as rows change. Since IDAA version 3, IBM Change Data Capture (CDC) has been adopted as the replication technology. In version 7.5.0, a superior replication technology - integrated synchronization - has been introduced as an alternative to IDAA's use of CDC. This paper shows how integrated synchronization improves performance by an order of magnitude and paves the way for near real-time hybrid transactional analytics (HTAP) processing."

Dennis Butterstein, Daniel Martin, Knut Stolze, Felix Beier, Jia Zhong, and Lingyun Wang, "Replication at the Speed of Change－a Fast, Scalable Replication Solution for Near Real-Time HTAP Processing," PVLDB, 13(12):3245-3257, 2020. DOI: https://doi.org/10.14778/3415478.3415548

Typically, in an HTAP system, data is moved from the OLTP database to the OLAP database periodically (e.g., once every 24 hours). However, in some cases, to perform analytical tasks or otherwise, the OLAP database needs access to the latest data that has not yet been moved from the OLTP database to the OLAP database. In such cases, the OLAP database may generate a federated query, which is a query by the OLAP database to request the transfer of a specific set of data in the form of a temporary table. However, such federated queries are computationally expensive and can increase the load on the OLTP database, potentially resulting in slower query processing times and reduced search performance.

Non-Patent Document 1 discloses a technique for accelerating HTAP processing by performing high-speed data replication from a high-performance transaction database system to a column store database system. However, in the technique described in Non-Patent Document 1, periodic migration timing is performed at regular intervals, and factors such as the network load and CPU load of the database system are not taken into account when determining the periodic migration timing.

In view of the above problems, it is therefore an object of the present disclosure to provide a data movement management technique for flexibly adjusting the timing of periodic data movement from an OLTP database to an OLAP database based on data freshness requirements, network and computing resource availability, and database status in order to facilitate search performance for analytical and transactional queries in an HTAP system.

One representative example of the present disclosure relates to a data movement management device for managing data in a hybrid transactional analysis architecture, the data movement management device including: a query log management unit for detecting a federated query requesting a first data set from the online transactional processing database from a first query log set of the online analytical processing database and identifying a first data freshness requirement of the online analytical processing database based on the federated query; a data movement management unit for determining a data management operation including at least copying the first data set from the online transactional processing database to the online analytical processing database based on at least the first data freshness requirement; and a data movement timing management unit for determining an updated periodic movement timing for transferring the bulk data from the online transactional processing database to the online analytical processing database when the frequency of the federated queries from the online analytical processing database reaches a first frequency threshold.

According to the present disclosure, it is possible to provide a data movement management technique for flexibly adjusting the timing of periodic data movement from an OLTP database to an OLAP database based on data freshness requirements, network and computing resource availability, and database status to facilitate search performance for analytical and transactional queries in an HTAP system.

Problems, configurations, and advantages other than those described above will become apparent from the following description of an embodiment for carrying out the invention.

FIG. 1 is a block diagram illustrating an example computing architecture for implementing embodiments of the present disclosure. FIG. 1 is a diagram illustrating an example of a configuration of a data migration management system according to an embodiment of the present disclosure. 1 is a block diagram showing a functional configuration of a data migration management system according to an embodiment of the present disclosure. FIG. 2 is a diagram showing an example of an operation flow in an HTAP system. FIG. 13 is a diagram illustrating an example of a data copy operation flow in a data migration management system according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of a data deletion operation flow in a data movement management system according to an embodiment of the present disclosure. 1 is a flowchart illustrating a flow of a data movement management method according to an embodiment of the present disclosure. FIG. 2 illustrates a specific example of determining a data movement management operation according to an embodiment of the present disclosure. 1 is a flowchart illustrating a data copy operation flow according to an embodiment of the present disclosure. 1 is a flowchart illustrating a data copy and delete operation flow according to an embodiment of the present disclosure. 1 is a flowchart illustrating a data deletion operation flow according to an embodiment of the present disclosure. 1 is a flowchart illustrating a periodic data movement timing update process flow according to an embodiment of the present disclosure. 10 is a flowchart illustrating a data management operation deferral process flow according to an embodiment of the present disclosure.

In this specification, embodiments of the present invention are described with reference to the figures. It should be noted that the embodiments described herein are not intended to limit the invention as claimed, and it should be understood that each of the elements and combinations thereof described with respect to the embodiments are not strictly necessary to implement aspects of the present invention.

Various embodiments are disclosed in the following description and associated figures. Alternative embodiments may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure.

The words "exemplary" and/or "example" are used herein to mean "serving as an example, illustration, or illustration." Any aspect described herein as "exemplary" and/or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Similarly, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the described feature, advantage or mode of operation.

Furthermore, many aspects are described as sequences of actions to be performed by, for example, elements of a computing device. It will be appreciated that the various actions described herein may be performed by specialized circuitry (e.g., an application specific integrated circuit (ASIC)), by program instructions executed by one or more processors, or by a combination of both.

Additionally, the sequences of operations described herein may be considered to be embodied entirely in the form of a computer-readable storage medium having stored thereon a corresponding set of computer instructions that, when executed, cause an associated processor to perform the functionality described herein. Thus, various aspects of the disclosure may be embodied in a number of different forms, all of which are contemplated to be within the scope of the claimed subject matter.

1 depicts a high level block diagram of a computer system 100 according to an embodiment for implementing various embodiments of the present disclosure. The mechanisms and apparatus of the various embodiments disclosed herein are equally applicable to corresponding computing systems. The main components of the computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, input/output (I/O) device interface 114, and network interface 115, all of which are communicatively coupled, directly or indirectly, for inter-component communication via memory bus 106, I/O bus 108, bus interface unit 109, and I/O bus interface unit 110.

Computer system 100 may contain one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to herein as processors 102. In embodiments, computer system 100 may contain multiple processors, however, in some embodiments, computer system 100 may alternatively be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include one or more levels of on-board cache.

In embodiments, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing or encoding data and programs. In some embodiments, memory 104 represents the entire virtual memory of computer system 100, and may also include virtual memory of other computer systems coupled to computer system 100 or connected via a network. While memory 104 may be conceptually considered a single monolithic entity, in other embodiments, memory 104 is a more complex arrangement, such as a hierarchy of caches and other memory devices. For example, memory may exist at multiple cache levels, and these caches may be further divided by function, such that one cache holds instructions while another holds non-instruction data used by one or more processors. Memory may be further distributed and associated with different CPUs or sets of CPUs, as known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures.

Memory 104 may store all or a portion of various programs, modules, and data structures for handling data transfers as described herein. Illustratively, memory 104 may store data movement management application 150. In an embodiment, data movement management application 150 may include instructions or statements that are executed by processor 102 or that are interpreted by processor 102 to perform functions as described further below.

In some embodiments, the data movement management application 150 is implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, the data movement management application 150 may include data in addition to instructions or statements. In some embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with the bus interface unit 109, the processor 102, or other hardware of the computer system 100. In such a configuration, the need for the processor 102 to access the memory 104 and the latent factor identification application is alleviated.

The computer system 100 may include a bus interface unit 109 that handles communication between the processor 102, memory 104, display system 124, and an input/output bus interface unit 110. The input/output bus interface unit 110 may be coupled to an input/output bus 108 for transferring data to and from various input/output units. The input/output bus interface unit 110 communicates through the input/output bus 108 with a number of input/output interface units 112, 113, 114, 115, also known as input/output processors (IOPs) or input/output adapters (IOAs). The display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both types of data to the display device 126. Additionally, the computer system 100 may include one or more sensors or other devices configured to collect and provide data to the processor 102.

By way of example, computer system 100 may include biometric sensors (e.g., collecting heart rate data, stress level data), environmental sensors (e.g., collecting humidity data, temperature data, pressure data), motion sensors (e.g., collecting acceleration data, movement data), and so forth. Other types of sensors are possible. Display memory may be dedicated memory for buffering video data. Display system 124 may be coupled to a display device 126, such as a stand-alone display screen, a computer monitor, a television, or a tablet or handheld device display.

In one embodiment, the display device 126 may include one or more speakers for providing audio. Alternatively, one or more speakers for providing audio may be coupled to the input/output interface unit. In an alternative embodiment, one or more of the functions provided by the display system 124 may be implemented in an integrated circuit that also includes the processor 102. Additionally, one or more of the functions provided by the bus interface unit 109 may be implemented in an integrated circuit that also includes the processor 102.

The input/output interface unit supports communication with a variety of storage and input/output devices. For example, the terminal interface unit 112 supports attachment of one or more user input/output devices 116, which may include user output devices (such as a video display device, speakers, and/or a television) and user input devices (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). A user may use the user input devices 116 and a user interface to provide input data and commands to the computer system 100 and receive output data via the user output devices. The user interface may be presented via the user input/output devices 116, for example, displayed on a display device, emitted via speakers, or printed via a printer.

The storage interface 113 supports attachment of one or more disk drives or direct access storage devices 117 (typically rotating magnetic disk drive storage devices, but alternatively may be other storage devices including an array of disk drives configured to be presented to the host computer as a single large storage device or as a solid state drive such as flash memory). In some embodiments, the storage device 117 may be implemented via some type of secondary storage device. The contents of the memory 104, or any portion thereof, may be stored in the storage device 117 and retrieved as needed. The input/output device interface 114 provides an interface to any of a variety of other input/output devices, or other types of devices such as a printer or fax machine. The network interface 115 provides one or more communication paths from the computer system 100 to other digital devices and computer systems. These communication paths may include, for example, one or more networks 130.

1 shows a particular bus structure that provides direct communication paths between the processor 102, memory 104, bus interface 109, display system 124, and I/O bus interface unit 110, but in alternative embodiments, computer system 100 may include a variety of buses or communication paths that may be arranged in any of a variety of configurations, such as hierarchical star or web configurations, multiple hierarchical buses, parallel and redundant paths, or point-to-point links in any other suitable type of configuration. Furthermore, while I/O bus interface unit 110 and I/O bus 108 are shown as a single such unit, computer system 100 may actually contain multiple I/O bus interface units 110 and/or multiple I/O buses 108. Although multiple I/O interface units are shown isolating I/O bus 108 from various communication paths to various I/O devices, in other embodiments, some or all of the I/O devices are directly connected to one or more system I/O buses.

In various embodiments, computer system 100 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface and accepts requests from other computer systems (clients). In other embodiments, computer system 100 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable type of electronic device.

Next, an example configuration of a data migration management system according to an embodiment of the present disclosure will be described with reference to FIG. 2.

Figure 2 shows an example configuration of a data movement management system 200 according to an embodiment of the present disclosure. The data movement management system 200 is a system for flexibly adjusting the periodic movement timing at which data is moved from the OLTP data to the OLAP database. As shown in Figure 2, the data movement management system 200 includes an OLTP device 210, an OLAP device 220, and a data movement management device 240 that are communicatively connected by a communication network 230. Here, the communication network 230 may include a local area network (LAN) connection, the Internet, a wide area network (WAN) connection, a metropolitan area network (MAN) connection, and the like.

OLTP device 210 is a hardware device that supports the operation of OLTP database 214. As shown in FIG. 2, OLTP device 210 may include a CPU 211, memory 212, a storage device 213 for storing OLTP database 214, and a network interface 215. In an embodiment, OLTP device 210 may be implemented using a computer system such as computer system 100 shown in FIG. 1. As described herein, OLTP database 214 may include a database configured to support fast query processing, real-time updates, high concurrency, and strong consistency. Each transaction requires an individual database record consisting of many fields or columns. Generally, a query modifies only a few rows of data at a time. Examples of OLTP database applications include banking and credit card transactions or retail cash register scanning.

OLAP device 220 is a hardware device that supports the operation of OLAP database 224. As shown in FIG. 2, OLAP device 220 may include a CPU 221, memory 222, a storage device 223 for storing OLAP database 224, and a network interface 225. In an embodiment, OLAP device 220 may be implemented using computer system 100, such as the computer system shown in FIG. 1. As described herein, OLAP database 224 may be configured to apply complex queries with low transaction volume against large amounts of historical data aggregated from OLTP databases (e.g., OLTP database 214) and other sources for data mining, analysis, and business intelligence projects. A query to OLAP database 224 may involve one or more columns of data aggregated from multiple rows. Examples of OLAP database applications include year-over-year financial performance or lead development trends.

The data movement management device 240 is a hardware device for managing the movement of data from the OLTP database 214 of the OLTP device 210 to the OLAP database 224 of the OLAP device 220. As shown in FIG. 2, the data movement management device 240 may include a CPU 241, a memory 242, a storage device 243, and a network interface 244. In an embodiment, the OLAP device 220 may be implemented using a computer system such as the computer system 100 shown in FIG. 1.

It should be noted that the functions of the data movement management device 240 will be described in detail later, so their description will be omitted here.

The data migration management system 200 configured as shown in FIG. 2 makes it possible to flexibly adjust the timing of periodic migration of data from the OLTP database to the OLAP database.

2 shows an example configuration in which the OLTP device 210 for storing the OLTP database 214, the OLAP device 220 for storing the OLAP database 224, and the data movement management device 240 are implemented as separate hardware devices connected via the communication network 230, but it should be noted that the present disclosure is not limited thereto. For example, the OLTP database 214 and the OLAP database 224 can be stored on the same hardware device and managed by the data movement management device 240 via the communication network 230, or the functionality of the OLTP database 214, the OLAP database 224, and the data movement management device 240 can all be implemented on the same hardware device.

Next, the functional configuration of the data migration management system 200 is described with reference to Figure 3.

Figure 3 is a block diagram showing the functional configuration of the data movement management system 200. As described in this specification, the data movement management system 200 includes an OLTP device 210, an OLAP device 220, and a data movement management device 240 that are communicatively connected by a communication network 230.

The OLTP device 210 includes OLTP database load data 312, an OLTP query log file 314, and an OLTP database 214. The OLTP database load data 312 is a collection of data representing CPU resource availability, memory resource availability, average data reads per second, average data writes per second, and other usage information for the OLTP database 214. The OLTP query log file 314 represents read and write queries for the OLTP database 214.

The OLAP device 220 includes OLAP database load data 322, an OLAP query log file 324, and an OLAP database 224. The OLAP database load data 322 is a collection of data representing CPU resource availability, memory resource availability, average data reads per second, average data writes per second, and other usage information for the OLAP database 224. The OLAP query log file 324 represents read and write queries for the OLAP database 224.

The data movement management device 240 includes a table status database 341, a table status management unit 342, a query log management unit 343, a resource management unit 344, a data movement management unit 345, and a data movement timing management unit 346.

Table status database 341 is a database for information representing data stored in OLTP database 214 and OLAP database 224. In an embodiment, table status database 341 may include status information representing data freshness requirements for OLTP database 214 and OLAP database 224. Generally, data freshness refers to how up-to-date (i.e., current) a particular set of data is. Different databases may have different requirements regarding data freshness for the data they store. These requirements, which define the limits of data freshness, are referred to herein as data freshness requirements. In an embodiment, these data freshness requirements may be defined by the application that utilizes the data stored in a particular database. As an example, a banking application may have a data freshness requirement that requires the data to be up-to-date as of the present, whereas a statistical analysis application may have a data freshness requirement that requires the data to be up-to-date for the past six months.

The table status management unit 342 is a functional unit for managing and updating the table status database 341. When new information is added to the OLTP database 214 or the OLAP database 224, or when data freshness requirements change, the table status management unit 342 may update the table status database 341. The table status management unit 342 may determine which data is stored in the OLTP database 214 and which data is stored in the OLAP database 224.

The query log management unit 343 is a functional unit for monitoring and analyzing the query logs (i.e., the OLTP query log file 314 and the OLAP query log file 324) of the OLTP database 214 and the OLAP database 224. In an embodiment, the query log management unit may determine the data freshness requirements of a particular database or a particular data request by analyzing the query log of the database. As an example, the query log management unit 343 may detect a federated query from the OLAP database 224 requesting a first set of data from the online transaction processing database 214 from the OLAP query log file 324, and identify a first data freshness requirement of the OLAP database based on the detected federated query. As described herein, the first data freshness requirement may be utilized in determining the appropriate time to perform a periodic timing of data movement from the OLTP database 214 to the OLAP database 224.

The resource management unit 344 is a functional unit for monitoring and analyzing the load and availability resources of the OLTP database 214 and the OLAP database 224. In an embodiment, the resource management unit 344 may access and analyze the OLTP database load data 312 and the OLAP database load data 322 to determine CPU resource availability, memory resource availability, average data reads per second, average data writes per second, and other usage information for each database. Additionally, the resource management unit may perform diagnostics on the communication network 230 to obtain information such as download speeds, upload speeds, and availability of the communication network 230 for data transfer.

The data movement management unit 345 is a functional unit for determining and implementing data management operations to facilitate the movement of data from the OLTP database 214 to the OLAP database 224. Here, the data management operations may include copy operations in which data from the OLTP database 214 is copied to the OLAP database 224 while remaining in the OLTP database 214, or copy-and-delete operations in which data from the OLTP database 214 is copied to the OLAP database 224 and deleted from the OLTP database 214. In an embodiment, the determination of the data management operation may be based on at least the data freshness requirements of the OLTP database 214 and/or the OLAP database 224.

The data movement timing management unit 346 is a functional unit for determining an appropriate periodic movement timing for moving bulk data from the OLTP database 214 to the OLAP database 224. In an embodiment, the data movement timing management unit 346 may determine based on the frequency of federated queries from the OLAP database 224. As an example, the data movement timing management unit 346 may determine to decrease the interval during periodic data movement when the frequency of federated queries from the OLAP database 224 is high in order to provide the OLAP database 224 with the latest data more frequently, thereby reducing the number of federated queries and reducing the processing load on the OLTP database 214. Alternatively, the data movement timing management unit 346 may determine to increase the interval during periodic movement when the data freshness requirements of the OLAP database 224 are relaxed and the latest data is not required, thereby reducing the processing load on the OLTP database 214.

The data movement management system 200 configured as shown in FIG. 3 can facilitate faster query processing times and improved search performance by flexibly adjusting the timing of periodic data movement from the OLTP database to the OLAP database.

Next, an example of the operation flow in the HTAP system is described with reference to Figure 4.

Figure 4 shows an example of an operational flow in the HTAP system.

Initially, the OLTP database 214 accepts data in the form of an insert query 410 from sensor devices such as sensors 401 and 402 via the gateway 405. The OLTP database 214 processes the insert query 410 and inserts and stores the received data in the OLTP database 214. The insert query 410 may be continuously streamed to the gateway 405 as the sensors 401, 402 collect measurements as part of a monitoring application. Upon receipt of a read query 420, the OLTP database 214 retrieves the requested data and provides it to the transaction application 430 for display in an interface. Since the OLTP database 214 is designed to support fast access to short-term data, by way of example, the transaction application 430 may display time series data collected by the sensors 401, 402 for the last 30 minutes in a graph or chart, show average measurements over 5-minute intervals, show measurements of a particular sensor at a particular time, etc.

At a predetermined data movement timing (e.g., once every 24 hours), data is moved from the OLTP database 214 to the OLAP database 224. However, as described herein, in some cases, the OLAP database 224 may receive an analytical query 450 from the analytical application 440 that specifies data that has reached a data freshness requirement that requires newer (e.g., more current) data than the data transferred and stored in the OLAP database 224. That is, the analytical query 450 requires the use of data that has not yet been transferred from the OLTP database 214 to the OLAP database 224. In such cases, the OLAP database 224 may generate a federated query, which is a query in which the OLAP database 224 requests the transfer of a specified set of data from the OLTP database 214 to fulfill the analytical query 450. However, such federated queries are computationally expensive and increase the load on the OLTP database 214, potentially resulting in slower query processing times and reduced search performance.

Accordingly, aspects of the present disclosure relate to providing data movement management techniques to flexibly adjust the timing of periodic data movement from an OLTP database to an OLAP database based on data freshness requirements, network and computing resource availability, and database status to facilitate search performance for analytical and transactional queries in an HTAP system. In this way, the OLAP database 224 may be kept up to date while reducing the frequency of federated queries and the processing load on the OLTP database 214.

Next, an example of the flow of a data copy operation in a data migration management system according to an embodiment of the present disclosure will be described with reference to FIG. 5.

As described herein, the data movement management device 240 is configured to determine and implement data management operations to facilitate the movement of data from the OLTP database 214 to the OLAP database 224. In an embodiment, the data management operation may include a data copy operation to copy a particular data set from the OLTP database 214 to the OLAP database 224. Additionally, in an embodiment, the data management operation may include a data copy and delete operation to copy a particular data set from the OLTP database 214 to the OLAP database 224 and subsequently delete the copied data from the OLTP database 214. FIG. 5 illustrates an example of a flow of a data copy operation in the data movement management system 200 according to an embodiment of the present disclosure.

As shown in FIG. 5, the data movement management device 240 may obtain information including a first data freshness requirement of the OLAP database 224, OLAP database load data, network status information of the communication network 230 (not shown in FIG. 5), OLTP database load data, and a second data freshness requirement of the OLTP database 214.

The first data freshness requirement includes information that indicates how recent a particular data set must be for use in responding to a query on the OLAP database 224. In an embodiment, the first data freshness requirement may be expressed as a date or date range that a particular data set must meet. As an example, the first data freshness requirement may include a date range of "between [09:30:00 on Jan. 7, 2019] and [11:00:00 on Mar. 31, 2021]." If the OLAP database 224 does not include a date until [11:00:00 on Mar. 31, 2021], it may be necessary for the OLAP database 224 to request data from the OLTP database 214 in response to a query that includes the first data freshness requirement. In an embodiment, the data movement management device may determine the first data freshness requirement by extracting the first data freshness requirement from a query (e.g., a federated query) in the OLAP query log file 324.

OLAP database load data 322 is a collection of data representing CPU resource availability, memory resource availability, average data reads per second, average data writes per second, and other usage information for OLAP database 224. Similarly, OLTP database load data 312 is a collection of data representing CPU resource availability, memory resource availability, average data reads per second, average data writes per second, and other usage information for OLTP database 214.

Network status information 506 is a collection of information including download and upload speeds and the availability of communication network 230 for transferring data. In an embodiment, data movement management device 240 may determine network status information 506 by performing a series of diagnostics on communication network 230.

The second data freshness requirement includes information representing how recent a particular data set must be for use in responding to a query on the OLTP database 214. In an embodiment, the second data freshness requirement may be expressed as a date or date range that a particular data set must meet. In an embodiment, the data movement management device 240 may determine the second data freshness requirement by extracting the second data freshness requirement from a query in the OLTP query log file 314.

Upon obtaining the first data freshness requirement of the OLAP database 224, the OLAP database load data, the network status information of the communication network 230 (not shown in FIG. 5), the OLTP database load data, and the second data freshness requirement of the OLTP database 214, the data movement management device may determine the data set to be copied from the OLTP database 214 to the OLAP database as part of the data copy operation, along with the timeliness for the data copy operation. The timeliness for the data copy operation may be determined based on the first data freshness requirement of the OLAP database 224, the OLAP database load data, the network status information of the communication network 230, the OLTP database load data, and the second data freshness requirement of the OLTP database 214. A detailed description of the determination of the timing of the data management operation will be given later, and therefore the description will be omitted here.

After determining the set of data to be copied from the OLTP database 214 to the OLAP database as part of the data copy operation along with the timeliness for the data copy operation, the data movement management device 240 may read the table 510 that stores the requested data in the OLTP database 214, generate a CSV file to store the data, perform a bulk import of the CSV file into the specified table 520 in the OLAP database 224, commit the data, and update the table status database 341 to reflect the newly added data.

Next, an example of the flow of a data deletion operation in a data movement management system according to an embodiment of the present disclosure will be described with reference to FIG. 6.

FIG. 6 is a diagram illustrating an example of the flow of a data deletion operation in a data movement management system according to an embodiment of the present disclosure. In an embodiment, the data deletion operation shown in FIG. 6 may be shown following the data copy operation shown in FIG. 5 as part of a data copy and delete operation.

The data movement management device 240 may first determine the data set to be deleted from the OLTP database 214 along with the appropriate time for the data deletion operation. In an embodiment, the data movement management device 240 may determine to delete the data set previously copied from the OLTP database 214 to the OLAP database 224. The appropriate time for the data deletion operation may be determined based on the first data freshness requirement of the OLAP database 224, the OLAP database load data, the network status information of the communication network 230, the OLTP database load data, and the second data freshness requirement of the OLTP database 214. A detailed description of the determination of the timing of the data management operation will be given later, and the description thereof will be omitted here.

The data movement management device 240 may then delete the determined data from the table 610 of the OLTP database 214 in which it was stored, and update the table status database 341 to reflect the deleted data.

The data movement management operations described above with reference to Figures 5 and 6 allow the data movement management device 240 to keep the OLAP database 224 up to date while reducing the frequency of federated queries and the processing load on the OLTP database 214.

Next, an example of a data movement management method according to an embodiment of the present disclosure will be described with reference to FIG. 7.

FIG. 7 is a flowchart illustrating a flow of a data movement management method 700 according to an embodiment of the present disclosure. The data movement management method 700 is a method for determining and performing data movement management operations based on data freshness requirements of the OLTP database and the OLAP database, and determining the timing of updated periodic data movement to promote query processing performance.

Initially, in step S705, the query log management unit 343 of the data movement management device 240 monitors the OLTP query log file (OLTP query log file 314) and the OLAP query log file (OLAP query log file 324) for federated queries. The query log management unit 343 may detect the federated query when a query in the OLAP query log file relates to (e.g., requires) data that is not available in the OLAP database.

Next, in step S710, the query log management unit 343 may determine whether a federated query from the OLAP database has been detected. If a federated query from the OLAP database has been detected, processing proceeds to step S715. Conversely, if a federated query from the OLAP database has not been detected, processing returns to step S705 to continue monitoring the query log file.

Next, if a federated query is detected, in step S715, the query log management unit 343 may identify a first data freshness requirement for the OLAP database based on the federated query. As described herein, the first data freshness requirement includes information that indicates how recent a particular data set must be for use in responding to a query of the OLAP database 224. In an embodiment, the first data freshness requirement may be expressed as a date or date range that a particular data set must meet. As an example, the first data freshness requirement may include a date range of "between [January 7, 2019 09:30:00] and [March 31, 2021 11:00:00]". The query log management unit 343 may identify the first data freshness requirement by extracting it from the detected federated query.

Next, in step S720, the data movement management unit 345 of the data movement management device 240 determines whether the amount of data requested by the federated query meets the data amount threshold. The amount of data requested by the federated query may be determined based on the first data freshness requirement identified in step S715. For example, the data movement management unit 345 may use the identified first data freshness requirement to reference the OLTP database (and/or the table status database) to ascertain the amount of data that needs to be moved from the OLTP database to the OLAP database to satisfy the federated query.

Here, the data volume threshold is a predetermined value that specifies a particular amount of data. The data volume threshold may be expressed as a particular data size (5 megabytes, 100 megabytes, 1 gigabyte) or as a number of data rows or columns (50 rows, 100 rows, 50 columns). When the amount of data requested by the federated query is less than the data volume threshold, the federated query is considered to satisfy the data volume threshold. In this case, the amount of data requested by the federated query is substantially small, so that the federated query may be executed without significantly increasing the load on the OLTP database. Conversely, when the amount of data requested by the federated query is large (e.g., the federated query does not satisfy the data volume threshold), it is desirable to perform data management operations to provide the requested data to the OLAP database and modify the periodic data movement timing to reduce the frequency of future federated queries.

Therefore, if the federated query meets the data volume threshold, processing proceeds to step S725. Conversely, if the federated query does not meet the data volume threshold, processing proceeds to step S730.

In step S725, the data movement management unit 345 executes the federated query to obtain data from the OLTP database required to process the federated query in the OLAP database.

In step S730, the data movement management unit 345 may identify a second data freshness requirement for the OLTP database based on an OLTP query log file (e.g., OLTP query log file 314). The second data freshness requirement includes information representing how recent a particular data set must be for use in responding to a query of the OLTP database. In an embodiment, the second data freshness requirement may be expressed as a date or date range that a particular data set must meet. In an embodiment, the data movement management unit 345 may determine the second data freshness requirement by analyzing queries recorded in an OLTP query log file that represents the most recently accessed data at the OLTP database (e.g., the most recently accessed data at the OLTP database is the data that is also required by the OLTP database to process the query).

Next, in step S735, the data movement management unit 345 may compare the first data freshness requirement identified in step S715 with the second data freshness requirement identified in step S730 to determine whether the first data freshness requirement corresponds to the second data freshness requirement. Here, the data movement management unit 345 may determine that the first data freshness requirement corresponds to the second data freshness requirement if the date range specified by the first data freshness requirement overlaps with at least a portion of the date range specified by the second data freshness requirement. Conversely, the data movement management unit 345 may determine that the first data freshness requirement does not correspond to the second data freshness requirement if the date range specified by the first data freshness requirement does not overlap with at least a portion of the date range specified by the second data freshness requirement.

If it is determined that the first and second data freshness requirements correspond, the process proceeds to step S745. Conversely, if the first and second data freshness requirements do not correspond, the process proceeds to step S740.

In step S740, the data movement management unit 345 may determine to perform a copy-and-delete operation as the data management operation, which copies the first data set requested by the federated query from the OLTP database to the OLAP database and deletes the first data set from the OLTP database. That is, since the data freshness requirements of the OLTP database and the OLAP database do not overlap, the first data set requested by the federated query is no longer in use by the OLTP database and therefore may be deleted from the OLTP database. The data copy-and-delete operation will be described later, and a detailed description thereof will be omitted here.

In step S745, the data movement management unit 345 may determine to perform a copy operation as a data management operation to copy the first data set requested by the federated query from the OLTP database to the OLAP database. That is, since the data freshness requirements of the OLTP database and the OLAP database overlap, the first data set requested by the federated query is still in use by the OLTP database and therefore should be retained in the OLTP database. The data copy operation will be described later, so a detailed description thereof will be omitted here.

In step S750, the data movement management unit 345 determines whether the frequency of the federated queries has reached a frequency threshold. Here, the frequency threshold defines a predetermined issuance rate of federated queries and may be expressed as the number of federated queries issued at the OLAP database per given time (e.g., 1 federated query per hour, 10 federated queries per day). If the frequency of federated queries calculated by the data movement management unit 345 for the OLAP database at a particular time is greater than the frequency threshold, the frequency of the federated queries is considered to have reached the frequency threshold. If the frequency of the federated queries has reached the frequency threshold, the process proceeds to step S755. Conversely, if the frequency of the federated queries has not reached the frequency threshold, the process returns to step S705 to continue monitoring the query log files of the OLTP database and the OLAP database.

In step S755, the data movement timing management unit 346 determines an updated periodic movement timing for transferring bulk data from the OLTP database to the OLAP database. That is, if the frequency of federated queries has reached a frequency threshold, it is desirable to modify the periodic movement timing to provide the latest data to the OLAP database and reduce the number of federated queries issued at the OLAP database. The process for determining the updated periodic data movement timing will be described later, and a detailed description thereof will be omitted here.

According to the data movement management method 700 illustrated in FIG. 7, it is possible to flexibly adjust the periodic movement timing of data from the OLTP database to the OLAP database based on data freshness requirements, network and computing resource availability, and database status to facilitate search performance of analytical and transactional queries in the HTAP system. In this way, the OLAP database 224 can be kept up to date while reducing the frequency of federated queries and the processing load of the OLTP database 214.

Next, a specific example of determining a data movement management operation according to an embodiment of the present disclosure will be described with reference to FIG. 8.

As described herein, the data movement management device 240 according to an embodiment of the present disclosure is configured to determine a data movement management operation based on data freshness requirements of the OLAP database and the OLTP database. FIG. 8 illustrates a specific example of determining a data movement management operation according to an embodiment of the present disclosure.

The query log management unit 343 of the data movement management device 240 may analyze the query log stored in the OLAP query log file 324 of the OLAP database to identify a first data freshness requirement 810 for the OLAP database. For example, the first data freshness requirement 810 may specify a date range of "January 7, 2019, 09:30 to March 31, 2021, 11:00." In an embodiment, the query log management unit 343 may analyze the OLAP query log file 324 upon detection of a federated query from the OLAP database.

The query log management unit 343 may then analyze the query logs stored in the OLTP query log file 314 of the OLTP database to identify a second data freshness requirement 820 for the OLTP database. In an embodiment, the query log management unit 343 may identify the second data freshness requirement 820 based on a query representing the most recently accessed data in the OLTP database. As an example, the second data freshness requirement 820 may specify a date range of "January 7, 2021, 09:30 to January 7, 2021, 11:00."

Then, the data movement management unit 345 of the data movement management device 240 may compare the first data freshness requirement 810 with the second data freshness requirement 820 to determine whether the first data freshness requirement 810 corresponds to the second data freshness requirement 820. Here, the data movement management unit 345 may determine that the first data freshness requirement 810 corresponds to the second data freshness requirement 820 if the date range specified by the first data freshness requirement 810 overlaps with at least a portion of the date range specified by the second data freshness requirement 820. Conversely, the data movement management unit 345 may determine that the first data freshness requirement 810 does not correspond to the second data freshness requirement 820 if the date range specified by the first data freshness requirement 810 does not overlap with at least a portion of the date range specified by the second data freshness requirement 820.

When the first data freshness requirement 810 corresponds to the second data freshness requirement 820, the data movement management unit 345 may determine that a copy operation of copying the first data set from the OLTP database to the OLAP database is to be performed as the data management operation. Conversely, when the first data freshness requirement 810 does not correspond to the second data freshness requirement 820, the data movement management unit 345 may determine that a copy/delete operation of copying the first data set from the OLTP database to the OLAP database and deleting the first data set from the OLTP database is to be performed as the data management operation.

In this example, since the first data freshness requirement 810 of "09:30, Jan. 7, 2019 to 11:00, Mar. 31, 2021" overlaps with the second data freshness requirement 820 of "09:30, Jan. 7, 2021 to 11:00, Jan. 7, 2021" (e.g., both data freshness requirements require data from 09:30, Jan. 7, 2021 to 11:00, Jan. 7, 2021), the data movement management unit 345 may determine to perform a copy operation to copy the first data set from the OLTP database to the OLAP database as the data management operation.

Next, an example of a data copy operation according to an embodiment of the present disclosure will be described with reference to FIG. 9.

FIG. 9 is a flow chart illustrating a flow of a data copy operation 900 according to an embodiment of the present disclosure. The data copy operation 900 is a method for copying a data set from an OLTP database to an OLAP database. The data copy operation 900 may be performed to provide requested data to the OLAP database to avoid the need to execute a federated query that may negatively impact resource usage of the OLTP database. The data copy operation 900 illustrated in FIG. 9 substantially corresponds to step S745 of FIG. 7.

First, in step S905, the resource management unit 344 analyzes the OLTP database load data (e.g., OLTP database load data 312) and the OLAP database load data (e.g., OLAP database load data 322) to determine the availability of computing resources at both the OLTP database and the OLAP database. For example, the resource management unit 344 may determine the availability of CPU resources and memory resources at the OLTP database and the OLAP database based on the OLTP database load data and the OLAP database load data.

Next, in step S910, the resource management unit 344 determines whether sufficient computing resources are available in both the OLTP database and the OLAP database to perform the data copy operation. In an embodiment, the data movement management unit 345 may determine whether sufficient computing resources are available by comparing the available computing resources of the OLTP and OLAP databases determined in step S905 with predetermined specification information that specifies the computing resources required for the data copy operation. If sufficient computing resources are available, the process proceeds to step S915. If sufficient resources are not available, the data movement management unit 345 waits a predetermined time and returns to step S905.

Next, in step S915, the data movement management unit 345 may generate a CSV file containing the first data set requested by the federated query detected in step S710 of the data movement management method 700.

Next, in step S920, the resource management unit 344 may analyze the communication network (e.g., the communication network 230 shown in FIGS. 2 and 3) to determine the availability of network resources and perform the data copy operation. In an embodiment, the resource management unit 344 may determine the availability of network resources by performing one or more diagnostics on the communication network 230 to check download speeds, upload speeds, availability, etc.

Next, in step S925, the resource management unit 344 may determine whether sufficient network resources are available to perform the data copy operation. In an embodiment, the resource management unit 344 may compare the network resources determined in step S920 to predefined specification information that specifies the network resources required to perform the data copy operation for a given data size. If sufficient network resources are available, processing proceeds to step S930. If sufficient resources are not available, the data movement management unit 345 waits a predefined period of time and returns to step S920.

In step S930, the data movement management unit 345 may transfer the CSV file generated in step S915 from the OLTP database (e.g., from the OLTP device 210) to the OLAP device (e.g., to the OLAP device 220) via the communications network 230. In an embodiment, the transfer of the CSV file may occur via a secure or encrypted network channel.

In step S935, the data movement management unit 345 may update the table status database 341 to reflect the data in each of the OLTP and OLAP databases after the transfer of the CSV file (e.g., update the status of the OLAP database to include the transferred data).

The data copy operation 900 transfers the necessary data from the OLTP database to the OLAP database, making it possible to keep the OLAP database up to date while reducing the frequency of federated queries and the processing load on the OLTP database.

Next, an example of a data copy/delete operation according to an embodiment of the present disclosure will be described with reference to FIG. 10.

FIG. 10 is a flow chart illustrating a flow of a data copy/delete operation 1000 according to an embodiment of the present disclosure. The data copy/delete operation 1000 is a method for copying a data set from an OLTP database to an OLAP database and subsequently deleting the copied data from the OLTP database. Performing the data copy/delete operation 1000 to provide requested data to the OLAP database and subsequently deleting the copied data from the OLTP database may also facilitate fast query processing of the OLTP database while avoiding the need to perform federated queries that may negatively impact resource usage of the OLTP database. The data copy/delete operation 1000 illustrated in FIG. 10 substantially corresponds to step S740 of FIG. 7.

Steps S1005, S1010, S1015, S1020, S1025, S1030, and S1035 shown in FIG. 10 substantially correspond to the steps shown in FIG. 9, and therefore, for ease of explanation, redundant descriptions of these steps are omitted here.

In step S1040, the data movement management unit 345 may determine data to be deleted from the OLTP database. In an embodiment, the data movement management unit 345 may determine to delete the first set of data requested by the federated query of the OLAP database and transferred from the OLTP database to the OLAP database in step S1030.

In step S1045, the data movement management unit 345 may perform a data deletion operation to delete the data determined in step S1040 from the OLTP database. The data deletion operation will be described later, and a detailed description thereof will be omitted here.

Next, an example of a data deletion operation according to an embodiment of the present disclosure will be described with reference to FIG. 11.

FIG. 11 is a flow chart illustrating a flow of a data deletion operation 1100 according to an embodiment of the present disclosure. The data deletion operation is a method for deleting from the OLTP database a data set that was copied from the OLTP database to the OLAP database. The data deletion operation 1100 shown in FIG. 11 substantially corresponds to step S1045 of FIG. 10.

First, in step S1105, the resource management unit 344 analyzes the OLTP database load data (e.g., the OLTP database load data 312) to determine the availability of computing resources at the OLTP database. For example, the resource management unit 344 may determine the availability of CPU resources and memory resources at the OLTP database based on the OLTP database load data.

Next, in step S1110, the resource management unit 344 determines whether sufficient computing resources are available in the OLTP database to perform the data deletion operation. In an embodiment, the resource management unit 344 may determine whether sufficient computing resources are available by comparing the available computing resources of the OLTP database determined in step S1105 with predefined specification information that specifies the computing resources required for the data deletion operation. If sufficient computing resources are available, processing proceeds to step S1115. If sufficient resources are not available, the resource management unit 344 waits a predefined time and returns to step S1105.

Next, in step S1115, the data movement management unit 345 identifies the data to be deleted. In an embodiment, the data movement management unit 345 may identify the data in the OLTP database determined in step S1040 of the data copy and delete operation 1000 (e.g., the first set of data requested by the federated query of the OLAP database and transferred from the OLTP database to the OLAP database).

Next, in step S1120, the data movement management unit 345 deletes the data identified in step S1115 from the OLTP database.

Next, in step S1125, the data movement management unit 345 may update the table status database 341 to reflect the data in the OLTP database after the data deletion.

The data copy/delete operation 1000 and the data delete operation 1100 make it possible to transfer necessary data from the OLTP database to the OLAP database to keep the OLAP database up to date, and reduce the frequency of federated queries while removing unnecessary data from the OLTP database.

Next, the periodic data movement timing update process 1200 is described with reference to FIG. 12.

FIG. 12 is a flow chart illustrating a flow of a periodic data movement timing update process 1200 according to an embodiment of the present disclosure. The periodic data movement timing update process 1200 is a process for updating the periodic data movement timing in which data is periodically transferred from the OLTP database to the OLAP database in order to provide the latest data to the OLAP database in a timely manner and reduce the frequency of federated queries. The periodic data movement timing update process 1200 illustrated in FIG. 12 substantially corresponds to step S755 of FIG. 7.

First, in step S1205, the query log management unit 343 of the data movement management device 240 monitors the OLTP query log file (OLTP query log file 314) and the OLAP query log file (OLAP query log file 324).

Next, in step S1210, the data movement timing management unit 346 determines a data insertion rate of the OLTP database and a first data freshness requirement of the OLAP database. Here, the data insertion rate of the OLTP database refers to the rate at which new data is inserted into the OLTP database, and may be expressed as data size per hour (e.g., 5 mb/sec) or number of rows/columns per hour (e.g., 50 rows/min). In an embodiment, the data movement timing management unit 346 may calculate the data insertion rate of the OLTP database based on the data insertion queries included in the OLTP query log file. As described herein, the first data freshness requirement may be determined based on the OLAP query log file 324 (e.g., from the federated queries detected in step S710).

Next, in step S1215, the data movement timing management unit 346 determines whether the current periodic movement time is effective in keeping the OLAP database up to date based on the data insertion rate of the OLTP database determined in step S1205 and the first data freshness requirement of the OLAP database.

More specifically, the data movement timing management unit 346 may compare the data insertion rate with a predefined first insertion rate threshold and the first data freshness requirement with a first recency threshold. The first insertion rate threshold may define a particular threshold rate at which data is inserted into the OLTP database. The data insertion rate of the OLTP database is considered to have reached the first insertion rate threshold if the data insertion rate calculated in step S1210 is greater than or equal to the first insertion rate threshold. The first recency threshold specifies a date or date range that defines the limit of the first data freshness requirement. The first data freshness requirement is considered to have reached the first recency threshold if the first data freshness requirement requests data that is more recent than the first recency threshold.

If the data insertion rate of the OLTP database does not reach the first insertion rate threshold or the first data freshness requirement of the OLAP database does not reach the first recency threshold, it is determined that the current periodic migration is not effective. This is because when the data insertion rate of the OLTP database is relatively low or the first data freshness requirement of the OLAP database is such that the most recent data is not needed, it is not necessary to perform frequent periodic data migration from the OLTP database to the OLAP database.

Conversely, if the data insertion rate of the OLTP database reaches the first insertion rate threshold or the first data freshness requirement of the OLAP database reaches the first recency threshold, it is determined that the current periodic migration is not effective. This is because when the data insertion rate of the OLTP database is relatively high or the first data freshness requirement of the OLAP database is such that the most recent data is required, it becomes necessary to perform more frequent periodic data migration from the OLTP database to the OLAP database to provide the most recent data.

Periodic data movement timing may be considered effective if the data insertion rate of the OLTP database reaches the first insertion rate threshold but the first data freshness requirement of the OLAP database does not reach the first recency threshold, or if the data insertion rate of the OLTP database does not reach the first insertion rate threshold but the first data freshness requirement of the OLAP database reaches the first recency threshold.

If it is determined that the current periodic movement is effective, the process may return to step S1205. If it is determined that the current periodic movement is not effective, the process may proceed to step S1220.

Next, in step S1220, if it is determined that the current periodic data movement time is not effective, the data movement timing management unit 346 may update the periodic data movement timing at which the bulk data transfer from the OLTP database to the OLAP database is performed. It should be noted that, typically, the periodic data movement timing is set to 24 hours, i.e., the bulk data transfer from the OLTP database to the OLAP database is performed once every day. However, as described herein, according to an embodiment of the present disclosure, it is possible to modify the periodic data movement timing (i.e., increase or decrease the length of its interval) based on the data insertion rate of the OLTP database and the first data freshness requirement of the OLAP database.

If the data insertion rate of the OLTP database does not reach the first insertion rate threshold or the first data freshness requirement of the OLAP database does not reach the first recency threshold, the data movement timing management unit 346 may increase the interval between periodic data movement timings. As a result, periodic data movements occur less frequently, which is acceptable because the OLAP database does not require the most recent data (e.g., due to the low insertion rate or relaxed data freshness requirements of the OLTP database).

Conversely, if the data insertion rate of the OLTP database reaches the first insertion rate threshold or the first data freshness requirement of the OLAP database reaches the first recency threshold, the data movement timing management unit 346 may decrease the interval between periodic data movement timings. As a result, periodic data movements are performed more frequently because the OLAP database requires more recent data (e.g., due to a high insertion rate or strict data freshness requirements of the OLTP database).

Here, the degree to which the periodic data movement timing is increased or decreased may be determined in several ways. In an embodiment, the data movement timing management unit 346 may increase or decrease the periodic data movement timing by a predetermined amount (e.g., increase or decrease by 1 hour, 2 hours, 6 hours, 12 hours). In some embodiments, the data movement timing management unit 346 may use statistical analysis or machine learning approaches to identify an appropriate periodic data movement timing based on the history of periodic data movement timing and the resource load of the OLTP and OLAP databases. In an embodiment, the data movement timing management unit 346 may increase or decrease the periodic data movement timing based on a prediction of when the data insertion rate or first data freshness requirement of the OLTP database has reached or has not reached a first insertion rate threshold (e.g., the period during which a system configuration change significantly increases or decreases data insertion) or a first recency threshold (e.g., when an application with a known data freshness requirement is scheduled to be executed). Other methods of determining the periodic data movement timing are also possible.

According to the periodic data movement timing update process 1200 described with reference to FIG. 12, the periodic data movement timing can be updated to the appropriate timing to promote resource efficiency at the OLTP database and the OLAP database. For example, by increasing the periodic data movement timing (i.e., decreasing the frequency) when the OLAP database does not require recency data, the computing resources and processing load required at the OLTP database can be reduced. Conversely, by decreasing the periodic data movement timing (i.e., increasing the frequency) when the OLAP database requires recency data, the frequency of federated queries from the OLAP database to the OLTP database can be reduced, which in turn reduces the computing resources and processing load required at the OLTP database.

Next, a periodic data movement delay process is described with reference to FIG. 13. Aspects of the disclosure relate to the recognition that if data in an OLTP database is being frequently updated, it may be desirable to delay data management operations until the frequency of updates decreases so that the latest data can be efficiently transferred to the OLAP database.

FIG. 13 is a flowchart showing the flow of a data management operation delay process 1300 according to an embodiment of the present disclosure. The data management operation delay process 1300 is a process for delaying a data management operation when data in an OLTP database is frequently updated. The data management operation delay process 1300 shown in FIG. 13 may be performed following step S735 in FIG. 7 (e.g., prior to performing a data management operation).

First, in step S1305, the query log management unit 343 of the data movement management device 240 monitors the OLTP query log file (OLTP query log file 314) and the OLAP query log file (OLAP query log file 324).

Next, in step S1310, the data movement management unit 345 determines an update frequency of the first data set of the OLTP database and a first data freshness requirement of the OLAP database. Here, the update frequency of the first data set refers to the rate at which the first data set of the OLTP database is updated and may be expressed as a number of updates per hour (e.g., 10 updates per minute). In an embodiment, the data movement management unit 345 may calculate the update frequency of the first data set of the OLTP database based on data insertion queries included in the OLTP query log file. As described herein, the first data freshness requirement may be determined based on the OLAP query log file 324.

Next, in step S1315, the data movement management unit 345 determines whether the currently scheduled data management operation is efficient to execute.

More specifically, the data movement management unit 345 determines that performing the current data management operation is inefficient if the update frequency of the first data set in the OLTP database reaches the first update frequency threshold and the first data freshness requirement does not reach the first recency threshold (e.g., it is more efficient to perform the data management operation later because the data in the OLTP database is frequently updated and the OLAP database does not require recency data). In this case, the process proceeds to step S1325. Here, the first update frequency threshold may define a particular threshold rate at which the first data set is updated in the OLTP database. The update frequency of the OLTP database is considered to have reached the first update frequency threshold if the update frequency calculated in step S1310 is greater than or equal to the first update frequency.

Conversely, if the update frequency of the first data set in the OLTP database has not reached the first update frequency threshold or the first data freshness requirement has reached the first recency threshold, the data movement management unit 345 determines that the OLAP database is efficient (e.g., the data in the OLTP database is updated infrequently or the OLAP database requires the most recent data, so it is more efficient to perform data management operations at that time to avoid federated queries). In this case, processing proceeds to step S1320.

In step S1320, data movement management unit 345 performs data management operations because the first data set is updated frequently and therefore it is desirable to delay copying the first data set to the OLAP database until the update frequency decreases, but the first data freshness requirements of the OLAP database are such that the first data set is needed sooner (e.g., to respond to a query).

In step S1325, the data movement management unit 345 delays the performance of the data management operation. In an embodiment, the data movement management unit 345 may delay the performance of the data management operation until the update frequency of the first data set falls below a first update frequency threshold. In this manner, by delaying performance of the data management operation until the update frequency of the first data set decreases, more up-to-date data can be provided to the OLAP database in an efficient manner.

The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memories (RAM), read only memories (ROM), erasable programmable read only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures having instructions recorded thereon, and any suitable combination of the above.

As used herein, computer-readable storage media is not to be construed as being a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating in a waveguide or other transmission medium (e.g., a light pulse traveling in a fiber optic cable), or an electrical signal transmitted in an electrical wire.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, are implemented by computer readable program instructions.

Computer-readable program instructions are provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing device to prepare a machine, such that the instructions executed by the processor of the computer or other programmable data processing device are means for implementing the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing device, and/or other device to function in a particular manner, such that a computer-readable storage medium on which instructions are stored encompasses an article of manufacture that includes instructions that implement aspects of the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams.

The computer-readable program instructions are loaded into a computer, other programmable data processing device, or other device to cause the computer, other programmable device, or other device to perform a series of operational steps to perform a computer-implemented process, such that the instructions executing on the computer, other programmable device, or other device implement the function/act specified in one or more blocks of the flowcharts and/or block diagrams.

Embodiments according to the present disclosure may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides abstraction between computing resources and their underlying technical architecture (e.g., servers, storage, networks) to enable convenient, on-demand network access to a shared pool of configurable computing resources that are rapidly provisioned and torn down with minimal administrative effort or service provider interaction. Thus, cloud computing enables users to access the virtual computing resources of the "cloud" (e.g., storage, data, applications, and fully virtualized computing systems) without regard for the underlying physical systems (or the location of these systems) used to provide the computing resources.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality required. It will also be noted that each block of the block diagrams and/or flowchart diagrams, and combinations of blocks in the block diagrams and/or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or a combination of dedicated hardware and computer instructions.

While the above is directed to exemplary embodiments, other and further embodiments of the present invention may be devised without departing from the basic scope thereof, the scope of which is determined by the claims that follow. The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations that do not depart from the scope and spirit of the described embodiments will be apparent to those skilled in the art. The terminology used herein has been selected to explain the principles of the embodiments, practical applications, or technical improvements over the art found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of various embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. "Set of," "group of," "bunch of," and the like are intended to include one or more. The words "includes" and/or "including," as used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. In the exemplary embodiments previously described in detail, reference is made to the accompanying drawings (where like numerals represent like elements) which form a part of the embodiments and which show by way of illustration specific exemplary embodiments in which the various embodiments may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the embodiments, but other embodiments may be used and logical, mechanical, electrical and other changes may be made without departing from the scope of the various embodiments. In the preceding description, numerous specific details are presented to provide a thorough understanding of the various embodiments. However, the various embodiments may be practiced without including these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail so as not to obscure the embodiments.

100 Computer system 102 Processor 104 Memory 106 Memory bus 108 Input/output bus 109 Bus IF
110 Input/Output bus IF
112 terminal interface 113 storage device interface 114 input/output device interface 115 network interface 116 user input/output device 117 storage device 124 display system 126 display 130 network 150 data movement management application 200 data movement management system 210 OLTP device 214 OLTP database 220 OLAP device 224 OLAP database 230 communication network 240 data movement management device 312 OLTP database load data 314 OLTP query log file 322 OLAP database load data 324 OLAP query log file 341 table status database 342 table status management unit 343 query log management unit 344 resource management unit 345 data movement management unit 346 data movement timing management unit

Claims (8)

1. A data movement management device for managing data in a hybrid transactional analytical processing architecture, comprising:
a query log management unit for detecting a federated query requesting a first data set from an online transaction processing database from a first query log set of the online analytical processing database, and for identifying a first data freshness requirement of the online analytical processing database based on the federated query;
a data movement management unit for determining a data management operation including at least copying the first data set from the online transaction processing database to the online analytical processing database based on at least the first data freshness requirement;
a data movement timing management unit for determining an updated periodic movement timing for transferring bulk data from the online transaction processing database to the online analytical processing database when a frequency of federated queries from the online analytical processing database reaches a first frequency threshold;
A data movement management device comprising : The query log management unit further comprises:
identifying a second data freshness requirement for the online transaction processing database based on a second set of query logs of the online transaction processing database;
It is configured as follows:
The data movement management unit further comprises:
determining that a copy operation of copying the first data set from the online transaction processing database to the online analytical processing database is to be performed as the data management operation when the first data freshness requirement of the online analytical processing database corresponds to the second data freshness requirement of the online transaction processing database;
when the first data freshness requirement of the online analytical processing database does not correspond to the second data freshness requirement of the online transaction processing database, determining that a copy/delete operation of copying the first data set from the online transaction processing database to the online analytical processing database and deleting the first data set from the online transaction processing database is to be performed as the data management operation;
It is configured as follows:
The data movement management device of claim 1 . To perform the copy operation, the data movement management unit:
generating a transfer file including the first set of data from the online transaction processing database when available CPU and memory resources reach a first resource threshold;
transferring the transfer file from the online transaction processing database to the online analytical processing database when available network resources reach a first network resource threshold;
It is configured as follows:
The data movement management device of claim 2 . To perform the copy and delete operations, the data migration management unit:
generating a transfer file including the first set of data from the online transaction processing database when available CPU and memory resources reach a first resource threshold;
transferring the transfer file from the online transaction processing database to the online analytical processing database when available network resources reach a first network resource threshold;
deleting the first data set from the online transaction processing database;
It is configured as follows:
The data movement management device of claim 2 . The query log management unit further comprises:
determining an update frequency of the first data set of the online transaction processing database based on the second query log set of the online transaction processing database in response to the determination of the data management operation;
The data movement timing management unit further comprises:
determining to delay execution of the data management operation for a predetermined time if the update frequency of the first data set of the online transaction processing database reaches a first update frequency threshold and the first data freshness requirement does not reach a first recency threshold;
determining to start performing the data management operation when the update frequency of the first data set of the online transaction processing database has not reached the first update frequency threshold or the first data freshness requirement has reached the first recency threshold;
It is configured as follows:
The data movement management device of claim 2 . The query log management unit further comprises:
determining a data insertion rate at which data records are inserted into the online transaction processing database based on the second set of query logs of the online transaction processing database;
It is configured as follows:
The data movement timing management unit further comprises:
increasing an interval of a periodic transfer timing at which the batch data is periodically transferred from the online transaction processing database to the online analytical processing database when the data insertion rate does not reach a first insertion rate threshold or the first data freshness requirement does not reach a first recency threshold;
and decreasing an interval between periodic bulk data transfers from the online transaction processing database to the online analytical processing database when the data insertion rate reaches the first insertion rate threshold or when the first data freshness requirement reaches a first recency threshold.
The data movement management device of claim 2 . A data movement management method for a data movement management device for managing data in a hybrid transaction analysis architecture, the data movement management device comprising a query management unit, a data movement management unit and a data movement timing management unit,
The query management unit detects, from a first set of query logs of the online analytical processing database, a federated query requesting a first set of data from the online transaction processing database ;
identifying a first data freshness requirement for the online analytical processing database based on the federated query;
the data movement management unit determines a data management operation including at least copying the first data set from the online transaction processing database to the online analytical processing database based on at least the first data freshness requirement;
The data movement timing management unit determines an updated periodic movement timing for transferring bulk data from the online transaction processing database to the online analytical processing database when a frequency of federated queries from the online analytical processing database reaches a first frequency threshold;
The method includes: 1. A data movement management system for managing data in a hybrid transactional architecture, comprising:
an online transaction processing device having an online transaction processing database;
an online analytical processing device having an online analytical processing database;
a data movement management device for managing the movement of data from the online transaction processing database to the online analytical processing database;
A data migration management system comprising:
The data movement management device,
a query log management unit for detecting a federated query requesting a first data set from the online transaction processing database from a first query log set of the online analytical processing database, and for identifying a first data freshness requirement of the online analytical processing database based on the federated query;
a data movement management unit for determining a data management operation including at least copying the first data set from the online transaction processing database to the online analytical processing database based on at least the first data freshness requirement;
a data movement timing management unit for determining an updated periodic movement timing for transferring bulk data from the online transaction processing database to the online analytical processing database when a frequency of federated queries from the online analytical processing database reaches a first frequency threshold;
Including,
Data movement management system.

Images