JP6375201B2 - Automatic data flow parallelization system - Google Patents

Description

  The present invention relates to an automatic data flow parallelization system, and more particularly to a technique for automatically executing a series of business processes achieved by cooperation of a plurality of programs without performing parallel design. .

As CPUs become multi-core, it is becoming necessary to shorten the overall processing time by parallelizing each process on the data flow.
Here, typical parallelization techniques include data parallelism, task parallelism, and pipeline parallelism (see Non-Patent Document 1).

First, data parallel means that the data to be processed is divided into multiple groups based on the key value specified in the settings file such as XML, and the same processing is executed for each group at the same time. It is a technique to combine.
The task parallel means that tasks having no dependency relationship are determined from the data transfer information, and the tasks are executed in parallel.
Furthermore, pipeline parallel refers to starting and succeeding tasks by simultaneously starting tasks that pass data and passing processed data to subsequent tasks before all input data is processed by the preceding task. Related tasks are parallelized.

I-10-8. Basics of parallel processing programming, parallel processing Internet URL: http://ossforum.jp/node/578 Search date: October 8, 2014

  When applying the above parallelization technique, by assigning a separate execution thread (CPU core) to each process, the efficiency of the process utilizing the characteristics of the multi-core CUP is realized.

  However, in order to realize parallelization of such data flow, the data flow designer or the person in charge of the operation analyzes the key information of each program and the data connection information between programs to determine whether parallelization is possible, Since the control program for parallel execution is coded based on the result, there is a problem that the design is difficult and bugs are easily created.

  The present invention has been devised in view of such a current situation, and an object of the present invention is to provide a technique for automatically realizing parallelization of a data flow without performing a design for parallelization.

In order to achieve the above object, an automatic data flow parallelization system according to claim 1 includes a plurality of types of business processing programs for generating different types of business processing means, and parallel outputs output from each business processing means. Analyzing the data for decision making and generating program design means for generating parallel design data and program execution means for allocating and starting an execution thread to each of the business processing programs and generating the respective business processing means In the system, each of the above-described business processing means converts the input / output data defined in each business processing program and a splittable key that is a data item that allows splitting of the input data into the parallel determination data The program analysis means outputs the output data of one business processing program as the input data of the other business processing program. When the splittable key defined in both programs is common, the processing result is transferred to the subsequent business processing means each time processing by the preceding business processing means in units of splittable keys is completed. Generating parallelized design data that specifies that the data is to be output and processed, and the program execution means simultaneously activates each of the business processing means based on the parallelized design data to execute necessary processing. It is a feature.

An automatic data flow parallelization system according to claim 2 includes a business processing program for generating business processing means, a data dividing program for generating a data dividing unit, and data for generating a data combining unit. Analyzing the combined program and the parallel determination data output from the business processing means to generate parallel design data, and executing threads for the business processing program, the data dividing program, and the data combined program A system comprising program execution means for generating and allocating and starting a business processing means, a data dividing section, and a data combining section, wherein the business processing means divides input data defined in the business processing program The splittable key, which is a data item that allows The program analysis means divides the file storing the input data of the business processing means into a predetermined number, and activates a plurality of the business processing means, and processes each divided file to each business processing means. After the completion of the processing by each business processing means, and the parallel execution design data that specifies that the output data is combined, the program execution means, based on the parallel design data, Activating the data dividing unit to divide the file storing the input data into a specified number, activating the same number of the business processing means to simultaneously execute necessary processing, and activating the data combining unit Thus, a process for combining output data from each business processing means is executed.

According to the data flow automatic parallelization system according to claim 1, the data flow for pipeline parallelization is also obtained by the cooperation of each business processing means, program analysis means, and program execution means. According to the automatic data flow parallelization system, a data flow for data parallelization is automatically formed.
In this way, as a result of automatically generating data flows for various types of parallelization, there is an advantage that it is less time-consuming and less likely to introduce bugs than when designing for parallelization artificially. Arise.

  FIG. 1 is a block diagram showing a functional configuration of an automatic data flow parallelization system 10 according to the present invention. A program analysis unit 12, a program execution unit 14, a sales totaling program 16, a store name assigning program 18, A product name extracting program 20, a product name assigning program 22, a data dividing program 24, and a data combining program 26 are provided.

  The program analysis unit 12 and the program execution unit 14 are realized by the computer CPU operating in accordance with a predetermined application program.

The data division program 24 and the data combination program 26 are general-purpose programs that execute data division and combination processing.
The sales totaling program 16, the store name assigning program 18, the product name extracting program 20 and the product name assigning program 22 are programs for executing specific business processes.

  The computer includes a plurality of CPU cores, and each program can be executed when an execution thread (CPU core) is assigned by the program execution unit 14.

  FIG. 2 is a block diagram showing an original data flow 30 realized by executing each business processing program. The sales file 32, the sales totaling unit 34, the sales total file 36, the store name giving unit 38, and the store name The file 40, the sales total file with store name 42, the product master file 44, the product name extracting unit 46, the product name file 48, the product name giving unit 50, and the total sales file 52 with store name and product name Flow 30 is structured.

The sales file 32 stores a large number of records including data items such as a store number (store code), a product code, an amount of money, and a date and time. Each record is sorted using store numbers (1 to 2000) as keys.
The sales totaling unit 34 is a processing function realized by starting the sales totaling program 16, and totals the amount of each data stored in the sales file 32 for each "store number / product code". Responsible for the process of storing in.

  The store name assigning unit 38 is a processing function realized by starting the store name assigning program 18, and is stored in the sales total file 36 with reference to the store name file 40 in which the correspondence between the store number and the store name is stored. After assigning the store name to the total data of the store number / product code unit, it is in charge of the process of storing in the total sales file with store name 42.

The product master file 44 stores a large number of records having data items such as product codes, product names, and suppliers.
The product name extraction unit 46 is a processing function realized by starting the product name extraction program 20, and extracts a combination of a product code and a product name from each data stored in the product master file 44. Responsible for processing stored in the file 48.

  The product name assigning unit 50 is a processing function realized by starting the product name assigning program 22, and stores the product stored in the product name file 48 for each aggregated data stored in the sales total file with store name 42. It is in charge of the process of assigning a name and storing it in the store name product name sales total file 52.

  The sales totaling program 16, the store name assigning program 18, the product name extracting program 20 and the product name assigning program 22 include the transmission units 54-60 in addition to the codes for realizing the original business processing unit of each program described above. A code for realizing each is provided, and a combination of each business processing unit and each transmission unit 54 to 60 corresponds to the “business processing means” of the present invention.

Each of the transmission units 54 to 60 has a function of passing input data and output data related to the original business process of the program to the program analysis unit 12.
In addition, when a splittable key that allows splitting of input data is defined in the program, the program has a function of passing the splittable key to the program analysis unit 12.

Hereinafter, the processing procedure of the system 10 will be described with reference to the flowchart of FIG.
First, the program execution unit 14 assigns an execution thread to each business processing program (sales counting program 16, store name assigning program 18, product name extracting program 20, product name assigning program 22), and activates them (S10).

Next, the program analysis unit 12 acquires parallel determination data from the transmission units 54 to 60 of each business processing program (S12).
That is, the transmission units 54 to 60 of each business processing program pass “input data and output data” defined in advance in the program to the program analysis unit 12. If a “divisible key” is defined in the program, it is also passed to the program analysis unit 12.

  Next, the program analysis unit 12 analyzes the separable key analysis process (S14), the pipeline connection analysis process (S16), and the file input / output relationship based on the parallelization determination data acquired from each transmission unit 54-60. Analysis processing (S18) is executed, parallelized design data based on the analysis result is generated (S20), and output to the program execution unit 14.

  Here, “Analyzing process of separable key” means that the file to be processed is the value of the key item when a key item that allows the division of the input data to be processed is set in the business processing program. It means a process of generating design data that instructs to divide by unit.

For example, the sales aggregation program 16 is intended to aggregate the sales amount in units of “store number x product code”, and when the so-called data tearing occurs in which data of the same store number is divided into multiple files, it is correct in file units Since the total value cannot be obtained, “store number” is designated in advance as a separable key.
On the other hand, the program analysis unit 12 divides the sales file into a predetermined number of “store numbers” to be used for parallel processing, and after the parallel processing is completed, the parallelized design data that instructs to combine the processing results Is generated.

  Next, “Pipeline connection analysis processing” is a process in which both business processes are piped when the separable key set in each business process program is common and each business process is a continuous step. It means a process of generating design data instructing line parallelization.

For example, since the store name assigning program 18 is intended to insert a store name in units of store numbers into the sales total file 36, “Store number” is designated in advance as a splittable key. In the sales totaling program 16, “store number” is set as a separable key as described above. Furthermore, since the sales total file 36 which is the output data of the sales totaling unit 34 is the input data of the store name giving unit 38, continuity is recognized between the two.
From the above, the program analysis unit 12 generates design data instructing pipeline parallel processing for the processing of the sales totaling unit 34 and the processing of the store name giving unit 38.

  “File input / output relationship analysis” means comparing the input data and output data of each business processing program and determining that task parallelism is possible if there is no dependency between them. .

  For example, the input data of the product name extraction program 20 is the product master file 44, the output data is the product name file 48, and these are different from the input data and output data of the sales aggregation program 16 and the store name assignment program 18, The program analysis unit 12 determines that the name extraction program 20 can be executed in parallel with respect to the sales counting program 16 and the store name assignment program 18, and generates parallel design data for instructing that.

  The program execution unit 14 that has received the parallelization design data from the program analysis unit 12 assigns an execution thread to each program in accordance with this (S22), thereby creating a data flow 70 for parallelization processing as shown in FIG. Realize.

  The data flow 70 for parallel processing includes a sales file 32, a data division unit 72, a first sales file 32a, a second sales file 32b, a first sales totaling unit 34a, and a second A sales totaling unit 34b, a first store name assigning unit 38a, a second store name assigning unit 38b, a store name file 40, a first store name-added sales total file 42a, a second store name-added sales file 42b, By the data combining unit 74, the store name sales total file 42, the product master file 44, the product name extraction unit 46, the product name file 48, the product name giving unit 50, and the store name product name sales total file 52. It is configured.

In configuring the data flow 70, the program execution unit 14 first activates the data division program 24 to generate the data division unit 72.
Next, the program execution unit 14 passes the “store number” as a separable key to the data division unit 72 and instructs it to be divided into two files.

  Receiving this, since the data from the store number 1 to the store number 2000 is stored in the sales file 32, the data dividing unit 72 stores the data in the first sales file 32a including the data of the store numbers 1 to 1000, The data is equally divided into the second sales file 32b including the data of the store numbers 1001 to 2000.

  Next, the program execution unit 14 activates the sales totaling program 16 and the store name giving program 18 to generate the first sales totaling unit 34a and the first store name giving unit 38a, and then the sales for the first sales file 32a. The aggregation process and the store name assignment process are executed.

On the other hand, the first sales totaling unit 34a totals the amount of each data stored in the first sales file 32a in units of “store number × product code”, and passes the total result to the first store name giving unit 38a. .
Receiving this, the first store name giving unit 38a refers to the store name file 40, adds a store name corresponding to the store number to each data, and stores it in the first store name-added sales total file 42a.
The first sales totaling unit 34a and the first store name assigning unit 38a continue the above processing in units of “store number × product code” until the processing is completed for all the data stored in the first sales file 32a. To do.

  In parallel with the above, the program execution unit 14 activates the sales totaling program 16 and the store name giving program 18 to generate the second sales totaling unit 34b and the second store name giving unit 38b, and then the second sales file. The sales aggregation process and store name assignment process for 32b are executed.

On the other hand, the second sales totaling unit 34b totals the amount of each data stored in the second sales file 32b in units of “store number × product code”, and passes the total result to the second store name giving unit 38b. .
Receiving this, the second store name assigning unit 38b refers to the store name file 40, adds a store name corresponding to the store number to each data, and stores it in the second store name-added sales total file 42b.
The second sales totaling unit 34b and the second store name assigning unit 38b continue the above processing in units of “store number × product code” until the processing is completed for all the data stored in the second sales file 32b. To do.

The program execution unit 14 completes the processing by the first sales totaling unit 34a and the first store name giving unit 38a and at the time when the processing by the second sales totaling unit 34b and the second store name giving unit 38b is completed. Then, the data combination program 26 is activated to generate the data combination unit 74.
The data combining unit 74 combines the first store-named sales total file 42a and the second store-named sales total file 42b and stores them in the store-named sales total file 42.

  The program execution unit 14 also performs the above-described processes (dividing processing of the sales file 32 by the data dividing unit 72, sales totaling and store name giving processing by the first sales totaling unit 34a and the first store name giving unit 38a, In parallel with the sales aggregation and store name assignment processing by the sales aggregation unit 34b and the second store name assigning unit 38b, and the combination processing of the sales total file with store name by the data combination unit 74), the product name extraction program 20 is started and the product name is started. An extraction unit 46 is generated to execute a product name extraction process.

  The product name extraction unit 46 extracts the combination of the product code and the product name from the data stored in the product master file 44 and stores it in the product name file 48.

  The program execution unit 14 performs the above-described processes (division processing of the sales file 32 by the data division unit 72, sales aggregation and store name assignment processing by the first sales aggregation unit 34a and the first store name assigning unit 38a, and second sales. Upon completion of sales aggregation and store name assignment processing by the aggregation unit 34b and the second store name assigning unit 38b, combination processing of the sales total file with store name by the data combination unit 74, and product name extraction processing by the product name extraction unit 46) The product name assigning program 22 is activated to generate a product name assigning unit 50 and execute product name assigning processing.

  The product name assigning unit 50 refers to the product name file 48, assigns a product name to each data stored in the store name sales total file 42, and stores the data in the store name product name sales total file 52.

As described above, according to the data flow 70 for parallel processing, the processing by the first sales totaling unit 34a and the first store name giving unit 38a, the second sales totaling unit 34b and the second store name giving unit The processing by 38b is so-called data parallelization, and these processing and the processing by the product name extraction unit 46 are so-called task parallelization.
Further, the processing by the first sales totaling unit 34a and the processing by the first store name giving unit 38a are parallelized in a so-called pipeline, and the processing by the second sales totaling unit 34b and the second store name giving The processing by the unit 38b is also pipeline parallelized.
As a result, the overall processing efficiency can be greatly improved.

  Moreover, since the data flow 70 for parallel processing is automatically formed by the cooperation of the transmission units 54 to 60, the program analysis unit 12 and the program execution unit 14 of each business processing program, Compared to designing for parallelization, it is less time-consuming and the risk of bugs is greatly reduced.

  Of course, the developer of each business processing program needs to write the code for realizing the transmission units 54 to 60 in addition to the code for realizing the original function. Is limited to the designation of input data and output data, and if there is a separable key, and these are matters that the developer himself knows, so there is no significant increase in burden.

  Traditionally, the biggest bottleneck in designing dataflows for parallel processing by dataflow design experts was to understand the contents of each business process. As described above, it is extremely meaningful that the business processing program itself outputs the data for determining parallelization and realizes a mechanism for automatically generating a data flow based on the data.

  When executing this parallel processing data flow 70, a maximum of five processes (processing by the first sales totaling unit 34a, processing by the first store name assigning unit 38a, processing by the second sales totaling unit 34b, 2 processing by the store name assigning unit 38b and processing by the product name extracting unit 46), and it is necessary to allocate CPU cores for the OS, so a computer having at least 6 CPU cores is required. It is desirable to use it.

  When a computer equipped with more CPU cores is used, the number of files divided by the data dividing unit 72 can be set to 3 or more, and the overall processing efficiency can be further increased accordingly. .

It is a block diagram which shows the function structure of the automatic parallelization system of the data flow concerning this invention. It is a block diagram which shows the original data flow implement | achieved by execution of each business processing program. It is a flowchart which shows the process sequence in this system. It is a block diagram which shows the data flow for the parallelization process implement | achieved by execution of each business processing program.

10 Automatic data flow parallelization system
12 Program analysis section
14 Program execution part
16 Sales summary program
18 Store name assignment program
20 Product name extraction program
22 Product name assignment program
24 Data division program
26 Data binding program
30 Original data flow
32 Sales file
32a First sales file
32b Second sales file
34 Sales Summary Department
34a 1st Sales Summary Department
34b Second Sales Summary Department
36 Total sales file
38 Store name assignment department
38a First store name assigning department
38b Second store name assigning department
40 Store name file
42 Total sales file with store name
42a Total sales file with first store name
42b Total sales file with second store name
44 Product master file
46 Product name extraction section
48 Product name file
50 Product Name Assignment Department
52 Total sales file with store name and product name
54 Transmitter
56 Transmitter
58 Transmitter
60 Transmitter
70 Data flow for parallel processing
72 Data division part
74 Data coupling part

Claims (2)

A plurality of types of business processing programs for generating different types of business processing means;
Program analysis means for analyzing parallelization judgment data output from each business processing means and generating parallelized design data;
A program execution means for assigning and starting an execution thread to each of the business processing programs described above, and generating each business processing means;
A data flow automatic parallelization system comprising:
Each of the business processing means outputs the input / output data defined in each business processing program and the splittable key that is a data item that allows splitting of the input data as the parallelization determination data,
If the output data of one business process program is the input data of the other business process program and the splittable key defined in both programs is common, the program analysis means Each time processing in units of splittable keys by the processing means is completed, parallelized design data that specifies that the processing result is output to the subsequent business processing means and processed is generated,
An automatic data flow parallelization system characterized in that the program execution means activates the business processing means at the same time to execute necessary processing based on the parallelization design data. A business processing program for generating business processing means;
A data division program for generating a data division unit;
A data combination program for generating a data combination part;
Program analysis means for analyzing the parallelization determination data output from the business processing means and generating parallelized design data;
Program execution means for allocating and starting an execution thread to the above-described business processing program, data division program, and data combination program, and generating business processing means, data division unit, and data combination unit;
A data flow automatic parallelization system comprising:
The business processing means outputs a splittable key, which is a data item that allows division of input data defined in the business processing program, as the parallelization determination data.
The program analysis means is configured to divide a file storing the input data of the business processing means into a predetermined number, and to activate a plurality of the business processing means and perform processing on each divided file. And generate parallel design data that specifies that the output data is to be combined after the processing by each business processing means is completed,
The program execution means activates the data dividing unit based on the parallelized design data to divide the file storing the input data into a specified number, and activates the same number of the business processing means. Data flow automatic parallelization system characterized in that the above-described processing is executed at the same time, and further, the above-described data combining section is activated to execute processing for combining output data from each business processing means.

Images