JP6287101B2 - Data processing system, data processing method and program - Google Patents

Description

  The present invention relates to a data processing system, a data processing method, and a program. More specifically, the present invention relates to a data processing technique capable of increasing the data processing speed.

  Conventionally, various measures for speeding up data processing, typically sorting processing, have been proposed.

  For example, Patent Document 1 describes that an operation time of data processing is recorded, and the record is used for analysis of whether or not a data processing algorithm and hardware can achieve a desired performance.

  Further, Patent Document 2 describes that a plurality of data processing algorithms are executed in parallel, and when a certain algorithm derives a processing result earliest, the processing result is adopted and processing by another algorithm is stopped. Has been.

  Patent Document 3 describes that a plurality of sort processing algorithms are properly used in accordance with the contents of sort target data.

  Furthermore, as a general measure, a method is known in which a large amount of memory is used and an auxiliary storage device having a high IO (Input-Output) speed is used.

Japanese Patent Laid-Open No. 05-324412 Japanese Patent Laid-Open No. 06-131312 JP 2002-116907 A

  However, in the techniques described in Patent Documents 1 to 3, the optimum calculation resource (usable memory, secondary storage device, CPU, calculation algorithm, etc.) for processing specific input data, that is, the data processing speed is the fastest. Such a combination of input data and computing resources cannot be known before executing data processing.

  Therefore, users still need to experiment with parameters related to input data and computational resources, such as memory size used, storage location and size of input and work files, maximum or average value of data record length, number of records, etc. However, it was necessary to improve the sorting speed by making adjustments. This imposes great effort and burden on the user.

  Also, the environment that the user can use (combination of usable memory, secondary storage device, CPU, calculation algorithm, etc.) is not always the same. The combination is not always optimal in other environments. Therefore, the user has to perform the above-described troublesome adjustment work for each environment.

  Furthermore, the general measures as described above, such as using a large amount of memory and using an auxiliary storage device having a high IO (Input-Output) speed, do not always work effectively, but rather operate. In some cases, the time required to complete the data processing is increased. For example, by securing a relatively large memory that exceeds the size of the input file that records the data to be processed and the size of the work file that is used as a temporary data storage location during data processing, the operation can be performed instead. Slow cases have been observed.

  Even if an auxiliary storage device with a high IO speed is used, the data processing time may be longer. For example, it has been observed that when the balance between the file IO and the memory cache is not optimal and the cache size is larger than necessary, the operation is rather slow.

  The present invention has been made to solve such problems, and it is an object of the present invention to provide a data processing system, a data processing method, and a program capable of increasing the data processing speed.

  Other problems and novel features will become apparent from the description of the specification and the accompanying drawings.

  A data processing system according to the present invention includes a data processing unit including an analysis unit and a sorting unit, and a storage unit including an analysis result storage unit, and the analysis unit includes one or more inputs including processing target data. A file and one or more environmental conditions indicating computing resources that can be used for the processing are input, the processing is tried by comprehensively combining the input file and the environmental conditions, and the trial result is the analysis result. Storing in the storage unit, the sorting unit inputs the input file, refers to the analysis result storage unit, specifies the environmental condition that the processing time related to the input file is the shortest, and the specified The processing is executed using environmental conditions.

  The data processing method according to the present invention inputs one or more input files including data to be processed and one or more environmental conditions indicating computing resources usable for the processing, and the input file and the environmental conditions are input. Trying the process in an exhaustive combination, storing the trial result in the analysis result storage unit, inputting the input file, referring to the analysis result storage unit, and relating to the input file A sorting step of specifying the environmental condition that causes the shortest processing time and executing the processing using the specified environmental condition.

  The program according to the present invention is a program for causing a computer to execute the above method.

  According to the present invention, it is possible to provide a data processing system, a data processing method, and a program capable of increasing the data processing speed.

It is a figure which shows the structure of the data processing system 100 of embodiment of this invention. It is a figure which shows the process of Embodiment 1 of this invention. It is a figure which shows the structure of Embodiment 1 of this invention. It is a figure which shows the process of Embodiment 2 of this invention.

Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.
<Embodiment 1>
First, the configuration of the data processing system 100 according to the first exemplary embodiment of the present invention will be described with reference to FIG. The data processing system 100 is a system that inputs an input file in which processing target data is recorded and performs a sorting process on the data.

  The data processing system 100 includes an input unit 1 such as a keyboard, a data processing unit 2 that operates under program control, a storage unit 3 that stores information, and an output unit 4 such as a display device or a printing device. .

  The data processing unit 2 inputs an input file for learning, measures sort processing times using various combinations of calculation resources (hereinafter referred to as environmental conditions), and inputs an analysis unit 21 that accumulates the results. A sort unit 22 that inputs a file and executes sort processing using an optimal computing resource.

  The storage unit 3 includes an analysis result storage unit 31 that stores the input file condition and the optimum environmental condition in association with each other.

  Next, the operation of the data processing system 100 will be described with reference to the flowcharts of FIGS.

  The data processing system 100 performs a two-stage operation of a learning stage shown in FIG. 2 and an actual processing stage shown in FIG. Steps A1 to A3 in FIG. 2 are stages for learning various conditions for providing a high-speed sort process. Steps B1 to B3 in FIG. 3 are stages for providing an optimum sort processing service using the learning result.

  First, processing related to the learning stage will be described.

  A1: The input unit 1 inputs the input file condition to the analysis unit 21.

  Here, the input file refers to a file in which data to be sorted is recorded. The input file condition is a requirement indicating the characteristics of this file. Input file conditions include, for example, the number of input files, size, number of divisions, file format, number of records, maximum record length, average record length, number of keys, presence / absence of equivalence keys, data arrangement, etc. . Also, it may be input file arrangement on the disk, work file arrangement / number of divisions, output file arrangement, used disk, memory usage, sort algorithm, and the like. For the input file condition, only one requirement may be specified, or a plurality of input file conditions may be specified simultaneously.

A2: The input unit 1 inputs environmental conditions to the analysis unit 21.
Here, the environmental conditions refer to characteristics of one or more calculation resources for executing the sort process, and combinations thereof. Examples of the environmental conditions include an input file directory, an output file directory, a work file storage directory, a memory usage amount, a CPU usage count, and a CPU usage rate. In addition, as an environmental condition, an index that may have a certain range, such as a maximum memory usage, a maximum CPU usage, and a maximum CPU usage, may be specified. There may be one environmental condition, or a plurality of environmental conditions may be specified simultaneously.

  A3: The analysis unit 21 measures the sorting time according to the input file condition and environmental condition that have been input.

  Here, when a plurality of requirements are specified as the input file condition or the environmental condition, the analysis unit 21 actually executes the sort process for each of the plurality of requirements and measures the processing time. I do.

  When the index range is specified as the environmental condition, the analysis unit 21 can pick up a plurality of values from the specified range and measure the sorting time for each value. For example, when “maximum memory available amount = 3 GB” is input as the environmental condition, the analysis unit 21 can perform four measurements with the memory usage amount being 0, 1 GB, 2 GB, and 3 GB. Here, how many values are picked up from the specified range and actually measured, and what value is picked up are determined as appropriate in consideration of the nature of the calculation resource, the time required for calculation, the amount of calculation, etc. It is possible. Generally, at this stage, the more accurate the measurement is performed, the more accurate results can be provided in later steps.

The analysis unit 21 stores the actually measured result in the analysis result storage unit 31 as the analysis result. In the present embodiment, when a certain input file condition is used, the analysis unit 21 stores only the fastest environmental condition in the analysis result storage unit 31 in association with the input file condition. Thereby, the size of the analysis result storage unit 31 can be suppressed.
It should be noted that all actual measurement results may be saved as necessary.

  Next, processing steps using the learning result will be described.

  B1: The input unit 1 inputs the input file condition to the sorting unit 22.

  B2: The sorting unit 22 obtains from the analysis result storage unit 31 an environmental condition that is optimal for processing the input file condition that has been input, that is, an environmental condition that would result in the fastest sorting time. That is, the sort unit 22 searches the stored contents of the analysis result storage unit 31 using the input file condition that has been input as a key, and acquires the environmental condition associated with the input file condition. Here, in the present embodiment, it is assumed that only one optimum environmental condition is stored in the analysis result storage unit 31 in association with the input file condition.

  The analysis result storage unit 31 may store a plurality of environmental conditions including optimum environmental conditions in association with a certain input file condition. In this case, the sorting unit 22 selects and acquires the environmental condition having the shortest actual measurement time from among them.

  B3: The sorting unit 22 executes the sorting process using the environmental condition acquired in step B2. That is, the calculation resources used by the data processing system 100 (for example, the amount of memory, the number of CPUs, the upper limit of CPU usage, etc.) are set to be the same as the acquired environmental conditions. After that, sort processing is performed.

The operation of this embodiment will be further described using a specific example.
A1: The input unit 1 inputs three files of “input file size = 1G, 2G, 3G” to the analysis unit 21 as input file conditions.

  A2: The input unit 1 designates and inputs “maximum memory usable amount = 3 GB” to the analysis unit 21 as an environmental condition.

  A3: The analysis unit 21 performs sort processing using the specified environmental condition for each of the three input file conditions, and measures the processing time. Here, the “maximum memory usable amount” of the environmental condition is not a fixed value but specifies a certain range that the value can take. Therefore, the analysis unit 21 designates a value of 1 or more from this range and measures the sorting time. In this example, actual measurement is performed using a total of four patterns of environmental conditions with memory usage = 0, 1 GB, 2 GB, and 3 GB.

  That is, the analysis unit 21 measures the sorting speed when the four environmental conditions of memory usage = 0, 1 GB, 2 GB, and 3 GB are used for each of the three files of the input file size = 1G, 2G, and 3G. . As a result, a total of 12 actual measurement values are obtained for four times for each of the three files.

For example, it is assumed that the following set of actual measurement values is obtained.
Input file size 1GB, memory usage 0GB, sort time 10,000ms
Input file size 1GB, Memory usage 1GB, Sort time 4,000ms
Input file size 1GB, Memory usage 2GB, Sort time 4,500ms
Input file size 2GB, Memory usage 0GB, Sort time 20,000ms
Input file size 2GB, Memory usage 1GB, Sort time 15,000ms
Input file size 2GB, Memory usage 2GB, Sort time 5,000ms
Input file size 3GB, Memory usage 0GB, Sort time 35,000ms
Input file size 3GB, Memory usage 1GB, Sort time 20,000ms
Input file size 3GB, Memory usage 2GB, Sort time 6,000ms

  After that, the analysis unit 21 specifies the environmental condition with the fastest sorting speed for each input file condition. In the above example, the following combinations are specified.

Input file size 1GB, Memory usage 1GB, Sort time 4,000ms
Input file size 2GB, Memory usage 1GB, Sort time 15,000ms
Input file size 3GB, Memory usage 2GB, Sort time 6,000ms

  The analysis unit 21 stores the set of the specified input file condition and environmental condition in the analysis result storage unit 31.

  B1: The input unit 1 inputs a file of “input file size = 2 GB” to the sorting unit 22 as an input file condition.

  B2: The sorting unit 22 refers to the analysis result storage unit 31 and acquires the environmental condition “memory usage = 1 GB” corresponding to “input file size = 2 GB”.

  B3: Reserve computing resources according to the acquired environmental conditions. That is, 1 GB of memory is secured and sort processing is executed.

  According to the present embodiment, the analysis unit 21 measures the processing time under various input file conditions and environmental conditions, and learns the optimal combination of the input file conditions and the environmental conditions. Further, when an input file condition is given, the sort unit 22 determines an environmental condition that minimizes the sort time using the learning result. Thereby, the sort part 22 can obtain | require automatically the optimal combination of calculation resources in the computer system.

  Further, when there are a plurality of computer systems to be used, by executing the analysis unit 21 in each computer system, it is possible to automatically obtain the optimal combination of computing resources in each computer system.

<Embodiment 2>
In the first embodiment, the case has been described where the input file condition input to the analysis unit 21 in the learning stage is an actual input file, that is, an input file including data. In the second embodiment, a case will be described in which the input file condition input to the analysis unit 21 in the learning stage is only the condition related to the input file. That is, only the attributes (for example, size) of the input file are input to the analysis unit 21, not the input file including actual data. The analysis unit 21 automatically creates a dummy file according to the conditions, executes sort processing using the dummy file as an input file, and measures the processing time.

  The operation of the data processing system 100 according to the second exemplary embodiment will be described using the flowchart of FIG. Steps C1 to C4 show the operation in the learning stage of the data processing system 100 according to the second exemplary embodiment. Note that the description of the same processes as those in Embodiment 1 is omitted unless otherwise specified. The operation related to actual data processing using the learning result is the same as that in the first embodiment (FIG. 3, Steps B1 to B3), and thus the description thereof is omitted.

C1: The input unit 1 inputs an input file condition to the analysis unit 21.
As the input file condition, various attributes relating to the input file can be input. For example, the number of input files, the size, the number of records, etc. can be specified as the input file condition.

C2: The analysis unit 21 creates a dummy input file according to the input file condition.
For example, if a specific input file size is designated as the input file condition, the analysis unit 21 generates a dummy file of that size. If an input file size range is specified as an input file condition, dummy files of various sizes of 1 or more are created within the range.

  C3: The input unit 1 inputs environmental conditions to the analysis unit 21.

C4: The analysis unit 21 executes sort processing using the input file conditions and environmental conditions that have been input, measures the processing time, and stores the measurement results in the analysis result storage unit 31.
Here, when a specific range that can be taken as a value is specified as the environmental condition instead of a specific value, the analysis unit 21 may create an environmental condition having one or more various values within the range. it can. For example, if the maximum memory usage is specified, the memory usage of various sizes is used within the range.

The operation of this embodiment will be further described using a specific example.
C1: The input unit 1 inputs “input file number = 2”, “input file size = 1 GB to 3 GB”, and “number of records = 1 million to 3 million” to the analysis unit 21 as input file conditions.

C2: The analysis unit 21 generates a dummy input file according to the input file condition.
For example, the following dummy files are generated.
Dummy file name (number of files, file size, number of records):
Input file 1 (2, 1G, 1 million)
Input file 2 (2, 1G, 2 million)
Input file 3 (2, 1G, 3 million)
Input file 4 (2, 2G, 1 million)
Input file 5 (2, 2G, 2 million)
Input file 6 (2, 2G, 3 million)
Input file 7 (2, 3G, 1 million)
Input file 8 (2, 3G, 2 million)
Input file 9 (2, 3G, 3 million)

  Here, the file size and the number of records of each dummy file are determined randomly by the analysis unit 21 within the range specified as the input file condition.

  C3: The input unit 1 inputs “input file directory = A or B”, “output file directory = C or D”, and “maximum memory usable amount = 3 GB” to the analysis unit 21 as environment information.

C4: The analysis unit 21 comprehensively creates combinations of input file conditions and environmental conditions, and measures the sort processing time for each combination.
For example, the sort processing time is measured for each of the following combinations.
(Number of input files, file size, number of records, input file directory, output file directory, memory usage):
(2, 1GB, 1 million, A, C, 0)
(2, 1GB, 1 million, A, C, 1G)
(2, 1GB, 1 million, A, C, 2G)
(2, 1GB, 1 million, A, C, 3G)
(2, 1GB, 1 million, A, D, 0)
(2, 1GB, 1 million, A, D, 1G)
...
(2, 3GB, 2 million, B, D, 2G)
(2, 3GB, 2 million, B, D, 3G)

  Here, the environmental condition included in each combination is determined randomly by the analysis unit 21 within the range specified as the environmental condition.

  The analysis unit 21 identifies the environmental condition with the shortest sort time for each input file condition. Then, the analysis unit 21 stores the set of the input file condition and the specified environmental condition in the analysis result storage unit 31.

  In addition, as the number of input data conditions and environmental conditions increases, and as an index with a range designation is used, it takes more time to identify the fastest combination.

  In that case, it is possible to weight the individual conditions constituting the input data condition and the environmental condition. That is, for conditions that have relatively little influence on the data processing time, the conditions are omitted. Thereby, the learning time can be shortened.

  For example, in the specific example shown in the second embodiment, as the actual measurement process in step C4 progresses, for example, “output file directory” which is one of the environmental conditions is changed to either C or D. However, if it is found that the processing time is not shortened by an arbitrary threshold (for example, 5%) or more, only one of the “output file directory” indicators, for example, only C is adopted in the subsequent processing. And actual measurement of D may be omitted.

  Alternatively, if an index that does not reduce the processing time by a certain amount or more is found from accumulation of past measurement results, the index may be fixed to an arbitrary value without changing the index in the learning stage. . In order to implement this, it is preferable to store the combination of the input data condition and the environmental condition in the past actual measurement results without discarding not only the fastest but also the other. Thereby, the accumulated measurement result can be analyzed, and the input file condition and the environmental condition having a small influence on the processing time can be specified.

  According to the present embodiment, since the analysis unit 21 generates various input files according to the input file conditions, the actual measurement patterns in more data structures, formats, sizes, and the like are simpler than those in the first embodiment. Can get to. As a result, it is possible to provide a sort process in a shortest time with higher accuracy.

<Other embodiments>
Note that the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention. For example, in the above-described embodiment, the present invention mainly targeting the sort processing has been described. However, the present invention can be applied not only to simple sort processing but also to any batch processing that requires sort processing.

  Further, the present invention can be applied not only to sort processing, but also to various information processing whose processing time is affected by adjusting input data conditions and calculation resource conditions, such as merge processing. .

  In the above-described embodiment, the present invention has been mainly described as a hardware configuration. However, the present invention is not limited to this, and a CPU (Central Processing Unit) executes a computer program for arbitrary processing. Can also be realized. In this case, the computer program can be stored and provided to the computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

The present invention can also be expressed as follows.
(Appendix 1)
A data processing unit including an analysis unit and a sorting unit;
A storage unit including an analysis result storage unit,
The analysis unit
Input one or more input files containing data to be processed and one or more environmental conditions indicating computing resources available for the processing,
Try the process by comprehensively combining the input file and the environmental conditions,
Storing the trial result in the analysis result storage unit;
The sorting unit
Enter the input file,
Referring to the analysis result storage unit, specify the environmental condition that the processing time related to the input file is the shortest,
A data processing system that executes the processing using the specified environmental condition.
(Appendix 2)
The analysis unit
The data processing system according to claim 1, wherein when a range of selectable values is input as the environmental condition, one or more values are generated within the range and used as the environmental condition.
(Appendix 3)
The analysis unit
The data processing system according to claim 1 or 2, wherein an input file condition indicating an attribute of the input file is input instead of the input file, and the input file is generated according to the input file condition.
(Appendix 4)
The analysis unit
A range of selectable values is input as the environmental condition, and two or more values are used within the range, and by changing one of the values to the other, any processing time can be set. The data processing system according to any one of supplementary notes 1 to 3, wherein the trial using any one of the values is not performed when only an influence below a threshold value occurs.
(Appendix 5)
The analysis unit
By referring to the analysis result storage unit, when two or more values are used in the environmental condition having a range of selectable values, by changing one of the values to the other , Specifying in advance the environmental conditions that only have an effect below the threshold on the processing time,
The data processing system according to any one of Supplementary notes 1 to 4, wherein the trial using any of the values is not performed in the trial.
(Appendix 6)
Input one or more input files containing data to be processed and one or more environmental conditions indicating computing resources available for the processing,
Try the process by comprehensively combining the input file and the environmental conditions,
An analysis step of storing the trial result in an analysis result storage unit;
Enter the input file,
Referring to the analysis result storage unit, specify the environmental condition that the processing time related to the input file is the shortest,
A data processing method including a sorting step of executing the processing using the specified environmental condition.
(Appendix 7)
The analysis step includes
The data processing method according to claim 6, wherein when a range of selectable values is input as the environmental condition, one or more values are generated within the range and used as the environmental condition.
(Appendix 8)
The analysis step includes
The data processing method according to claim 6 or 7, wherein an input file condition indicating an attribute of the input file is input instead of the input file, and the input file is generated according to the input file condition.
(Appendix 9)
The analysis step includes
A range of selectable values is input as the environmental condition, and two or more values are used within the range, and by changing one of the values to the other, any processing time can be set. The data processing method according to any one of appendices 6 to 8, wherein the trial using any one of the values is not performed when only an influence below a threshold value occurs.
(Appendix 10)
The analysis step includes
By referring to the analysis result storage unit, when two or more values are used in the environmental condition having a range of selectable values, by changing one of the values to the other , Specifying in advance the environmental conditions that only have an effect below the threshold on the processing time,
The data processing method according to any one of appendices 6 to 9, wherein the trial using any of the values is not performed in the trial.
(Appendix 11)
A program for causing a computer to execute the method according to any one of appendices 6 to 10.

DESCRIPTION OF SYMBOLS 100 Data processing system 1 Input part 2 Data processing part 21 Analysis part 22 Sort part 3 Storage part 31 Analysis result storage part 4 Output part

Claims (8)

A data processing unit including an analysis unit and a sorting unit;
A storage unit including an analysis result storage unit,
The analysis unit
Input one or more input file conditions indicating the attributes of the input file including the processing target data, and one or more environmental conditions indicating calculation resources available for the processing,
Try the process by comprehensively combining the input file conditions and the environmental conditions,
Storing the trial result in the analysis result storage unit;
The sorting unit
Enter the input file,
Referring to the analysis result storage unit, specify the environmental condition that the processing time related to the input file is the shortest,
A data processing system that executes the processing using the specified environmental condition. The analysis unit
The data processing system according to claim 1, wherein when a range of selectable values is input as the environmental condition, one or more values are generated within the range and used as the environmental condition. The analysis unit
A range of selectable values is input as the environmental condition, and two or more values are used within the range, and by changing one of the values to the other, any processing time can be set. The data processing system according to claim 1 or 2 , wherein when only an influence below a threshold value occurs, the trial using any one of the values is not performed. The analysis unit
By referring to the analysis result storage unit, when two or more values are used in the environmental condition having a range of selectable values, by changing one of the values to the other , Specifying in advance the environmental conditions that only have an effect below the threshold on the processing time,
The data processing system according to any one of claims 1 to 3 , wherein the trial using any of the values is not performed in the trial. The analysis unit inputs one or more input file conditions indicating attributes of the input file including processing target data and one or more environmental conditions indicating calculation resources usable for the processing, and the input file conditions and the An analysis step of exhaustively combining environmental conditions to try the process and storing the trial result in an analysis result storage unit;
The sorting unit inputs the input file, refers to the analysis result storage unit, identifies the environmental condition that minimizes the processing time related to the input file, and uses the identified environmental condition, A data processing method for a data processing system, comprising a sorting step for executing the processing. The analysis step includes
The data processing method according to claim 5 , wherein when a range of selectable values is input as the environmental condition, one or more values are generated within the range and used as the environmental condition. The analysis step includes
A range of selectable values is input as the environmental condition, and two or more values are used within the range, and by changing one of the values to the other, any processing time can be set. The data processing method according to claim 5 or 6 , wherein when only an influence below a threshold value occurs, the trial using any one of the values is not performed. A program for causing a computer to execute the method according to any one of claims 5 to 7 .

Images