JP4905989B2 - Metadata search device - Google Patents

Description

  The present invention relates to a technique for processing a large amount of data files such as large-scale simulation data in the field of computer science.

  The result of a simulation performed in a large-scale simulation system such as an ocean simulation system is composed of a large amount of data files. It is not uncommon for a large number of data files to be tens of terabytes or more in size and 10,000 or more files. In general, such a large amount of data files are not created in the same format and often have slightly different formats depending on the contents of the data.

  In numerical simulation, the intersection of line segments as shown in FIG. 16A is generally defined as a calculation grid, and processing (for example, calculation of physical quantities) on numerical data proceeds based on this calculation grid. However, due to the physical characteristics of the numerical data and various circumstances such as the convenience of calculation formulas, processing (calculation of physical quantities) for some numerical data is performed based on different calculation grids as shown in FIG. 16B. There is often.

  Here, the first data file calculated based on the calculation grid shown in FIG. 16A (referred to as “first calculation grid”) and the calculation grid shown in FIG. 16B (“second calculation grid”). Suppose that there is a second data file calculated based on “. Furthermore, it is assumed that data existing in an area (cutout range) from 120.1 degrees to 121.6 degrees east longitude is cut out from the first and second data files.

  In this case, data corresponding to two calculation grids are cut out from the first data file, and data corresponding to four calculation grids are cut out from the second data file. However, when it is desired to cut out data corresponding to the four calculation grids from the first data file, the data cut out in accordance with the above-described cutting range (cutout data) is insufficient.

  Data indicating the details of the data in the data file, such as what kind of calculation grid the first data file includes, is referred to as “metadata” in this specification. .

  Conventionally, metadata is stored separately from simulation data, or a user designates and inputs corresponding metadata for each simulation data file to be processed.

  When the simulation data generated as a result of a large-scale simulation is to be further processed after the simulation, it is a great effort for the user to specify and input metadata for each of these large data files. Will be forced to pay. There is also a risk that the user may specify the metadata by mistake.

  An object of the present invention is to provide a technique that does not require a user to specify metadata for processing target data.

  In order to achieve the above object, the present invention employs the following means.

That is, the present invention accepts designation of a file identifier selected from a list in which a plurality of file identifiers of data files including a file path of a data file stored in one of a plurality of directories constituting the directory structure are displayed. Receiving means;
Metadata storage means for storing metadata representing details of the data file;
Extraction means for extracting a keyword for searching for metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier ;
Search means for searching for metadata corresponding to the extracted keyword from the metadata storage means ;
A keyword list having keyword groups associated with the metadata,
The extraction means compares a character string that forms part of a file path portion of the specified file identifier with the keyword list, and extracts a character string that matches at least one keyword in the keyword list as a keyword. This is a metadata search device.

  Preferably, in the present invention, the metadata storage means is provided in a storage area different from the storage area of the data file to which the file identifier is assigned.

  Further, the present invention can be specified as a metadata search method and program having the same characteristics as the above-described metadata search apparatus, and a recording medium on which this program is recorded.

  According to the present invention, it is not necessary for a user to specify metadata for processing target data.

It is a figure which shows the structural example of the simulation system which can apply this invention. It is a figure which shows the structural example of the computer for control shown in FIG. It is a figure which shows the structural example of the node shown in FIG. It is a figure which shows the example of a directory structure of the file database which stores the process target data file shown in FIG. FIG. 3 is a diagram illustrating an example of a data structure of a metadata table illustrated in FIG. 2. It is a figure which shows the example of a data structure of the use and load distribution condition table shown in FIG. It is a figure which shows the example of a display of the user interface (designation screen) provided to the user of a system. It is a figure which shows the example of a description of the file of the parallel processing designation | designated information input using a user interface. It is a flowchart which shows the main routine of the creation process of the job script for parallel processing, and the setting file for parallel processing programs. It is a flowchart which shows the main routine of the creation process of the job script for parallel processing, and the setting file for parallel processing programs. It is a flowchart which shows the main routine of the creation process of the job script for parallel processing, and the setting file for parallel processing programs. It is a flowchart which shows the subroutine which concerns on analysis and acquisition of metadata. It is a flowchart which shows the subroutine of the search of the node used as the arrangement destination of a process target data file, and a determination process. It is a figure which shows the example of a description of the setting file for parallel processing programs. It is a flowchart which shows the execution process of a parallel processing program. It is a figure which shows the example of the calculation grid prepared as metadata with respect to process target data. It is a figure which shows the example of the calculation grid different from the calculation grid of FIG. 16A prepared as metadata with respect to process target data.

Explanation of symbols

X ... parallel computer group Y ... control computer 1, 11 ... CPU
2, 12 ... Main memory 3, 14 ... External storage device 7 ... Input device 8 ... Display device 6, 15 ... Communication interface 31 ... File database 32 ... Metadata table 33 ... Usage and load balancing information table

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The configuration in the embodiment is an exemplification, and the present invention is not limited to the configuration in the embodiment.

[Simulation system]
FIG. 1 is a diagram showing a configuration example of a simulation system to which the present invention can be applied. In the example shown in FIG. 1, the simulation system includes a parallel computer group X and a control computer (information processing device) Y connected to the parallel computer group X via a communication line (network).

  The parallel computer group X includes a plurality of nodes # 0 to #n (n is a natural number) that performs parallel processing on a large number of data files constituting large-scale simulation data such as an ocean general circulation model.

  The computer Y manages simulation data (processing target data) to be processed in the parallel computer group X, and causes the parallel computer group X to execute parallel processing using the simulation data in accordance with a user operation. Control.

  The user of the simulation system provides parallel processing designation information for executing parallel processing of a large amount of processing target data (processing target data group) using the parallel computer group X through a UI (user interface) provided by the computer Y. input.

  Here, the parallel processing designation information includes a plurality of simulation data files (processing target data file group) to be processed in parallel processing, processing details of the parallel computer group Y for the processing target data file group (processing type, processing detailed parameters), It can include designation of a plurality of nodes (number of nodes) performing parallel processing, a storage position of a file (processed data file (processing result file)) generated as a result of parallel processing, and the like.

  The computer Y uses a parallel processing job script (a control program for the parallel computer group Y: hereinafter referred to as “script”) to give a parallel computer group X a control instruction related to parallel processing based on the input parallel processing designation information. And a parallel processing program configuration file (hereinafter also referred to as a “configuration file”) that is referenced when each node that executes parallel processing processes the data file to be processed. Generate automatically.

  In the script generation process, the computer Y acquires metadata (detailed information of the processing target data) for each processing target data file and determines the arrangement of the processing target data file group for the parallel computer group X. The metadata and the arrangement determination result are reflected in the description content of the script.

  The computer Y distributes (distributes) the processing target data file group to a plurality of nodes through execution of the script, and instructs the nodes to execute a parallel processing program (job). Each node executes the parallel processing program according to the description of the setting file, and performs processing on the distributed processing target data file based on the corresponding metadata. A processing result file is created through this processing. The processing result file is stored at a storage location specified as parallel processing specification information.

<Computer Y>
FIG. 2 is a diagram illustrating a configuration example of the computer Y. In FIG. 2, a computer Y includes a CPU 1, a main memory (MM: for example, RAM) 2, an external storage device (for example, a hard disk) 3, and input / output interfaces (I / F) 4 and 5 that are connected to each other via a bus B. , And a communication interface 6.

  An input device (keyboard, pointing device (eg, mouse)) is connected to the I / F 4, and a display device (display) 8 is connected to the I / F 5. . Further, the communication I / F 6 is connected to each of the nodes # 0 to #n via a communication line (network).

  The external storage device 3 includes a file database (file DB) 31 that stores a large amount of simulation data files constituting large-scale simulation data, and a meta data that stores metadata (detailed information of simulation data) corresponding to each data file. A data table 32 and a use and load distribution status table 33 (hereinafter referred to as “situation table 33”) of each node that is referred to when a processing target data file group is distributed and arranged in a plurality of nodes are stored. . The file DB 31 and the metadata table 32 are created on different storage areas.

  Further, the external storage device 3 stores a program for causing the computer Y to function as a simulation data and metadata management device and to function as a control device for the parallel computer group Y (nodes # 0 to #n). ing.

The CPU 1 realizes the following functions, for example, by loading the program recorded in the external storage device 3 into the MM 2 and executing the program.
(1) Provide an input (designation) environment (UI: user interface) for parallel processing designation information using the input device 7 and the display device 8 to the user of the simulation system.
(2) Create a script and a configuration file based on the parallel processing designation information.
(3) When a script is created, metadata corresponding to each of a plurality of simulation data files (processing target data file group) designated by the user as processing targets is searched and acquired.
(4) At the time of creating a script, a node (arrangement of processing target data files) for processing each processing target data file constituting the processing target data file group is determined.
(5) The transfer control of the processing result file generated by the parallel processing of the processing target data file group and the processing target data file group is performed.

  The CPU 1 corresponds to an accepting unit, an extracting unit, and a searching unit according to the present invention, and the external storage device 3 corresponds to a metadata storage unit (storage unit) according to the present invention.

<Parallel computer group X>
The nodes # 0 to #n constituting the parallel computer group X have the same configuration. FIG. 3 is a diagram illustrating a configuration example of a node. The node includes a CPU 11, a main memory 12, a calculation processor 13, an external storage device (for example, a hard disk) 14, and a communication interface (communication I / F) 15 that are connected to each other via a bus B 1. The communication I / F 15 is connected to the computer Y and other nodes via a network.

  The node receives the processing target data file transferred from the computer Y by the communication I / F 15 and stores it in the external storage device 14. Further, the node receives a parallel processing command and a setting file from the computer Y via the communication I / F 15.

  Then, the CPU 11 starts executing the parallel processing program stored in advance in the external storage device 3 in accordance with the description of the setting file. A calculation processor 13 is used for the calculation using the processing target data. The calculation processor 13 reads out the processing target data file stored in the external storage device 14 on the MM 12, and executes predetermined processing (for example, extraction of a predetermined area in the data file, calculation of a physical quantity) using the processing target data file. This predetermined process is executed based on the metadata.

  A processing result file is generated by a predetermined process and stored in the external storage device 14. The processing result file stored in the external storage device 14 is moved (transferred) to a predetermined storage location.

  The CPU 11 corresponds to an accepting unit, an extracting unit, and a searching unit according to the present invention, and the external storage device 14 corresponds to a metadata storing unit according to the present invention.

<Data structure of DB and table>
Next, details of the file DB 31, the metadata table 32, and the usage and load situation table (situation table) 33 shown in FIG. 2 will be described.

<< File DB31 >>
The file DB 31 classifies and stores a large amount of simulation data files (hereinafter sometimes simply referred to as “data files”) using a directory structure.

  FIG. 4 is a diagram illustrating an example of the directory structure of the file DB 31. A directory tree starting from the root directory (the directory “data” in FIG. 4) is formed in the file DB 31, and a predetermined directory name is assigned to each directory in the hierarchy. The data file is stored in a directory located at the end of the directory tree and given a predetermined data file name.

  A data file is identified using a file identifier. The file identifier is represented by a list of names (path names) of the directories located on the path (path) of the directory tree from the root directory to the terminal directory and the data file name.

  For example, the file identifier of the data file having the data file name “timeXXX.000.000.dat” in FIG. 4 is “/data/experimentA/3D/statisticsA/variableB/timeXXX.000.000”. As described above, the file identifier includes the storage location information (file path) of the data file.

  In addition, the directory name (“3D”, “statsticsA”, “variableB”, etc.) and the data file name (“timeXXX.000.000”) in the file identifier are keywords that indicate the details (properties, etc.) of the data in the data file. It is prescribed. The keyword is composed of any one or more characters, and is arranged in at least one place in the directory name and the data file name. However, no keyword is set in the extension part of the file name. The keyword functions as a search key for searching for metadata corresponding to the processing target data.

  The data file does not necessarily need to be stored in one storage area, and may be distributed and stored in a plurality of storage areas arranged inside or outside the computer Y.

<Metadata table>
The metadata table 32 stores metadata corresponding to keywords in the file identifier. FIG. 5 is a diagram illustrating a data structure example of the metadata table 32.

  In the example illustrated in FIG. 5, the metadata table 32 includes a plurality of records storing search keys (keywords) and metadata corresponding to the search keys (keywords). The keyword is extracted as a search key from the file identifier of the data file (processing target data file) designated by the user.

  Metadata is information indicating details (properties, attributes, etc.) of simulation data (processing target data) .For example, it is information indicating physical properties of processing target data, statistical processing and time-space (vertical, horizontal, high). It may be information about time (year / month / day). For example, the information on the calculation grid shown in FIGS. 16A and 16B is information about the space. As a keyword representing such calculation grid information, for example, a variable name represented by an arbitrary number of characters is applied.

  FIG. 5 shows a case where one of the directory names included in the file identifier corresponds to one metadata. Instead of this, for example, one metadata may be searched from a combination of a plurality of keywords included in one file identifier. Further, a keyword may be included in a part of the directory name or data file name (excluding the extension), and the keyword may be extracted from the file identifier by the partial match search. A configuration in which a keyword is set only in the file path portion in the file identifier may be employed.

<< Situation Table 33 >>
FIG. 6 is a diagram illustrating an example of the data structure of the situation table 33. The situation table 33 includes a plurality of small tables 34 prepared for each node. Each small table 34 has the same data structure. The small table 34 includes identification information (user ID) of a user who is permitted to use the node, a maximum size of the external storage device (hard disk) of the node that can be used by the user, and an external currently used by the user. It consists of a set of records whose elements (items) are the capacity (load) of the storage device. Each small table 34 is given a node identifier, and information corresponding to the node identifier is stored in the small table.

<User interface (UI)>
In the computer Y shown in FIG. 2, the CPU 1 provides an input environment (UI) for parallel processing designation information to the user of the computer Y through execution of the program.

  The user can use the UI to process the processing target data file group (file identifier), a plurality of nodes that process the processing target data file group, and the processing content for the processing target data file group (elements) of the parallel processing designation information (item). The processing type and detailed parameters), the storage location of the processing result file, etc. can be specified.

  FIG. 7 is a diagram illustrating an example of a designation screen for parallel processing designation information provided as a UI. The designation screen is displayed on the screen of the display device 8 through execution of the program by the CPU 1.

  In the example shown in FIG. 7, the designation screen includes a file path display field 81, a file list display field 82, and a command input field 83. The file path display field 81 displays a directory (file path) in the file DB 31 selected by the user using the input device 7.

  The file list display field 82 displays a list (file list) of data files (data files stored in the terminal directory in the file path) corresponding to the file path displayed in the file path display field 81. . The command input field 83 is used to input a command related to processing for the processing target data file.

  The user can operate the input device 7 to display a desired file path (select a file path) in the file path display field 81. Depending on the selection result of the file path, the display content of the file list display field 82 is changed, and the file list corresponding to the file path is displayed in the display field 82.

  The user can designate the file identifier of the data file to be processed by designating a desired file name from the file list displayed in the file list display field 82 by a cursor operation using the input device 7. At this time, a plurality of data files can be designated at a time through the cursor operation. Thus, the user can specify the file identifier of the processing target data file using the file path display field 81 and the file list display field 82.

  Further, the user can designate and input the node (number of nodes) used for parallel processing, the processing content for the processing target data file group, the storage location of the processing result file, and the like using the command input field 83.

  It should be noted that when specifying the number of nodes, processing parameters, and storage position, options of the specified contents are displayed on the screen, and these are specified by the user selecting a desired option by operating the cursor. it can.

<Generation of script and setting file>
When the user designates each element of the parallel processing designation information using the UI as described above and confirms the designated content, the parallel processing designation information is stored in a parallel processing designation information file described in a predetermined format. Is stored in a predetermined position of the external storage device 3.

  FIG. 8 is a diagram illustrating a description example of the parallel processing designation information file. In FIG. 8, the parallel processing specification information file includes a computer resource specification line, a processing detail (processing content) specification line, a processing target data file, and a processing result storage position specification line corresponding thereto.

  In the computer resource designation line (first line in FIG. 8), an identifier (“NODE”) and an argument (“3” in the example in FIG. 7) indicating the number of nodes used for parallel processing are described.

  In the process detail designation line (second line in FIG. 8), the identifier (“PROC”), the process type (“PROC_A”), and the process parameter (“120.0 150.0 20.0 50.0”) indicating the process details are displayed. The argument to represent is described.

  In addition, in the designated rows (third and fourth rows in FIG. 8) of the processing target data file and the storage location, the identifier (“DATA”), the file identifier of the processing target data file, and the storage location of the corresponding processing result file are stored. Identification information (illustrated by “xxxxx” or “xxxxy”) is described. The designated line is created for each data file to be processed.

  Such a description (parallel processing designation information file) is automatically created by the CPU 1 when the user designates the number of nodes, processing contents, processing target data file group, and storage location using the UI. .

  9, 10 and 11 are flowcharts showing an example of a main routine of script and setting file creation processing executed by the CPU 1 (FIG. 2). The execution of the process is started when, for example, the creation of the parallel processing designation information file is completed or a process start instruction is input from the user.

  When the process shown in FIG. 9 is started, first, the CPU 1 performs an initialization process (step S001). Next, the CPU 1 reads the parallel processing designation information file (FIG. 8) stored in the external storage device 3 into the MM 2 (step S002).

  Next, the CPU 1 executes an analysis loop process for the parallel processing designation information. In this analysis loop process, the CPU 1 extracts specified lines one by one from the parallel processing specification information file, sets the extracted lines as analysis target lines, and analyzes the analysis target lines.

  The CPU 1 determines whether or not the analysis target row extracted from the parallel processing designation information file is a designated row of computer resources (step S003).

  At this time, if the analysis target row is a computer resource designated row (S003; YES), the CPU 1 performs parallel processing on the argument (number of nodes: “3” in the example of FIG. 8) in this analysis target row. The computer resource parameter is determined and stored in a predetermined position (a predetermined work area on MM2) (step S004). Thereafter, the CPU 1 determines the next designated line as an analysis target line, and returns the process to step S003.

  If it is determined in step S003 that the analysis target row is not a computer resource designation row (S003; NO), the CPU 1 determines whether or not the analysis target row is a processing detail designation row (step S005). .

  At this time, if the analysis target line is a process detail designation line (S005; YES), the CPU 1 designates a process type and an argument (designated process parameter: in the example of FIG. “PROC_A” (procedure A) corresponds to the processing type designation, and “120.0 150.0 20.0 50.0” corresponds to the processing parameter), and this processing type and argument are determined as processing parameters related to parallel processing, and the predetermined position (work (Area) (step S006). Thereafter, the CPU 1 determines the next designated line as an analysis target line, and returns the process to step S003.

  If it is determined in step S005 that the analysis target line is not a process detail designation line (S005; NO), the CPU 1 determines that the analysis target line is a process target data file and a storage position designation line, and follows this determination. Then, the file identifier and the storage position identification information in the analysis target line are extracted and stored in a predetermined position (work area) (S007).

  The analysis loop process described above ends when the process for the last line of the parallel processing designation information file ends. Subsequently, the CPU 1 advances the processing to step S008 in FIG.

  In step S008, the CPU 1 outputs the header portion of the parallel processing job script. The header is stored in a predetermined position in the external storage device 3 in advance as a fixed sentence. The header includes a setting file transfer instruction. In step S008, a node used for parallel processing is determined based on the number of data files to be processed and the number of nodes used for the specified parallel processing. The usage and load status for each of the nodes # 0 to #n is managed by, for example, the OS (operating system) of the computer Y. The number of processing target data files and the number of nodes in the parallel processing designation information file are delivered to the OS.

  For example, the OS extracts nodes that are allowed to be used by the user from the nodes # 0 to #n, and considers the use of the plurality of extracted nodes, the load status, and the number of files, and the number of nodes specified. Select. For example, a specified number of nodes are determined as nodes to be used for parallel processing in the order of decreasing load from the extracted nodes. The determined usage and load status of each node is set as a small table 34 in the status table 33. As a result, the processing target data file group is processed in parallel by the specified number of nodes determined by the OS.

  The status table 33 (FIG. 6) stores a small table 34 for all the nodes # 0 to #n, and the OS refers to the small table 34 so that nodes corresponding to the designated number of nodes are arranged in order of decreasing load. The mask may be set in the small table 34 corresponding to the node that has been selected and not selected (cannot be referred to).

  Subsequently, the CPU 1 executes a loop process for analyzing and processing the processing target data file. The loop processing is executed for each file identifier (processing target data file) obtained in step S007. In this loop, the CPU 1 first identifies one of the specified processing target data file groups (processing target data file group having the file identifier obtained in step S007) (referred to as an analysis target file). Subsequently, the CPU 1 activates a subroutine for the metadata analysis processing of the analysis target file (step S009), and passes the file identifier of the analysis target file to the subroutine.

  FIG. 12 is a flowchart illustrating an example of a metadata analysis / acquisition subroutine. In FIG. 12, first, the CPU 1 accepts an input for specifying a data file (step S101). That is, the CPU 1 receives the file identifier of the analysis target file.

  Next, the CPU 1 determines whether or not the file identifier has a correct format (step S102). At this time, if the file identifier does not have the correct format (S102; NO), it is determined that the process has failed (NG), and the script and setting file creation process ends. In this case, an error display process can be performed to notify the user of the error.

  On the other hand, if the file identifier is in the correct format (S102; YES), the CPU 1 starts a keyword acquisition loop process. In the loop processing, first, the CPU 1 determines whether or not a keyword representing metadata is included in the file identifier (S103).

  For example, the CPU 1 extracts the directory name next to the root directory in the file identifier, and lists the directory name and a keyword list (keyword group stored in the metadata table 32) in the metadata table 32 (FIG. 5). And search for a keyword that matches the extracted directory name.

  At this time, if the keyword cannot be searched, the CPU 1 extracts the next directory name and collates with the keyword list. In this manner, the CPU 1 repeats the directory name or data file name extraction processing and the keyword list matching processing described above until a directory name or data file name that matches one of the keywords is found.

  If the CPU 1 finds a keyword that matches the extracted directory name or data file name (S103; YES), the CPU 1 interrupts the extraction process, and retrieves and acquires metadata corresponding to the keyword from the metadata table 32 ( Step S104).

  For example, for the file identifier “/data/experimentA/3D/statisticsA/variableB/timeXXX.000.000.dat” (FIG. 4), the above processing is performed using the metadata table 32 of the stored contents shown in FIG. When the directory name “3D” is extracted from the file identifier and collated with the keyword list, the metadata “meta01” corresponding to “3D” is acquired from the metadata table 32.

  When acquiring the metadata from the metadata table 32, the CPU 1 resumes the extraction of the directory name or data file name and the matching process with the keyword list for the file identifier. Accordingly, for example, the corresponding metadata “meta1” is acquired from the metadata table 32 using the directory name “statisticsA” next to the directory name “3D” as a keyword.

  Thereafter, the keyword acquisition loop process ends at the time when the collation process for the data file name is completed (when the matching keyword is searched, the acquisition of the corresponding metadata is completed) (S104; NO), the subroutine (S009) shown in FIG. 12 ends, and the process returns to step S010 (FIG. 10) of the main routine.

  In this way, when the user specifies the file identifier of the processing target data, the computer Y automatically specifies (acquires) the metadata corresponding to the processing target data using the property information (keyword) included in the file identifier. To do.

  In step S010, the CPU 1 analyzes the metadata, and in parallel processing of the processing target data file (analysis target file) that is the target of the loop processing, the CPU 1 relates not only to the analysis target file but also to the analysis target file. It is determined whether or not data (related data file) to be used is necessary.

  For example, when the fluid flow velocity calculation is executed in parallel processing, the velocity X component, Y component, and Z component are required. Here, if the analysis target file that is the target of the determination process in step S010 is a data file that indicates the X component of the velocity, each data file that indicates the Y and Z components is required as the related data file.

  Here, the file identifier can include a character or a character string indicating component information indicating whether it is an X component, a Y component, or a Z component in a directory name or a data file name. The file identifier of the data file of the Y component and the Z component corresponding to the data file of a certain component (for example, the X component) is a description of the character or character string of the component information in the file identifier of the data file of the X component. Created by changing. For example, if the character “X” of the component information included in the file identifier is replaced with the characters “Y” or “Z” indicating the Y component or Z component, the file of the corresponding Y component or Z component data file It becomes an identifier.

  In step S010, if the CPU 1 finds that the analysis target file is, for example, an X component data file through the analysis of the metadata obtained in step S009, it determines that a related data file is necessary (S010; YES), the process proceeds to step S011. Otherwise (S010; NO), the CPU 1 advances the process to step S012.

  In step S011, the CPU 1 generates a file identifier of the related data file. For example, as described above, the file identifier of the related data file can be generated by changing a part of the file identifier of the analysis target file. The file identifier of the generated related data file is stored in the work area on the MM 2 as a set with the file identifier of the analysis target file.

  The related data file is stored in the file DB 31 so that the actual related data file is stored on the file path indicated by the file identifier of the generated related data file. Thereafter, the process proceeds to step S012.

  In step S012, a subroutine for determining the arrangement of the analysis target file (designated data file) or the analysis target file and the related data file is executed.

  FIG. 13 is a flowchart illustrating an example of an arrangement determination subroutine (S012). In FIG. 13, when the process is started, the CPU 1 first estimates the size of the data file arranged in the node and the computer resource A required for the process (step S201).

  That is, the CPU 1 acquires the size of the analysis target file (for example, obtained from metadata). Subsequently, the CPU 1 approximates the size of the processing result file created when the processing specified by the processing detailed parameter obtained in step S006 (FIG. 9) is executed according to the corresponding metadata for the analysis target file. . The CPU 1 calculates the total value of the size of the analysis target file and the size of the processing result file as the computer resource A.

  The size of the processing result file is determined from the extraction range, for example, when the processing content specified by the processing detail parameter is processing for extracting a part of the analysis target file from the specified extraction range.

  When there is a related data file for the analysis target file, it is preferable to increase the processing efficiency that the analysis target file and the related data file are processed by the same node. For this reason, when the related data file exists in step S201, the size of the related data file and the size of the processing result file for the related data file are also included in the computer resource A. The size of the related data file and the size of the processing result file corresponding thereto can be estimated from the size of the analysis target file and the size of the processing result file corresponding thereto, for example.

  Next, the CPU 1 refers to the situation table 33 (FIG. 6), can provide the user with a capacity corresponding to the computer resource A, and is predicted to have the lightest load in the current load distribution situation. A node is searched (step S202).

  That is, the CPU 1 refers to the situation table 33 and refers to the user record in each small table 34. For example, when the use of the simulation system is started, the user ID has already been input to the computer Y by the user, and the CPU 1 refers to the record corresponding to the user ID.

  Next, the CPU 1 calculates the remaining usable size of the user in each node by subtracting the load (current used size) from the maximum size in each record. Subsequently, the CPU 1 determines the node having the largest usable size (the smallest load) as the node where the analysis target file (and the related data file) should be arranged.

  Next, the CPU 1 updates the status table 33 based on the computer resource A (step S203). That is, the CPU 1 adds the value of the computer resource A to the load value (used size) of the small table 34 corresponding to the determined node.

  For example, in the example shown in FIG. 6, when it is determined that the computer resource A of the user A (for example, 10 gigabytes) is allocated to the node # 0, the load value in the corresponding small table 34 Is updated to "20Gbyte".

  When the update of the status table 33 is finished, the CPU 1 finishes the processing of the subroutine, and passes the identifier of the node determined as the file placement destination to the main routine.

  When the process proceeds to step S013 of the main routine, the CPU 1 outputs a command statement relating to data placement on the node (referred to as “data placement command statement”).

  That is, the CPU 1 reads a template of the data arrangement command statement (stored in the external storage device 3 in advance). The template is configured so that a statement is completed when a file identifier and a node identifier to be arranged are described at predetermined positions of a fixed statement. The CPU 1 describes the identifier of the analysis target file (and related data file) at a predetermined position of the template and also describes the node identifier obtained in step S012. In this way, the completed data arrangement command becomes part of the parallel processing job script.

  Next, after completion of the parallel processing, the CPU 1 outputs a command statement (referred to as “processing result transfer command statement”) that moves the processed data (processing result file) to the storage position (step S014).

  That is, the CPU 1 reads a template of the processing result movement command statement (stored in the external storage device 3 in advance). The template is configured such that when a storage position specified by the UI is described at a predetermined position of a standard command statement, the command statement is completed. The CPU 1 writes the storage position of the processing result file for the analysis target file obtained in step S007 at a predetermined position of the template. In this way, the completed processing result movement command statement becomes a part of the parallel processing job script.

  Next, the CPU 1 stores data arrangement information (step S015). That is, the CPU 1 stores the correspondence relationship between the file identifier and the node identifier as data arrangement information in a predetermined storage area.

  If there is a file identifier of the processing target data file that is not the analysis target file when step S015 ends, the processing returns to step S009, and the above-described processing of steps S009 to S015 is executed. When the processes for the file identifiers of all the data files to be processed are completed, the process proceeds to step S016.

  By the loop processing described above, the placement destination of each processing target data file included in the processing target data file group is determined so that the load in parallel processing is minimized.

  In step S016, the CPU 1 outputs a parallel processing program execution statement. That is, the CPU 1 reads a parallel processing program execution statement stored in advance in the external storage device 3 and sets it as a part of the parallel processing job script. In this manner, a parallel processing job script including a header, a data arrangement command statement, a processing result movement command statement, and a parallel processing program execution statement is automatically generated.

  Next, the CPU 1 starts processing for creating a setting file for the parallel processing program (step S017: FIG. 11). The CPU 1 starts a parallel processing program setting creation loop process. This loop process is executed for each data file to be processed.

  When the processing is started, the CPU 1 creates a setting for the processing target data file based on the data arrangement information (correspondence between the file identifier and the node identifier) (S018).

  That is, the CPU 1 extracts a portion related to one processing target data file from the data arrangement information obtained in step S015 and combines it with the processing parameter (obtained in step S006) corresponding to this file identifier. The CPU 1 describes the result of the combination in a predetermined format for the setting file.

  The CPU 1 performs such processing for each processing target data file, and when the processing of step S019 for all processing target data files ends, the main routine ends.

  FIG. 14 is a diagram illustrating a description example of a setting file for a parallel processing program. In the example illustrated in FIG. 14, the setting file includes a plurality of lines described for each processing target data file.

  In each line, a node identifier, a process designation (in this example, “PROC_A”), a file identifier of a processing target data file, and a processing parameter are described in order from the left in FIG. Such a setting file is referred to when each node executes a parallel processing program.

<Run script>
When the creation of the script and the setting file is completed, the CPU 1 starts executing the script. By executing the script, the computer Y transfers the setting file to each node of the parallel computer group X according to the setting file transfer command statement in the header.

  In addition, the computer Y transfers each processing target data file (processing target data file group) stored in the file DB 31 to the placement destination node according to the data placement information by executing the data placement command.

  Also, the computer Y executes the processing result movement command statement, and for each node, stores the processing result file (processed data) created by processing the processing target data file at each node at the designated storage location ( For example, it is instructed to be stored in the file DB 31.

  In addition, the computer Y instructs each node to start executing the parallel processing program by executing the parallel processing program execution statement.

<Parallel processing>
Each node (FIG. 3) where the processing target data file group is arranged receives the setting file and the processing target data file from the computer Y via the network. These are stored in the external storage device 14 in the node. Thereafter, when the CPU 11 of each node receives an instruction to execute the parallel processing program from the computer Y, the CPU 11 starts executing the parallel processing program.

  FIG. 14 is a flowchart showing execution processing of a parallel processing program executed by the CPU 11. When the process shown in FIG. 14 is started, the CPU 11 first executes an initialization process (step S301). When the initialization is completed, the CPU 11 reads the setting file stored in the external storage device 14 into the MM 12 (step S302).

  Next, the CPU 11 executes a processing loop of the processing target data file according to the setting file. In this processing loop, the CPU 11 sets one line in the setting file as a processing target line, and executes processing for the processing target data file according to the setting contents described in the processing target line.

  In the loop, first, the CPU 11 refers to the node identifier in the setting file, and determines whether or not this node identifier is equal to the identifier of the own node (step S303).

  At this time, when the node identifiers are not equal (S303; NO), the next line in the setting file is set as a process target line, and the process of step S303 is executed.

  On the other hand, if the node identifiers are equal (S303; YES), the CPU 11 performs a process of acquiring metadata corresponding to the file identifier described in the processing target line (step S304).

  The processing in step S304 is the same processing as the subroutine shown in FIG. That is, the CPU 11 refers to the metadata table 32A (the data structure is the same as that of the metadata table 32 (FIG. 5)) stored in the external storage device 14, and searches and acquires corresponding metadata.

  Next, the CPU 11 executes processing for the processing target data file in accordance with the processing type designation, processing parameter, and metadata in the processing target row (step S305). That is, the CPU 11 gives a processing type designation, a processing parameter, a file identifier, and metadata to the calculation processor 13. Then, the calculation processor 13 reads the processing target data file corresponding to the file identifier from the external storage device 14 to the MM 12, and executes processing according to the processing type designation and processing parameters based on the metadata.

  Thereafter, when the processing by the calculation processor 13 is completed, the CPU 11 outputs the processing result data (processed data) as a processing result file (step S306). The processing result file is transferred to, for example, the computer Y, and the computer Y stores the processing result file in a storage location designated by the user (for example, prepared in the file DB 31).

  The above-described processing is performed with each row in the setting file as a processing target row, and when the processing for all the rows is completed, the execution processing of the parallel processing program is finished.

<Modification>
In the above-described embodiment, the case where the computer Y and each node have the metadata table has been described. Instead of such a configuration, a configuration in which metadata acquired by the computer Y is transferred to each node may be applied.

  In the present embodiment, the example in which the storage area of the processing target data file (simulation data file) is provided on the external storage device 3 of the computer Y has been described. The storage area may be included in each node, and may be provided on a file server independent of the computer Y and the parallel computer group X.

<Effects of Embodiment>
According to this embodiment, when a user specifies a file identifier, the number of nodes, a processing type, a detailed processing parameter, and a storage location using an input environment (UI) for parallel processing specification information, parallel processing for a processing target data file group is performed. A control program (script) and a parallel program execution setting file are automatically created.

  In the past, when executing parallel processing, the user had to write a script of several hundred lines or more, including data file transfer control.

  According to the present embodiment, a desired script and a setting file are automatically created simply by designating or inputting information that is an element of the above-described parallel processing designation information using a UI. This can greatly reduce the user's labor. Further, since the time required for script description is reduced, the time required for obtaining the parallel processing result can be reduced. Furthermore, it is possible to eliminate the possibility of redoing parallel processing due to a description mistake of the user.

  Further, the metadata for the processing target data is automatically retrieved and acquired by designating the file identifier by the user. That is, when a user specifies a file identifier, a keyword is extracted from the file identifier, and metadata corresponding to the keyword is handled as the specified metadata. This eliminates the need for the user to input metadata designation for each data file to be processed. Therefore, it is possible to reduce the user's labor, shorten the processing time, and prevent the user's input error.

  In the automatic specification of metadata, in the present embodiment, a file identifier including data storage location information (file path) is applied to the processing target data file, and a keyword indicating the nature of the processing target data (keyword for metadata search) Is included.

  That is, data associating process target data and metadata is embedded in the file identifier. This eliminates the need to manage the associated data separately from the processing target data and the metadata. Therefore, effective use of the storage area and reduction of the management burden can be achieved. A file identifier can include a plurality of keywords.

  Further, when the user designates a processing target data file, the user designates a file identifier including a file path. Thereby, the designation of the file identifier also serves as a keyword input. Therefore, the work burden on the user can be reduced.

  Further, in the present embodiment, the metadata is configured to be stored in a storage area different from the processing target data file. As a result, the processing target data file can be efficiently stored in the storage area.

  The present invention can be applied to data processing in various numerical simulation systems, for example.

Claims (4)

Receiving means for receiving designation of a file identifier selected from a list in which a plurality of file identifiers of data files including a file path of a data file stored in one of a plurality of directories constituting a directory structure are displayed ;
Metadata storage means for storing metadata representing details of the data file;
Extraction means for extracting a keyword for searching for metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier;
Search means for searching for metadata corresponding to the extracted keyword from the metadata storage means ;
A keyword list having keywords associated with the metadata;
It includes,
The extraction means compares a character string that forms part of a file path portion of the specified file identifier with the keyword list, and extracts a character string that matches at least one keyword in the keyword list as a keyword. to <br/> metadata search apparatus. The metadata search device according to claim 1, wherein the metadata storage means is stored in a storage area different from a storage area of the data file to which the data file identifier is assigned. Computer
Accepts the specification of a file identifier selected from a list in which multiple file identifiers of data files including the file path of the data file stored in one of the multiple directories constituting the directory structure are displayed ,
A keyword for searching for metadata corresponding to the data file specified by the specified file identifier is extracted from the file path portion of the specified file identifier,
Retrieving metadata corresponding to the extracted keyword from the metadata storage means;
Characters that match at least one keyword in the keyword list by comparing a character string that forms part of the file path portion of the specified file identifier with a keyword list that includes a keyword group associated with metadata A metadata search method in which columns are extracted as keywords . On the computer,
Receiving a designation of a file identifier selected from a list in which a plurality of file identifiers of data files including file paths of data files stored in one of a plurality of directories constituting the directory structure are displayed ;
Extracting a keyword for searching metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier;
A program that executes a step of searching the metadata corresponding to the extracted keyword from the metadata storage means ,
In the extracting step, at least one of the keyword lists is compared by comparing a character string forming a part of a file path portion of the specified file identifier with a keyword list including a keyword group associated with metadata. A program in which character strings that match one keyword are extracted as keywords .

Images