JP4494892B2 - Data processing apparatus and program - Google Patents

Description

  The present invention relates to a data storage technique for storing an input file in a predetermined database.

As a conventional data storage method, a method for improving the efficiency of data storage by providing additional information related to data storage in addition to an input file has been proposed (for example, Patent Document 1). The method proposed here refers to the “database conversion table” that specifies the correspondence between each item of the database and each item of the extraction pattern and the conversion format in the character string of the extraction pattern extracted from the input file. Processing to convert to a character string according to the database specifications.
Japanese Patent Laid-Open No. 7-152780

  The conventional data storage system provides additional information related to data storage in order to improve the efficiency of data type conversion, and does not improve the efficiency of storage with respect to data arrangement. In the conventional method, in order to perform data arrangement such that one data is stored in a plurality of records, there is a problem that the input file needs to be shaped in advance so as to correspond to the database format. In a similar pattern search of time series data, a plurality of data values of a certain item are stored in one record in succession, and the next record has the same number of data values as one record again from a position shifted by a certain period. A storage format is used. In such a case, it is necessary to perform data shaping processing in accordance with the database format in advance.

  The present invention has been made to solve the above-described problems, and stores a plurality of data of the same item in one record and data that stores one data corresponding to a plurality of records. The main purpose is to realize the storage method.

The data processing apparatus according to the present invention
A plurality of data corresponding to each other, a buffer unit for classifying and storing the data according to the data type of each data;
A control information reading unit for reading out control information indicating the number of continuous readings when reading data from the buffer unit for each data type;
A plurality of data corresponding to each other to be read is specified, a plurality of data corresponding to each other to be read is read from the buffer unit, and the number of continuous readings indicated in the control information is plural And a data reading unit for continuously reading out data of the data type from the buffer unit according to the number of continuous readings of the data type.

  According to the present invention, for data of a data type for which a plurality of continuous reading numbers are designated, the data of the data type is continuously read, so that it is possible to specify the data without performing a process in advance according to the database format. Can be stored in one record continuously.

Embodiment 1 FIG.
FIG. 1 is a diagram showing a configuration of a data storage device (data processing device) according to the present embodiment. The data storage device according to the present embodiment performs processing for creating and storing a record in a specified format from a given input file and control file.

  In FIG. 1, a data storage device 1 includes an input file reading device 11, a record buffer management device 12, a control file reading device 13 (control information reading unit), a record creation device 14, a record writing device 15 (data writing unit), and The input file 4 and the control file 3 are input from the record buffer 16 (buffer unit). The control file 3 includes input data value information 6 describing the data arrangement and data type in the input file, data arrangement information 7 (control information) describing how to form a record from the data of the input file, and It consists of database definition information 8 describing the table name, column name, and data type of the database that actually stores the data.

  FIG. 2 shows a configuration example of the input data value information 6 in the present embodiment. The input data value information 6 includes a serial number 21, data type information 22, a data length 23 when the data type is a character type and a date type, and an accuracy 24 when the data type is a numeric type. Here, the precision represents the range that can be expressed when the data type is a numerical value (such as INTERGER), and in the example of FIG. 2, means that the numerical value is expressed by 32 bits.

  FIG. 3 shows a configuration example of the data arrangement information 7 in the present embodiment. The data arrangement information 7 includes a serial number 31, a continuous reading number 32, a next record offset 33, and a storage method designation 34. The numerical value described in the storage method designation 34 indicates the order in which the read data is stored as a record.

  FIG. 4 shows a configuration example of the database definition information 8 in the present embodiment. The database definition information 8 includes table name information 41, column name information 42, data type information 43, a data length 44 when the data type is a character type and a date type, and an accuracy 45 when the data type is a numeric type. Is done.

  FIG. 6 is a configuration diagram of the record buffer management device according to the present embodiment. The record buffer management device 12 includes a record dividing device 71, a record buffer writing device 72, a record buffer reading device 73 (data reading unit), and a data arithmetic unit 74 (data reading unit).

  Next, the operation will be described.

  FIG. 5 is a flowchart of the data storage method in this embodiment.

  First, in step 52, the data storage device 1 reads the control file 3 by the control file reading device 13 (control information reading step). If there is no unprocessed data in the record buffer 16 and not all the data in the input file 4 has been processed, in step 55, the input file reading device 11 reads the data from the input file. The record buffer management device 12 updates the record buffer for the read data (data accumulation step) in step 56 and the necessary data from the current position of the record buffer based on the information in the control file 3 in step 57. In reading (data reading step), in step 58, the current position in the record buffer is updated.

  The operation from step 56 to step 58 will be described with reference to FIG. Assume that the input file 4 is data as shown in FIG. 6, and the input data value information 6 and the data arrangement information 7 are in the formats shown in FIGS. 2 and 3, respectively. Note that “AA, 1,200,3003”, etc. of the input file correspond to a plurality of data having a corresponding relationship with each other. The record dividing device 71 divides the data read via the input file reading device 11 for each item. For example, in the case of “AA, 1,200,301”, it is divided into “AA”, “1”, and “20030301”. Since the record buffer 16 prepares buffers as indicated by the buffer 75 to the buffer 77 for each item (for each data type), the record buffer writing device 72 classifies the divided data for each item and corresponds. Write to the record buffer (step 56). Each record buffer has a pointer indicating the current position, and the record buffer reading device 73 (data reading unit) obtains data corresponding to the number of continuous readings specified by the data arrangement information 7 from the current position of each record buffer. At the same time as reading (step 57), an update process for advancing the current position of each record buffer by the next record offset specified by the data arrangement information 7 is performed (step 58).

If unprocessed data exists in the record buffer in step 53, necessary data is read from the current position of the record buffer in step 57.
In the present embodiment, in the data arrangement information 7 shown in FIG. 3, the storage method designation 34 for each item is all the designation for arranging the read data in order, so that the read data is the data calculation device 74. In step 59, the record creation device 14 simply arranges and creates a record image. In step 60, the record writing device 15 writes the record via the database management system 2. Here, the record writing device 15 refers to the database definition information in FIG. 4 and writes a record.
In the present embodiment, the record image created by the record creation device 14 has the format shown in FIG. That is, serial number 1 in FIG. 3 corresponds to “CHARACTER” in serial number 1 in FIG. 2, and data “AA” of “AA, 1,200,3101” is the target. Further, since the number of continuous reading is “1”, only “AA” is read. The serial number 2 in FIG. 3 corresponds to the “INTEGER” in serial number 2 in FIG. 2, and the data “1” of “AA, 1,200,2101” is the target. Further, since the number of continuous readings is “8”, the data of the same item after “AA, 1,200,3101” is continuously read. As a result, “1, 2, 3, 4, 5, 6, 7, 8 "is obtained. The serial number 3 in FIG. 3 corresponds to “DATE” in the serial number 3 in FIG. 2, and the data “20030301” of “AA, 1,200,2101” is targeted. Further, since the number of continuous readings is “1”, only “20030101” is read. Further, since the next record offset is “1” in any of the serial numbers 1 to 3 in FIG. 3, the same processing is performed on “BB” and “20030102”, and the result of FIG. 7 is obtained in this way.

  If it is determined in step 54 that all the data of the input file has been processed, the data storage process of the present embodiment ends.

  As described above, the data storage device 1 according to the present embodiment includes a record buffer (buffer unit) that classifies and stores a plurality of data (input files) that correspond to each other according to the data type of each data. The control file reading unit (control information reading unit) for reading data arrangement information (control information) indicating the number of continuous readings for each data type when reading data from the record buffer is in a mutual correspondence relationship Specify a plurality of data, read a plurality of data corresponding to each other to be read from the record buffer, and, for a data type having a plurality of continuous reading numbers indicated in the data arrangement information, continuously According to the number of reads, a record buffer that continuously reads data of that data type from the record buffer. And a § reading device (data reading portion).

  As described above, in the data storage process, the number of continuous reads and the storage method designation are separately given as information regarding the data arrangement, so the process of shaping the input file according to the database format is not performed in advance. In addition, data of an item in the input file can be continuously stored in one record.

  In addition, by specifying the offset of the data to be stored in the next record, it is possible to store one data in multiple records without processing the input file according to the database format in advance. Become.

  In order to perform a similar pattern search of time-series data, it is required to store data values for a continuous period in a single record. This will be described with reference to FIG. It is assumed that the time series data 81 is stored in the input file 82 in the format shown in FIG. At this time, when time series data in eight measurement periods is cut out and a similar pattern search for calculating the degree of similarity of the cut out pattern with respect to other patterns is performed, data in a format as shown in records 83 to 86 is used. Storage is performed. That is, each record stores time and eight consecutive measurement values, and the next record stores eight measurement values continuously from a position shifted by one measurement period. In the case of data storage corresponding to a similar pattern search of such time series data, it is possible to realize data storage without the need to shape the input file in advance by this embodiment that provides data arrangement information. .

Embodiment 2. FIG.
In the first embodiment described above, a plurality of continuous data of a certain item is stored in one record. However, in the case where the value of the operation result between the next consecutive data is stored in the record. The form is shown.

  FIG. 9 shows data arrangement information in such a case. In the item indicated by the serial number “2”, the value “Diff” indicated by the storage method designation 94 means that a difference value from the data immediately after the read data is calculated.

  FIG. 11 is a configuration diagram of the record buffer management device according to the present embodiment. In the record buffer 16, an operation result buffer 120 for storing data operation results is prepared.

  Next, the operation will be described.

  FIG. 10 is a flowchart of the data storage method in this embodiment.

  First, in step 102, the data storage device 1 reads the control file 3 by the control file reading device 13. If there is no unprocessed data in the record buffer 16 and all the data in the input file 4 has not been processed, the input file reading device 11 reads the data from the input file in step 105. The record buffer management device 12 updates the record buffer for the read data in step 106, reads the necessary data from the current position of the record buffer based on the information in the control file 3 in step 107, and in step 108 Update the current position in the record buffer.

  The operation in steps 106 to 108 will be described with reference to FIG. Assume that the input file 4 has data as shown in FIG. 11, and the input data value information 6 and the data arrangement information 7 have the formats shown in FIGS. 2 and 9, respectively. The record dividing device 71 divides the data read via the input file reading device 11 for each item. Since the record buffer 16 prepares buffers as indicated by the buffer 75 to the buffer 77 for each item, the record buffer writing device 72 writes the data divided for each item to the corresponding record buffer (step 106). . Each record buffer has a pointer indicating the current position, and the record buffer reading device 73 reads data corresponding to the number of continuous readings specified by the data arrangement information 7 from the current position of each record buffer (step 107). At the same time, an update process for advancing the current position of each record buffer by the next record offset specified by the data arrangement information 7 is performed (step 108).

  In step 109, the data operation device 74 (data operation unit) performs an operation on the data read out as many as necessary. In this embodiment, since the difference value calculation “Diff” is included in the storage method designation of the item of the serial number “2”, the data calculation device calculates the difference value between the two read data, and calculates in the record buffer. The result of operation is stored in the result buffer 120. In step 110, the record creation device 14 creates a record image using the contents of the calculation result buffer 120, and in step 111, the record writing device 15 writes the record via the database management system 2. In the present embodiment, the record image created by the record creation device 14 has the format shown in FIG.

  As described above, in the data storage process, the number of continuous reads and the storage method specification including the calculation information are separately given as the information related to the data arrangement, so the input file is formatted according to the database format beforehand. The result of the arithmetic processing between continuous data of a certain item in the input file can be stored as a record value.

  By specifying the difference calculation as the calculation information, it is possible to store the increment value of a certain item value in the record, and realize the trend analysis using the increment value without performing the input file shaping process in advance. Data storage can be realized.

  In each of the embodiments described above, the data storage device can be realized by a computer.

  Although not shown, the data storage device includes a CPU (Central Processing Unit) that executes a program.

  For example, the CPU is connected to a ROM (Read Only Memory), a RAM (Random Access Memory), a communication board, a display device, a K / B (keyboard), a mouse, an FDD (Flexible Disk Drive), and a CDD (Compact Disc) via a bus. Drive), magnetic disk device, optical disk device, printer device, scanner device and the like.

  The magnetic disk device stores an operating system (OS), a window system, a program group, and the like.

  The program group is executed by a CPU, OS, and window system.

  Each part of the data storage device may be constituted by a part or all of a program operable by a computer. Alternatively, it may be realized by firmware stored in the ROM. Alternatively, it may be implemented by software, hardware, or a combination of software, hardware, and firmware.

  The program group stores a program that causes the CPU to execute the processing described as “˜unit” in the description of the embodiment. These programs are created by computer languages, such as C language, HTML, SGML, and XML, for example.

  The program is stored in another recording medium such as a magnetic disk device, FD (Flexible Disk), optical disk, CD (compact disk), MD (mini disk), DVD (Digital Versatile Disk), and read by the CPU. And executed.

1 is a block diagram illustrating a configuration example of a data storage device according to a first embodiment. FIG. 10 is a diagram illustrating a configuration example of input data value information in the first and second embodiments. 6 is a diagram illustrating a configuration example of data arrangement information according to Embodiment 1. FIG. 6 is a diagram illustrating a configuration example of database definition information in Embodiment 1. FIG. FIG. 6 is a flowchart showing a data storage processing procedure in the first embodiment. 3 is a block diagram illustrating a configuration example of a record buffer management device in Embodiment 1. FIG. FIG. 6 is a diagram showing a stored record image in the first embodiment. It is explanatory drawing which shows the storage image of time series data. 10 is a diagram illustrating a configuration example of data arrangement information according to Embodiment 2. FIG. FIG. 11 is a flowchart showing a data storage processing procedure in the second embodiment. 10 is a block diagram illustrating a configuration example of a record buffer management device according to Embodiment 2. FIG. It is a figure which shows the storage record image in Embodiment 2. FIG.

Explanation of symbols

  1 data storage device, 2 database management system, 3 control file, 4 input file, 5 database, 6 input data value information, 7 data allocation information, 8 database definition information, 11 input file reading device, 12 record buffer management device, 13 Control file reading device, 14 record creating device, 15 record writing device, 16 record buffer, 71 record dividing device, 72 record buffer writing device, 73 record buffer reading device, 74 data operation device, 75 buffer, 76 buffer, 77 buffer, 120 Operation result buffer.

Claims (7)

Means for inputting a file containing a plurality of data of a plurality of data types;
A buffer unit that classifies a plurality of data for each file according to a data type , and accumulates a plurality of files in the order of input ;
A control information reading unit that reads the number of data to be read from the buffer unit at the time of data reading for each data type, and reads control information indicating a plurality of data to be read for at least one data type ;
A data processing apparatus comprising: a data reading unit for reading out data corresponding to the number indicated in the control information from the buffer unit in accordance with an input order of the file to the buffer unit for each data type . The data processing device further includes:
The data processing apparatus according to claim 1, further comprising a data writing unit that writes the data of each data type read by the data reading unit to a predetermined data writing destination in association with each other. The control information reading unit
Read out the control information that specifies the execution of a predetermined operation for the data type indicated as the number of data to be read,
The data processing device further includes:
For the data type operations are designated in the control information, a data calculation unit for performing an operation specified in the control information by using a plurality of data Desa read by the data reading unit,
An operation is performed by the data operation unit on data read by the data reading unit for a data type for which no operation is specified in the control information and a data type for which an operation is specified in the control information. The data processing apparatus according to claim 1, further comprising: a data writing unit that associates the calculation result after being transmitted with each other and writes the result in a predetermined data writing destination . In the buffer part,
There is a pointer for each data type,
The data reading unit
2. The data processing apparatus according to claim 1, wherein the number of data indicated by the control information is read from a pointer position for each data type. In a computer having a buffer unit for storing data,
Processing to input a file containing multiple data of multiple data types,
A data accumulation process for classifying a plurality of data for each file according to a data type, and accumulating a plurality of files in the buffer unit in an input order ;
A control information read process for reading the number of data to be read from the buffer unit at the time of data read for each data type, and for reading at least one data type, the control information indicating a plurality as the number of data to be read;
A program for executing , for each data type, a data reading process for reading out data corresponding to the number indicated in the control information from the buffer unit in accordance with a file input order to the buffer unit . The program further includes:
6. The program according to claim 5, which causes the computer to execute a data writing process in which data of each data type read by the data reading process is associated with each other and written to a predetermined data writing destination. The control information reading process includes
Causing the computer to read control information that designates the execution of a predetermined operation for a data type in which a plurality of pieces of data to be read are indicated;
The program further includes:
A data calculation process for performing a calculation specified in the control information using a plurality of data read out by the data read process for a data type in which the calculation is specified in the control information;
An operation is performed by the data operation process on the data read by the data read process for a data type for which an operation is not specified in the control information, and a data type operation is specified by the control information. 6. The program according to claim 5, which causes the computer to execute a data writing process in which a calculation result after being transmitted is associated with each other and written to a predetermined data writing destination.

Images