JP3886872B2 - Access log information multidimensional analysis system and multidimensional analysis environment construction method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an access log information multidimensional analysis system that performs multidimensional analysis of log information output from a network device, and in particular, can provide an efficient multidimensional analysis environment for the log information in the shortest time when necessary. The present invention relates to an access log information multidimensional analysis system and a multidimensional analysis environment construction method.
[0002]
[Prior art]
Normally, network access log information output from a network device is in a flat file format, which is a list of character strings, as shown in FIG. 6, for example, and the network access log information is always output while the network is operating. Is done. Therefore, the network administrator checks all of the output network access log information in a large amount of flat file format and confirms whether there is unauthorized access.
[0003]
In addition, when the network administrator grasps the network usage status on a desired day or time zone, the network access log information on the desired day or time zone is retrieved each time, and the retrieved data is read into a spreadsheet program or the like. In other words, processing such as graphing was performed to grasp the network usage status under desired conditions.
[0004]
Therefore, in order to improve the efficiency of work imposed on the network administrator and to realize detailed analysis, it has been attempted to perform multidimensional analysis of network access log information using an OLAP (OnLine Analytical Processing) system.
[0005]
Here, in order to perform multidimensional analysis of network access log information by the OLAP system, all the operations shown in FIG. 8 must be performed by the analyst (network administrator) himself. Hereinafter, the work process when performing multidimensional analysis of network access log information will be specifically described with reference to FIGS. FIG. 7 is a diagram showing a conventional multidimensional analysis environment, and FIG. 8 is a flowchart showing a conventional multidimensional analysis processing procedure.
[0006]
First, the person in charge of analysis performs processing for extracting information necessary for analysis from the network access log information in the flat file format sequentially stored in each network device to the access log analyzer (step ST21). Next, the analyst converts the extracted access log information extracted in the access log analyzer from a flat file format to a table format in a relational database in order to perform multidimensional analysis by the OLAP system ( Step ST22). In this way, table format access log information is created on the relational database. Further, an OLAP database is constructed using the created table format access log information (step ST23). In this way, access log information in a multidimensional analysis-capable format can be obtained.
[0007]
Through the above processing steps, the creation of the multidimensional analysis environment is completed, and the multidimensional analysis processing can be performed. Then, the person in charge of the analysis performs multidimensional analysis processing (step ST24), and as a result, if there is necessary log information other than the information extracted in the log extraction processing in step ST21 (step ST25), the person in charge of analysis The operator again performs an extraction process for extracting necessary log information from each network access log information in the flat file format, performs the processes from step ST21 to step ST24 again, and requires other access log information in step ST25. If there is not, the multidimensional analysis process by the access log information multidimensional analysis system according to the present embodiment ends.
[0008]
However, in the above-described conventional technology, every time multi-dimensional analysis of network access log information is performed using the OLAP system, the OLAP database is constructed from the flat file format through the table format for the network access log information to be analyzed. Therefore, there is a problem in that it takes a lot of man-hours and time until the network administrator constructs an OLAP system environment for performing the analysis work.
[0009]
Here, there is an invention according to the following prior art document as a system for improving the above inconvenience and efficiently constructing a multidimensional analysis environment.
[0010]
[Patent Document 1]
Japanese Patent Laid-Open No. 11-316766
[Problems to be solved by the invention]
However, when performing multidimensional analysis of network access log information using the OLAP system, an OLAP database must be constructed from the flat file format through the table format for the network access log information to be analyzed. If the automation of the format conversion work from the flat file format to the table format is not realized, the efficiency of the multidimensional analysis processing of the access log information cannot be achieved.
[0012]
In addition, all network access log information must be stored for a certain period and must be stored for a long time (permanently). There was also a problem.
[0013]
The network access log information in the flat file format has a configuration in which each access log data is arranged in time series as shown in FIG. 6, and the contents of the network access log information are access date / time data, access source IP, access destination Although it consists of various items such as IP and status, the data appearing in each item often overlaps between the network access log information, and as a result, the network access log information in the flat file format is generally redundant. Data structure. If network access log information with such a data structure is stored for a long period of time, the number of duplicate data will increase, and the log information to be stored will become even more redundant, so a larger storage area must be secured. I must.
[0014]
Accordingly, an object of the present invention is to improve the above inconvenience, to quickly construct an analysis environment using an OLAP system of network access log information when necessary, and to reduce the capacity of a storage area for network access log information. An object of the present invention is to provide a multidimensional analysis system and a multidimensional analysis environment construction method capable of minimizing access log information.
[0015]
[Means for Solving the Problems]
In order to achieve the above object, the present invention provides an access comprising a plurality of network devices and an access log analyzer connected to the network devices via a network line and performing multidimensional analysis processing of log information of the network devices. A log information multidimensional analysis system,
The access log analyzer includes a non-analysis target item database for storing item data to be excluded from access log analysis targets, a relational database for storing tabular access log information, and the non-analysis target item database and the relational database. A processing device to control,
The processing device is
A transfer log information acquisition unit for acquiring access log information transferred from a plurality of network devices;
From the access log information acquired by the transfer log information acquisition unit, the actual data of each element of the access date and time and the access destination IP address included in the access log information, the master table ID corresponding to each of the actual data, and non-analysis A step of generating a master table storing numerical data indicating whether or not a target is stored, a master table ID corresponding to each element of an access date and time and an access destination IP address stored in the master table, and whether the target is a non-analysis target A data format conversion unit for executing a step of creating a data table in a table format for storing numerical data indicating
A storage unit for storing pre Kide Tateburu as the access log information table format in a relational database,
Wherein among the access log information stored in a relational database, analyte the numerical data extracts the access log information including numerical data indicating that it is not a non-analyte, to build an OLAP database is multi-dimensional analyzable analysis environment An extractor;
A multi-dimensional analysis processing unit that performs multi-dimensional analysis processing based on access log information in the OLAP database is a first feature.
[0016]
In addition, the present invention includes a plurality of network devices, a non-analysis target item database that stores item data that is connected to the network devices via a network line, and that is excluded from access log analysis targets, and table-type access log information. A method for constructing a multidimensional analysis environment for an access log information multidimensional analysis system, comprising: a relational database to be stored; and a processing device that controls each database, and an access log analysis device that performs multidimensional analysis processing of log information of network devices. Because
The processing device is
A transfer log information acquisition step of acquiring access log information transferred from a plurality of network devices;
From the access log information acquired in the transfer log information acquisition step, the actual data of each element of the access date and time and the access destination IP address included in the access log information, the master table ID corresponding to each of the actual data, and the non-analysis target Generating a master table storing numerical data indicating whether or not,
A data table creation step of creating a data table in a table format for storing the access date and time stored in the master table, the master table ID corresponding to each element of the access destination IP address, and the numerical data indicating whether or not the non-analysis target When,
A storing step of storing front Kide Tateburu a relational database as the access log information table format,
Wherein among the access log information stored in a relational database, analyte the numerical data extracts the access log information including numerical data indicating that it is not a non-analyte, to build an OLAP database is multi-dimensional analyzable analysis environment An extraction process;
A second feature is that a multidimensional analysis processing step of performing multidimensional analysis processing based on access log information in the OLAP database is performed.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an access log information multidimensional analysis system according to an embodiment of the present invention will be described in detail with reference to FIGS. FIG. 1 is a diagram illustrating a configuration of an access log information multidimensional analysis system according to the present embodiment, and FIG. 2 is a diagram illustrating a configuration of a non-analysis target item database in the access log information multidimensional analysis system according to the present embodiment. It is. 3 is a flowchart showing a multidimensional analysis processing procedure by the access log information multidimensional analysis system according to the present embodiment, and FIG. 4 shows a storage format in the access log information multidimensional analysis system according to the present embodiment. FIG. FIG. 5 is a flowchart showing the procedure of a data format conversion process performed by the access log information multidimensional analysis system according to this embodiment.
[0019]
[Configuration of Access Log Information Multidimensional Analysis System]
First, the configuration of the access log information multidimensional analysis system 1 according to the present embodiment will be described with reference to FIG. The access log information multidimensional analysis system 1 includes network devices 10 and 11 (in this embodiment, a proxy server 10 and a firewall server 11), an access log analysis device that constructs a multidimensional analysis environment and performs multidimensional analysis processing. 12 are connected via a network line 50. Although the number of network devices in this embodiment is two, the number of network devices is not limited to this. Hereinafter, each component will be described.
[0020]
Each network device 10, 11 has a function of periodically transferring network access log information 21 stored in each network device 10, 11 to the access log analyzer 12. A transfer program 31 is implemented.
[0021]
The access log analysis device 12 includes a processing device 13 that controls each processing, a non-analysis target item database 14 that stores data of items determined by the network manager to be excluded from the analysis target, and format conversion as described later. A relational database 15 for storing the tabular access log information 22 that has been recorded.
[0022]
The processing device 13 includes the transfer log information acquisition unit 13a that acquires the access log information 21 transferred from each network device 10 and 11, and the transfer log information based on the data in the non-analysis target item database 14. The data format conversion unit 13b that converts the access log information 21 acquired by the acquisition unit 13a from the flat file format to the table format, and the log information to be analyzed from the table format access log information 22 in the relational database 15 An analysis object extraction unit 13c that builds an OLAP database 23, which is an analysis environment that can be extracted and analyzed in a multidimensional manner, and a multidimensional analysis processing unit 13d that performs multidimensional analysis processing based on log information in the OLAP database 23. It has.
[0023]
Here, as shown in FIG. 2, the non-analysis target item database 14 stores the types of items in log information (date and time, access source IP, access destination IP, status, etc. in FIG. 6). “Data” item and “Non-analysis target content data” item in which the contents determined by the network administrator that the log information items need not be analyzed are stored. According to FIG. 2, the log information whose date and time is “March 29, 2002” and the log information whose access destination IP is “202.23.112.34” are excluded from the multidimensional analysis target, that is, many The network administrator determines in advance that the dimension analysis is not performed, and is registered and stored in the non-analysis target item database as non-analysis target log information in the format shown in FIG.
[0024]
[Multidimensional analysis processing procedure]
The procedure of the multidimensional analysis process in the access log information multidimensional analysis system 1 configured as described above will be described below with reference to FIG.
[0025]
First, the flat file format network access log information 21 stored and stored in the network devices 10 and 11 is periodically accessed via the network line 50 by the transfer program 31 installed in the network devices 10 and 11. When transferred to the analysis device 12, the transferred network access log information 21 is acquired by the transfer log information acquisition unit 13a of the access log analysis device 12 (step ST1), and further flattened by the data format conversion unit 13b. The file format is automatically converted into the table format access log information 22 (step ST2).
[0026]
Here, the processing in step ST2 will be described in detail with reference to FIGS. 4 and 5. In the data format conversion unit 13b, each element in the log information that is synchronized with the data in the flat file format as shown in FIG. A data table as items and a master table from which each item of log information is extracted are generated (step ST11).
[0027]
In each master table, an ID value is assigned to each actual data (step ST12). Each master table is provided with a “non-analysis target” item in addition to the “ID” item and the “element” item (step ST13). For data that matches the data already stored in the non-analysis target item database, Arbitrary numerical data (for example, “0”) is assigned to the “non-analysis target” item, and other numerical data (for example, “1”) is assigned to the “non-analysis target” item for other data. (Step ST14). Each item of the data table is given not the actual data but the ID value defined in the master table at the location corresponding to the actual data (step ST15). Specifically, among the data registered in the master table, the data “date and time is March 29, 2002” already registered in the table of FIG. 2 and the “access destination IP is 202.23.112.34”. For data, “0” is assigned to the “non-analysis target” item.
[0028]
In addition, “0” is assigned to the “non-analysis target” item of the data table in the same way as the master table for log information including at least one data item that matches the data stored in the non-analysis target item database. For log information that does not include any matching data item, “1” is assigned to the “non-analysis target” item (step ST16).
[0029]
The process of step ST2 is performed as described above, and by this process, table format access log information 22 as shown in FIG. 4 is created. This table format access log information 22 is stored in the relational database in the access log analyzer 12. 15 is stored and stored (step ST3).
[0030]
As described above, the access log information 21 is periodically acquired from each of the network devices 10 and 11 and converted into a table format. Then, the obtained table format access log information 22 is accumulated and stored in the access log analyzer 12. The processing from step ST1 to step ST3 is automatically performed, and when the network administrator who is the person in charge of analysis starts the multidimensional analysis processing, all access log information is stored in a table format as an access log analyzer. 12 is accumulated and stored.
[0031]
When the network administrator starts the multidimensional analysis process (step ST4), the network administrator selects log information to be used for analysis, and is used for analysis from the table format access log information 22 in the relational database. Only the log information is extracted by the analysis object extraction unit 13c (step ST5), and the OLAP database 23, which is an analysis environment capable of multidimensional analysis, is constructed (step ST6).
[0032]
Here, in the extraction work of step ST5, among the log information of the table format access log information 22, those for which “0” is given to the “non-analysis target” item are preliminarily excluded from the analysis target by the network administrator. Since it is determined, only log information in which “1” is assigned to the “non-analysis target” item is not extracted. As described above, in the extraction work in step ST5, the extraction log information filtering process for further narrowing down the analysis target log information selected by the network administrator is automatically performed.
[0033]
Through each processing step as described above, the network administrator can obtain the OLAP database 23 which is the final analysis environment, and then performs multidimensional analysis processing (step ST7).
[0034]
Here, when access log information other than the log information extracted by the analysis target extraction unit 13c is required as a result of the multidimensional analysis processing in step ST7 (step ST8), the network administrator performs the above step ST6. The process is performed again, and the OLAP database 23 may be recreated. As described above, according to the access log information multidimensional analysis system 1 according to the present embodiment, a new multidimensional analysis environment is provided again by performing the creation process of the OLAP database 23 in step ST6. Multidimensional analysis processing can be performed quickly without requiring a large number of man-hours for the construction of the environment.
[0035]
If multi-dimensional analysis processing is thus performed and no other access log information is required in step ST8, all steps of multi-dimensional analysis processing by the access log information multi-dimensional analysis system 1 according to the present embodiment are completed.
[0036]
As described above, according to the access log information multidimensional analysis system 1 according to the present embodiment, each conversion from the flat file format to the OLAP database 23 through the table format is performed for the network access log information 21 to be analyzed. It is possible to perform multidimensional analysis processing promptly without requiring a lot of man-hours and time for work.
[0037]
Further, in the conventional flat file format, the same data (the same character string) often appears in each item, and the data for each item is redundant. However, the access log information multidimensional analysis system 1 according to the present embodiment 1 Since log information is stored in the table format shown in FIG. 4, the data in the master table can be added and deleted without duplication. As a result, the network administrator can work more efficiently. Can be achieved.
[0038]
Furthermore, since the ID value (numerical value) defined in the corresponding master table is used in the data table instead of the actual data of each item, a flat file format in which all the actual data is stored in chronological order Compared to, the table format requires less resource consumption (hard disk consumption) when storing log information. Thus, the storage of network access log information can also minimize hard disk consumption and improve the system operating environment.
[0039]
In the access log information analysis device 12 of the above embodiment, instead of providing the data format conversion unit 13b, a program for performing the data format conversion processing operation and the analysis target extraction processing operation of the present invention is created, and the access log information The analysis apparatus 12 may be caused to execute a processing operation based on this program.
[0040]
【The invention's effect】
As described above, according to the access log information multidimensional analysis system of the present invention, it is possible to shorten the time required for the network administrator to start the multidimensional analysis work of a huge amount of network access log information. As a result, the work efficiency in the multidimensional analysis process can be improved. In addition, since the network access log information is stored in the table format for a long time, all redundant data as character strings generated in the flat file format can be omitted, and the resource consumption can be minimized. This makes it possible to improve the operating environment of the system.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of an access log information multidimensional analysis system according to an embodiment of the present invention.
FIG. 2 is a diagram showing a configuration of a non-analysis target item database in the access log information multidimensional analysis system according to the present embodiment.
FIG. 3 is a flowchart showing a multidimensional analysis processing procedure by the access log information multidimensional analysis system according to the present embodiment.
FIG. 4 is a diagram showing a storage format in the access log information multidimensional analysis system according to the present embodiment.
FIG. 5 is a flowchart showing a procedure of a data format conversion process performed by the access log information multidimensional analysis system according to the embodiment.
FIG. 6 is a diagram showing an example of network access log information data in a flat file format.
FIG. 7 is a diagram showing a conventional multidimensional analysis environment.
FIG. 8 is a flowchart showing a conventional multidimensional analysis processing procedure.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Access log information multidimensional analysis system 10, 11 ... Network equipment, 12 ... Access log analysis apparatus, 13 ... Processing apparatus, 13a ... Transfer log information acquisition part, 13b ... Data format conversion part, 13c ... Analysis object extraction part , 13d ... multidimensional analysis processing unit, 14 ... non-analysis target item database, 15 ... relational database, 21 ... network access log information (flat file format), 22 ... table format access log information, 23 ... OLAP database, 31 ... transfer program.

Claims (2)

An access log information multidimensional analysis system comprising a plurality of network devices and an access log analyzer connected to the network devices via a network line and performing multidimensional analysis processing of log information of the network devices,
The access log analyzer includes a non-analysis target item database for storing item data to be excluded from access log analysis targets, a relational database for storing tabular access log information, and the non-analysis target item database and the relational database. A processing device to control,
The processing device is
A transfer log information acquisition unit for acquiring access log information transferred from a plurality of network devices;
From the access log information acquired by the transfer log information acquisition unit, the actual data of each element of the access date and time and the access destination IP address included in the access log information, the master table ID corresponding to each of the actual data, and non-analysis A step of generating a master table storing numerical data indicating whether or not a target is stored, a master table ID corresponding to each element of an access date and time and an access destination IP address stored in the master table, and whether the target is a non-analysis target A data format conversion unit for executing a step of creating a data table in a table format for storing numerical data indicating
A storage unit for storing pre Kide Tateburu as the access log information table format in a relational database,
Wherein among the access log information stored in a relational database, analyte the numerical data extracts the access log information including numerical data indicating that it is not a non-analyte, to build an OLAP database is multi-dimensional analyzable analysis environment An extractor;
An access log information multidimensional analysis system comprising: a multidimensional analysis processing unit that performs multidimensional analysis processing based on access log information in the OLAP database. A plurality of network devices, a non-analysis target item database for storing item data to be excluded from an access log analysis target, a relational database for storing tabular access log information, A multi-dimensional analysis environment construction method for an access log information multi-dimensional analysis system comprising a processing device that controls each database, and an access log analysis device that performs multi-dimensional analysis processing of log information of network devices,
The processing device is
A transfer log information acquisition step of acquiring access log information transferred from a plurality of network devices;
From the access log information acquired in the transfer log information acquisition step, the actual data of each element of the access date and time and the access destination IP address included in the access log information, the master table ID corresponding to each of the actual data, and the non-analysis target Generating a master table storing numerical data indicating whether or not,
A data table creation step of creating a data table in a table format for storing the access date and time stored in the master table, the master table ID corresponding to each element of the access destination IP address, and the numerical data indicating whether or not the non-analysis target When,
A storing step of storing front Kide Tateburu a relational database as the access log information table format,
Wherein among the access log information stored in a relational database, analyte the numerical data extracts the access log information including numerical data indicating that it is not a non-analyte, to build an OLAP database is multi-dimensional analyzable analysis environment An extraction process;
A multidimensional analysis environment construction method for an access log information multidimensional analysis system, comprising: performing a multidimensional analysis processing step of performing multidimensional analysis processing based on access log information in the OLAP database.

Images