JP7823745B2 - Data generation device, data generation method, and program - Google Patents

Description

One aspect of the present invention relates to a data generation device, a data generation method, and a program for generating training data for a machine learning model that detects abnormalities such as failures.

Systems are developed with the ultimate goal of being applied to a target production environment. However, before the product can be released, verification work in an environment that mimics the production environment, such as a testing or staging environment, is essential. The situation is the same in AI (Artificial Intelligence) development, which is increasingly being discussed in conjunction with MLOps (Machine Learning Operations).

In recent years, attempts have been made to operate networks using AI models trained on learning data related to network operation. Training an AI model requires providing large amounts of both data on the network's normal operating state (normal state data) and data on when a failure occurs (failure data) to the untrained model. However, reproducing a failure in a production environment involves significant risks, making it difficult to collect such data in large quantities. Therefore, a technology has been proposed in which a mirror environment of the production environment is constructed and both the normal and abnormal data required for machine learning are generated in the mirror environment (see non-patent document 1).

D. Li, K. Akashi, H. Nozue and K. Tayama, "A Mirror Environment to Produce Artificial Intelligence Training Data," in IEEE Access, vol. 10, pp. 24578-24586, 2022, doi: 10.1109/ACCESS.2022.3154825. L. Han, F. Gao, Z. Li and O. A. Dobre, "Low Complexity Automatic Modulation Classification Based on Order-Statistics," in IEEE Transactions on Wireless Communications, vol. 16, no. 1, pp. 400-411, Jan. 2017, doi: 10.1109/TWC.2016.2623716.

Log data collected from a mirror environment that mimics the target environment and log data from the target environment are often text-based. By comparing the log data obtained in each environment and clustering or rewriting alarms with the same meaning, events for the target environment can be generated from the learned events created in the mirror environment. Attribute information, including response methods, can also be used as is.

However, the types and number of network logs collected from the target environment are enormous, and as the number of words increases, the clustering processing and learning time also increases. Furthermore, as the operation period progresses, the amount of logs collected from the target network increases unilaterally. Ultimately, the clustering processing time between the target environment and the mirror environment cumulatively slows down. If this is left unaddressed, log analysis will take an extremely long time, and a solution is desired.

This invention was made with the above circumstances in mind and aims to provide technology that can shorten the time required for log analysis.

According to one aspect of the present invention, a data generation device includes a memory unit that stores text-based log data collected from a network of a target environment, and a processor. The processor includes a unique extraction processing unit, a word list generation unit, a word vector generation unit, a clustering processing unit, and a data generation unit. The unique extraction processing unit extracts unique logs from the log data to generate extracted log data. The word list generation unit divides the extracted log data into multiple words using morphological analysis to generate a word list. The word vector generation unit vectorizes the word list to generate multiple word vectors in a multi-dimensional vector space. The clustering processing unit clusters the multiple word vectors based on a distance index between the multiple word vectors in the vector space. The data generation unit generates events for the target environment from learned events created in a mirror environment of the target environment based on the clustering results.

One aspect of this invention provides technology that can reduce the time required for log analysis.

FIG. 1 is a diagram showing an example of a system to which a data generating device according to an embodiment of the present invention is applied. FIG. 2 is a block diagram showing an example of the data generating device 3 shown in FIG. FIG. 3 is a functional block diagram showing an example of the data generating device 3 shown in FIG. FIG. 4 is a flowchart showing an example of a clustering processing procedure according to the embodiment. FIG. 5 is a flowchart showing an example of a processing procedure in the cleansing process. FIG. 6 is a flowchart illustrating an example of a processing procedure in the cluster determination process. FIG. 7 is a flowchart showing an example of the processing procedure for "File Reading and Preprocessing (Step S72 in FIG. 6)." FIG. 8 is a flowchart showing an example of the processing procedure of the clustering determination process (step S74 in FIG. 6). FIG. 9 is a diagram showing the results of clustering in relation to whether cleansing has been performed. FIG. 10 is a graph showing an example of processing time in the case of only the unique extraction process. FIG. 11 is a graph showing an example of the processing time required for clustering including the unique extraction process and the clustering process. FIG. 12 is a diagram showing another example of the result of clustering performed without cleansing. FIG. 13 is a flowchart showing an example of a procedure for an existing hierarchical clustering process.

The following describes an embodiment of the present invention with reference to the drawings.

[One embodiment]
<Configuration>
Fig. 1 is a diagram showing an example of a system to which a data generation device according to an embodiment of the present invention is applied. The system shown in Fig. 1 is a system that operates a target system 1, such as a network system, by using an AI model 2. The system further includes a data generation device 3. The data generation device 3 generates training data for training the AI model 2.

Figure 2 is a block diagram showing an example of the data generation device 3 shown in Figure 1. The data generation device 3 is a computer comprising a processor 10, memory 20, storage 30, an input/output interface (I/F) 40, and a bus 45 interconnecting these components. The input/output I/F 40 establishes communication links between the data generation device 3 and the target system 1 and AI model 2, and transmits and receives various types of data. The processor 10 is a computing device such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), and realizes the processing functions of the embodiment in accordance with a program loaded from the storage 30 to the memory 20.

Figure 3 is a functional block diagram showing an example of the data generation device 3 shown in Figure 2. In Figure 3, the memory 20 is a semiconductor memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory).

Storage 30 is a non-volatile memory such as a hard disk drive (HDD) or solid state drive (SSD), and stores basic software such as an operating system (OS), as well as a program 30g for implementing the processing according to the embodiment. In other words, program 30g can be installed in the data generation device 3.

The storage 30 also stores log data 30a, extracted log data 30b, word list 30c, regenerated word list 30d, word vector 30e, and clustering results 30f. Of these, the log data 30a is text-based log data collected from the network of the target environment.

The processor 50 has, as processing functions according to one embodiment of the present invention, a unique extraction processing unit 50a, a word list generation unit 50b, a cleansing processing unit 50c, a word vector generation unit 50d, a clustering processing unit 50e, and a data generation unit 50f.

The unique extraction processing unit 50a, word list generation unit 50b, cleansing processing unit 50c, word vector generation unit 50d, clustering processing unit 50e, and data generation unit 50f are realized by processor 10 executing a program loaded into memory 20. In other words, program 30g includes instructions that cause processor 50 to function as unique extraction processing unit 50a, as word list generation unit 50b, as cleansing processing unit 50c, as word vector generation unit 50d, as clustering processing unit 50e, and as data generation unit 50f.

The unique extraction processing unit 50a extracts unique logs from the log data and generates extracted log data. The extracted log data is sent to the storage 30 and stored as extracted log data 30b.

The word list generation unit 50b generates a word list by dividing the extracted log data 30b into multiple words, for example by morphological analysis. The word list is sent to the storage 30 and stored as a word list 30c.

The cleansing processing unit 50c performs a cleansing process to remove unnecessary words from the word list 30c and generate a regenerated word list. The regenerated word list is sent to the storage 30 and stored as the regenerated word list 30d.

The word vector generation unit 50d vectorizes the regenerated word list 30d to generate multiple word vectors in a multi-dimensional vector space. The word vectors are sent to the storage 30 and stored as word vectors 30e.

The clustering processing unit clusters multiple word vectors based on the distance index between the multiple word vectors in vector space. The clustering results are sent to storage 30 and stored as clustering results 30f.

The data generation unit 50f generates events for the target environment from learned events created in the mirror environment of the target environment based on the clustering results 30f.

<Effect>
Next, the operation of the above configuration will be described.
4 is a flowchart showing an example of a clustering process according to an embodiment. In FIG. 4, the processor 50 reads the log data 30a into the memory 20 (step S1) and preprocesses the data (step S2). The preprocessing in step S2 may be general processing such as shaping and extraction for utilizing text data.

Next, the processor 50 performs a unique extraction process, for example by deleting duplicate lines on a line-by-line basis (step S3), to generate post-extraction log data 30b. Next, the processor 50 performs a word segmentation process on the post-extraction log data 30b using morphological analysis (step S4). This generates a word list 30c.

Next, the processor 50 performs a cleansing process (step S5) to remove words that have no meaning for clustering from the divided word list (word list 30c). For example, words to be removed include numeric numbers for identifying network devices, or serial numbers or numeric numbers for simple counting. The processor 50 removes such words that reflect characteristics unique to the network from the word list 30c to generate a regenerated word list 30d.

Furthermore, the processor 50 generates word vectors 30e from the regenerated word list 30d through vectorization processing (step S6). Finally, the processor 50 calculates distance indices between vectors in vector space from the word vectors 30e using a method such as the shortest distance method, the longest distance method, or the group average method.

Clustering is a method of grouping data based on the similarity (how similar) between the data. Clustering methods can be broadly divided into hierarchical methods and partition optimization methods.

When clustering network logs, one method obtains a hierarchical structure by iteratively merging two clusters that are close to each other based on a distance function between clusters. On the other hand, there are also non-hierarchical methods that group data without creating a hierarchy.

Distance indices in hierarchical clustering can be calculated using the shortest distance method, the longest distance method, or the group average method. These indices can be calculated using Euclidean distance or cosine similarity, and clustering can be performed by discriminating these values using a predetermined threshold.

For example, the Euclidean distance between n-dimensional vectors (x, y) is defined by equation (1).

Euclidean distance is the most common distance measure between data in clustering.

Figure 5 is a flowchart showing an example of the processing steps in the cleansing process (step S5 in Figure 4). In Figure 5, the processor 50 generates a word list 30c (step S51) and then performs a data cleansing process on the word list (step S52). Furthermore, the processor 50 regenerates the word list from the remaining words to create a regenerated word list 30d (step S53).

Figure 6 is a flowchart showing an example of the processing procedure for the cluster determination process (step S7 in Figure 4). In Figure 6, the processor 50 reads the cluster history (step S71), and performs file reading and preprocessing (step S72). Next, the processor 50 recalculates the center coordinates of the cluster (step S73) and performs clustering determination processing (step S74). The processor 50 then outputs the cluster history obtained by the clustering determination processing (step S75).

Figure 7 is a flowchart showing an example of the processing procedure for [File Reading and Preprocessing (Step S72 in Figure 6)]. In Figure 7, the processor 50 reads the input file (Step S21) and performs preprocessing (Step S22). Next, the processor 50 combines the cluster history obtained in the procedure in Figure 6 with the input file to create a word list (Step S23), and creates one-hot data from the word list (Step S24).

Figure 8 is a flowchart showing an example of the processing procedure for the clustering determination process (step S74 in Figure 6). In Figure 8, the processor 50 reads the one-hot data line by line (step S41) and checks whether or not there is data (step S42). If there is no data (No), the process ends.

If the data can be read (Yes in step S42), the processor 50 calculates the Euclidean distance from the existing cluster (step S43) and determines whether there is a cluster within the threshold (step S44). If the Euclidean distance is less than the threshold (Yes in step S44), the processor 50 adds the word vector to the nearest cluster (step S45). If the Euclidean distance is greater than the threshold (No in step S44), the processor 50 registers the word vector in a new cluster (step S46). After step S45 or step S46, the processor updates the center coordinates of the cluster (step S47), and the processing procedure returns to step S41.

Figure 9 shows the clustering results with and without cleansing. In Figure 9, for example, seven log data files were integrated as sample data, providing 850,000 lines of log data. Data corresponding to messages was extracted from this and used for clustering. When processed on a PC server with the same specifications, the case without cleansing took 13 hours and 15 minutes. In contrast, the case with cleansing completed processing in just 35 minutes. In this case, the number of words was reduced by 45% compared to the case without cleansing, and the execution time was reduced by 96%.

Figure 10 is a graph showing an example of processing time when only unique extraction processing is performed. The horizontal axis is in seconds. The results of the distance calculation to the cluster roughly increase to the right.

Figure 11 is a graph showing an example of the processing time required for clustering, including unique extraction processing and clustering processing. The horizontal axis is in seconds. The update of the results of distance calculations between clusters increases exponentially. This indicates that the update process of the results of distance calculations between clusters is a bottleneck. For example, the graph shows that it took 60,000 seconds (16 hours) to process approximately 10,000 rows.

This process calculates the Euclidean distance between all clusters and new records (rows) and adds the results to a list, so the list becomes larger as the number of clusters increases. Furthermore, since the sample data before extracting unique logs contains approximately 850,000 rows, processing takes several days.

Figure 12 shows another example of the results of clustering performed without cleansing. The CPU specifications were slightly lower than those in Figure 9. It can be seen that the execution time was over 35 hours.

Figure 13 is a flowchart showing an example of an existing hierarchical clustering processing procedure. Unlike the flowchart in Figure 4, existing technology does not perform the unique extraction process (step S3) or the cleansing process (step S5). Therefore, the more log data collected from the network to be managed, the longer the processing time required to cluster the logs becomes.

<Effects>
As described above, in the embodiment, data that does not affect the clustering process is deleted from the log data in advance to reduce the clustering process time. Here, examples of data to be deleted include duplicate lines in the log and IDs for identifying devices and information.

In other words, in this embodiment, before processing such as clustering and learning is performed on log data collected from the target network, unnecessary data is not provided to subsequent processing (cluster determination processing and learning processing). In other words, unique extraction of words that affect the message classification judgment results and duplicate data that takes up calculation time, and data cleansing are performed. This significantly reduces duplicate data such as numbers that simply identify equipment, the number of pieces of equipment, and numbers that increase over time, thereby reducing the number of data to be processed. Because the number of data to be processed is reduced, the overall learning time can also be significantly reduced.

The amount of collected log data increases over time. As the amount of data increases, subsequent processing and learning times also increase. The collected log data contains data that is meaningless in subsequent processing, such as large amounts of numbers and duplicated text. Pre-processing removes this data, reducing the amount of data to be processed and significantly shortening and speeding up processing times.

In particular, when sequentially processing large amounts of text data, the processing time increases as the amount of target data increases. By removing duplicate data and unnecessary feature data, only data with the necessary features can be processed. This can significantly reduce processing time and promote faster processing.

To cluster network logs, the logs are divided into words using morphological analysis to create a word list. One possible approach is to convert the word list into vectors using a method such as one-hot, and then calculate the Euclidean distance for each vector to separate them into clusters. The processing time required to calculate the distance increases as the number of words resulting from the morphological analysis increases.

Network logs are automatically generated by each device based on configured specifications. As a result, there are many duplicate logs. Therefore, clustering all logs, including duplicates, would waste computer resources and processing time.

Therefore, in this embodiment, unique logs are extracted and data cleansing is performed to reduce the amount of data provided to subsequent stages as much as possible, thereby promoting speedup.

Logs collected from a network contain numerous numerical numbers used simply for differentiation, such as numbers to identify devices that make up the network and Syslog record numbers output by the OS. Generating clusters using these numerical numbers as clustering targets would be useless for clustering message texts and would require a considerable amount of processing time. As explained in the embodiments, by preprocessing the word list to cleanse words before clustering, the amount of data to be clustered can be significantly reduced and processing time can be accelerated.

As a result, according to the embodiment, it is possible to reduce the time required for log analysis, which in turn reduces the time required to train AI models related to network operation, thereby speeding up total processing time. Furthermore, in network management work, by collecting alarm information from the network to be managed and comparing and analyzing it with learned alarm information, it is possible to improve the efficiency of work to analyze the status of the managed network.

Note that this invention is not limited to the above-described embodiments. For example, in FIG. 1, target system 1 may be, for example, a server device, communications equipment, office equipment, medical equipment, in-vehicle equipment, various types of residential equipment, or home appliances. Furthermore, AI model 2 and data generation device may be included in target system 1, or may be connected to each other via a communications line. The data generation device may be included in AI model 2, or may be connected to each other via a communications line.

In addition, any type of AI model can be applied, such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), or RNN (Recurrent Neural Network).

Furthermore, even if the cleansing process (step S5) in the flowchart of Figure 4 is omitted, the objective can be achieved by executing the unique extraction process (step S3), which reduces the amount of data required for clustering and shortens the processing time. In this configuration, the word vector generation unit 50d vectorizes the word list 30c generated by the word list generation unit 50b to generate multiple word vectors 30e.

In short, this invention is not limited to the above-described embodiments, and can be embodied by modifying the components in the implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all of the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

1...Target system 2...AI model 3...Data generation device 10...Processor 20...Memory 30...Storage 30a...Log data 30b...Extracted log data 30c...Word list 30d...Regenerated word list 30e...Word vector 30f...Clustering result 30g...Program 40...Input/output interface 45...Bus 50...Processor 50a...Unique extraction processing unit 50b...Word list generation unit 50c...Cleansing processing unit 50d...Word vector generation unit 50e...Clustering processing unit 50f...Data generation unit.

Claims (6)

a storage unit for storing text-based log data collected from the network of the target environment;
a processor;
The processor:
a unique extraction processing unit that extracts a unique log from the log data by deleting at least duplicated lines in the log data and IDs for identifying devices and information, and generates extracted log data;
a word list generation unit that divides the extracted log data into a plurality of words by morphological analysis to generate a word list;
a word vector generation unit that vectorizes the word list to generate a plurality of word vectors in a multi-dimensional vector space;
a clustering processing unit that clusters the plurality of word vectors based on a distance index between the plurality of word vectors in the vector space;
a data generation unit that generates events for the target environment from learned events created in a mirror environment of the target environment based on the results of the clustering. The processor:
a cleansing processing unit that removes unnecessary words from the word list by a cleansing process to generate a regenerated word list;
The data generating device according to claim 1 , wherein the word vector generating unit generates the word vectors by vectorizing the regenerated word list. The data generation device described in claim 2, wherein the cleansing processing unit generates the regenerated word list by removing words that reflect characteristics unique to the network from the word list. The data generation device of claim 1, wherein the distance metric is Euclidean distance in the vector space. 1. A data generation method by a computer including a memory unit that stores text-based log data collected from a network of a target environment, and a processor, the method comprising:
a step in which the processor extracts, from the log data , a unique log in which at least duplicate lines in the log data and IDs for identifying devices and information have been deleted, and generates extracted log data;
a step of the processor dividing the extracted log data into a plurality of words by morphological analysis to generate a word list;
the processor vectorizing the word list to generate a plurality of word vectors in a multi-dimensional vector space;
the processor clustering the plurality of word vectors based on a distance metric between the plurality of word vectors in the vector space;
and generating events for the target environment from learned events created in a mirror environment of the target environment based on the results of the clustering by the processor. A program installable on a computer having a processor and a storage unit that stores text-based log data collected from a network of a target environment, the program comprising:
The processor,
an instruction to function as a unique extraction processing unit that extracts a unique log from the log data by deleting at least duplicate lines in the log data and IDs for identifying devices and information, and generates extracted log data;
an instruction to function as a word list generation unit that divides the extracted log data into a plurality of words by morphological analysis and generates a word list;
an instruction to function as a word vector generator that vectorizes the word list and generates a plurality of word vectors in a multi-dimensional vector space;
an instruction to function as a clustering processor that clusters the plurality of word vectors based on a distance index between the plurality of word vectors in the vector space;
and instructions to cause the program to function as a data generation unit that generates events for the target environment from learned events created in a mirror environment of the target environment based on the results of the clustering.