JP7694702B2 - Fault information estimation device, fault information estimation method, and fault information estimation program - Google Patents

Description

The present invention relates to a fault information estimation device, a fault information estimation method, and a fault information estimation program.

In service maintenance work, when a service failure occurs, data is collected and analyzed from a large number of monitored targets (devices, applications, etc.) within the monitored system to estimate failure information such as the status and cause of the failure in the monitored target.

Zhang, Chuxu et al. “A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data.” ArXiv abs/1811.08055 (2019).

In monitored systems, it is desirable to estimate fault information efficiently and quickly in order to minimize the adverse impact on users after a fault occurs.

The present invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a fault information estimation device, a fault information estimation method, and a fault information estimation program that efficiently and quickly estimate fault information of a monitored object in which a fault has occurred.

One aspect of the present invention is a fault information estimation device, which includes a data acquisition unit that acquires data having time series data and metadata of multiple metrics of multiple monitoring targets in a monitored system, a pruning unit that extracts metrics data related to a fault from the multiple metrics data, a fault information estimation unit that estimates fault information of a monitoring target in which a fault has occurred based on the data extracted by the pruning unit , and a time series data encoder that converts a timestamp representing an absolute time of the time series data extracted by the pruning unit into a timestamp representing a relative time within a time window .

One aspect of the present invention is a fault information estimation method, which includes acquiring data having time series data and metadata of multiple metrics of multiple monitored targets in a monitored system, extracting metrics data related to a fault from the multiple metrics data, estimating fault information of a monitored target in which a fault has occurred based on the metrics data related to the fault, and converting a timestamp representing an absolute time of the time series data of the metrics related to the fault into a timestamp representing a relative time within a time window .

One aspect of the present invention is a fault information estimation program. The fault information estimation program causes a computer to execute the functions of each component of the fault information estimation device described above.

According to the present invention, there is provided a fault information estimation device, a fault information estimation method, and a fault information estimation program that efficiently and quickly estimate fault information of a monitored object in which a fault has occurred.

FIG. 1 is a block diagram illustrating an example of a functional configuration of a fault information estimation device according to an embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the fault information estimation device according to the embodiment. FIG. 3 is a diagram illustrating a flow of a process of estimating fault information executed by the fault information estimation device according to the embodiment. FIG. 4 is a diagram illustrating an example of an anomaly detected by one-dimensional time series anomaly detection. FIG. 5 is a diagram showing an example of time stamp conversion. FIG. 6 is a diagram illustrating an example of pruning of metrics data. FIG. 7 is a diagram illustrating an example of an abnormal termination of an application. FIG. 8 is a diagram illustrating an example of a scale-out of a container. FIG. 9 is a diagram illustrating an example of input and output in the fault information estimation device according to the embodiment. FIG. 10 is a flowchart showing the procedure and contents of a fault information estimation process executed by the fault information estimation device according to the embodiment.

Below, an embodiment of the present invention is described with reference to the drawings.

[Configuration example]
(Functional configuration)
First, the functional configuration of a fault information estimation device according to an embodiment will be described. Fig. 1 is a block diagram showing an example of the functional configuration of a fault information estimation device 30 according to an embodiment. In addition to the fault information estimation device 30, Fig. 1 also shows a node 10 in a monitored system and a monitoring system 20. Although many nodes 10 exist in the monitored system, Fig. 1 shows only one representative node 10 for convenience.

As shown in Fig. 1, each node 10 has an application 11, a monitoring agent 12, and a data recording unit 13. The monitoring agent 12 is placed on the same node 10 and collects time series data and metadata of monitoring items related to the application 11, and records this in the data recording unit 13. The monitoring agent 12 also transmits the time series data and metadata recorded in the data recording unit 13 to the monitoring system 20 by polling/telemetry.

The monitoring system 20 collects metrics data for each monitored object from multiple nodes 10 in the monitored system. Hereinafter, the metrics data will be referred to as metrics data for convenience.

The fault information estimation device 30 is a device that acquires multiple metrics data for multiple monitored targets from the monitoring system 20, estimates fault information, and outputs a fault report.

The fault information estimation device 30 has a data acquisition unit 31, a pruning unit 33, a time series data encoder 34, a metadata encoder 35, a fault information estimation unit 36, and a fault report output unit 37.

The data acquisition unit 31 acquires multiple metrics data for multiple monitored targets from the monitoring system 20. Each metric data has time series data and metadata. Each time series data consists of a timestamp at each time and a collection of other data values. Each metadata consists of text information such as the name assigned to the metric, variable names, container names, etc.

The pruning unit 33 extracts (prunes) only the metrics data related to the fault from the multiple metrics data acquired by the data acquisition unit 31. For example, the pruning unit 33 extracts several tens of metrics data from several thousand metrics data. This reduces the metrics data used to estimate fault information. The metrics data related to the fault is time series data with abnormal fluctuations during a time window and the corresponding metadata. The metrics data is extracted, for example, by calculating an anomaly score for the time series data using a one-dimensional time series anomaly detection model. For one-dimensional time series anomaly detection, methods such as the Spectral Residual (SR) method and a Fourier transform-based anomaly detection method can be used. The pruning unit 33 supplies the extracted metrics data to the time series data encoder 34 and the metadata encoder 35.

The time series data encoder 34 simultaneously encodes the timestamps and data values of the time series data. The encoding includes converting the timestamps of the time series data. The timestamp conversion converts a timestamp representing an absolute time into a timestamp representing a relative time within a time window. Furthermore, the time series data encoder 34 calculates a vector representation from the timestamp representing the relative time and other data values for each metric, and aggregates these. This allows asynchronous time series data to be handled in a unified manner. The time series data encoder 34 supplies the encoding results to the metadata encoder 35.

The metadata encoder 35 simultaneously learns the time series data supplied from the time series data encoder 34 and the metadata supplied from the pruning unit 33 for each metric. This allows the meaning of the time series data to be understood from the text information of the metadata. It also allows the relationships between the time series data to be understood. The metadata encoder 35 supplies the encoding results to the fault information estimation unit 36.

The fault information estimation unit 36 estimates fault information such as the status and cause of the fault in the monitored object in which a fault has occurred, based on the encoding result of the time-series data encoder 34 and the encoding result of the metadata encoder 35. The fault information estimation unit 36 also creates a fault report based on the estimation result, and supplies this to the fault report output unit 37.

The fault report output unit 37 receives a fault report from the fault information estimation unit 36 and outputs it.

(Hardware configuration)
Next, a description will be given of a hardware configuration of the fault information estimation device 30. The fault information estimation device 30 is configured by a computer. For example, the fault information estimation device 30 is configured by a personal computer, a server computer, or the like.

2 is a block diagram showing an example of a hardware configuration of the fault information estimation device 30 according to an embodiment. As shown in FIG. 2, the fault information estimation device 30 has an input device 41, a CPU 42, a storage device 45, and an output device 48. In addition to these, the fault information estimation device 30 may further have other peripheral devices.

The input device 41, CPU 42, memory device 45, and output device 48 are electrically connected to each other via a bus 49, and exchange data and commands via the bus 49.

The input device 41 is a device that receives data from the monitoring system 20. For example, the input device 41 is configured as a receiving device. The input device 41 is not limited to this and may be configured as any other input device.

The output device 48 is a device that outputs a fault report. For example, the output device 48 is configured as a display or a transmission device. The output device 48 is not limited to this and may be configured as any other output device.

The storage device 45 stores programs and data necessary for the processing executed by the CPU 42. The CPU 42 reads out the necessary programs and data from the storage device 45 and executes them to perform various types of processing.

The storage device 45 has a main storage device 46 and an auxiliary storage device 47. The main storage device 46 and the auxiliary storage device 47 exchange programs and data between each other.

The main memory device 46 stores programs and data temporarily required for processing by the CPU 42. For example, the main memory device 46 is composed of a volatile memory such as a RAM (Random Access Memory).

The auxiliary storage device 47 stores programs and data supplied via an external device or a network, and provides the programs and data temporarily required for processing by the CPU 42 to the main storage device 46. For example, the auxiliary storage device 47 is composed of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD).

The CPU 42 is a processor and is hardware that processes data and instructions. The CPU 42 has a control device 43 and an arithmetic unit 44.

The control device 43 controls the input device 41, the calculation device 44, the memory device 45 and the output device 48.

The computing device 44 reads the program and data from the main memory device 46, executes the program to process the data, and provides the processed data to the main memory device 46.

In such a hardware configuration, the input device 41 constitutes the data acquisition unit 31. The CPU 42 and the storage device 45 constitute the pruning unit 33, the time series data encoder 34, the metadata encoder 35, and the fault information estimation unit 36. The output device 48 constitutes the fault report output unit 37.

For example, the CPU 42 loads a program that executes the functions of the pruning unit 33, the time series data encoder 34, the metadata encoder 35, and the fault information estimation unit 36 from the auxiliary storage device 47 into the main storage device 46, and executes the loaded program to perform the operations of the pruning unit 33, the time series data encoder 34, the metadata encoder 35, and the fault information estimation unit 36.

[Example of operation]
(Fault information estimation process)
Next, a flow of a process of estimating fault information executed by the fault information estimation device 30 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a schematic flow of a process of estimating fault information executed by the fault information estimation device 30.

In the input layer, the data acquisition unit 31 acquires multiple metrics data. Each metric data has time series data and metadata.

In the pruning layer, the pruning unit 33 applies one-dimensional time series anomaly detection to the time series data (td1) using the Spectral Residual (p1) to calculate an anomaly score (td2).

Figure 4 is a schematic diagram showing an example of an anomaly detection by one-dimensional time series anomaly detection. The left side of Figure 4 shows time series data, which is input data. Time series data is a collection of time stamps and data values at each time. The center of Figure 4 shows a graph of the time series data obtained for this input data. This graph includes point a1, which has a particularly high value compared to other points due to the occurrence of a fault. The right side of Figure 4 shows the anomaly score obtained by processing the graph in the center of Figure 4. This anomaly score includes anomaly point a2, which has a particularly high value compared to other points, which have a value of almost 0 due to the occurrence of a fault.

Next, the pruning unit 33 extracts time series data (td3), anomaly scores (td4), and metadata (md2) related to the fault based on the anomaly scores (td2) by pruning (p3). Pruning (p3) compares the anomaly scores with a predetermined threshold value to determine the presence or absence of anomalies, and extracts the anomaly scores (td4) including the anomalies, and the corresponding time series data (td3) and metadata (md2).

Next, in the encoding layer shown in FIG. 3, the time series data encoder 34 uses Transformer (p4) or other models to simultaneously encode the timestamps and data values of the time series data (td3, td4). In this encoding, timestamps representing absolute times are converted to timestamps representing relative times within a time window. This allows asynchronous time series data to be handled in a unified manner. Furthermore, for each metric, a vector representation is calculated from the timestamps representing relative times and other data values, and these are aggregated.

Figure 5 is a diagram showing an example of timestamp conversion. The left side of Figure 5 shows the time series data before the timestamp conversion, and the right side of Figure 5 shows the time series data after the timestamp conversion.

The timestamp of the converted time series data is the timestamp of the time series data before conversion minus a certain timestamp (1628143990). For example, the converted timestamp of the first row is 1628142121-1628143990=-1866.

In addition, in the encoding layer shown in Fig. 3, the metadata encoder 35 simultaneously learns the time-series data and the metadata (md2) using a Transformer (p3) or other model. As a result, the encoding result (d1) by the time-series data encoder 34 and the metadata encoder 35 is obtained.

The series of processes described above is performed for each metric. This series of processes is shown in Figure 3 by a dashed box. If the number of metrics is M, then this series of processes is repeated M times.

Figure 6 is a schematic diagram showing an example of pruning of metrics data by the pruning unit 33. The left side of Figure 6 shows the metrics data before pruning, and the right side of Figure 6 shows the metrics data after pruning. The metrics data before pruning on the left side of Figure 6 is shown together with a time series graph and anomaly score obtained by one-dimensional time series anomaly detection described with reference to Figure 4.

As can be seen from Figure 6, the metrics data after pruning consists of time series data corresponding to anomaly scores with abnormal values and metadata corresponding to that time series data. In addition, the time series data after pruning consists of the time series data before pruning and the anomaly scores.

Generally, the number of metrics monitored within a monitored system is huge. Furthermore, the data for these metrics contains a large amount of time-series data that is not related to failures. This increases the time required for analysis work to estimate failure information.

In an embodiment, in the pruning layer, metrics data related to failures is extracted from the multiple metrics data acquired in the input layer. This reduces the amount of metrics data used in the analysis work to estimate failure information. This contributes to shortening the time required for the analysis work to estimate failure information.

Time series data for metrics of monitored targets distributed within a monitored system is collected asynchronously, so the timestamps do not match. This results in missing values when aggregating the time series data into a matrix format. In such cases, preprocessing of the missing values is required, for example, by interpolating the missing values or correcting the data. This increases the effort and cost required for analysis work to estimate fault information.

The time series data encoder 34 converts timestamps that represent absolute times into timestamps that represent relative times, and calculates vector expressions from the timestamps that represent relative times and other data values, and aggregates them. This allows asynchronous time series data to be handled in a unified manner without processing missing values. This makes it possible to capture relationships between asynchronous metrics.

The number and type of metrics monitored within a monitored system may change dynamically. Metrics changes can occur due to an abnormal termination of an application or the scale-out of a container. Figure 7 shows a schematic diagram of an example of an abnormal termination of an application. Figure 8 shows a schematic diagram of an example of a scale-out of a container. When metrics change, the meaning of the time series data cannot be understood without metric metadata.

The metadata encoder 35 learns the time series data and metadata simultaneously. This allows the meaning of the time series data to be understood from the text information of the metadata. It also allows the encoder to understand the relationships between time series data. This allows the encoder to respond to dynamic changes in the number and type of metrics. In other words, even if the number and type of metrics change, the correspondence before and after the change can be understood.

3, the fault information estimation unit 36 uses a Transformer (p5) or other model to estimate fault information (d2) such as the fault status and cause based on the encoding result (d1). Then, the fault information estimation unit 36 uses a Fault Report Decorder (p6) or other model to create a fault report (d3) based on the fault information (d2).

Next, in the output layer, the fault report output unit 37 outputs the fault report.

Figure 9 is a diagram showing an example of input and output in the fault information estimation device 30 according to the embodiment. An example of metrics data, i.e., time series data and metadata, which are the input of Figure 9, is shown, and the right side of Figure 9 shows an example of a fault report, which is the output.

(flowchart)
Next, a process procedure and process contents of the fault information estimation process executed by the fault information estimation device 30 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing a process procedure and process contents of the fault information estimation process executed by the fault information estimation device 30 according to the embodiment.

In step S1, the data acquisition unit 31 acquires multiple metrics data, i.e., time series data and metadata, from the monitoring system 20.

In step S2, the pruning unit 33 extracts only the time series data related to the fault from among the multiple metrics data. The pruning unit 33 supplies the extracted time series data and the corresponding metadata to the time series data encoder 34 and the metadata encoder 35. This reduces the metrics data used to estimate the fault information.

In step S3, the time series data encoder 34 simultaneously encodes the time series data and the timestamps. In this encoding, the timestamps representing absolute times are converted into timestamps representing relative times within a time window. Furthermore, for each metric, a vector representation is calculated from the timestamps representing relative times and other data values, and these are aggregated. This allows asynchronous time series data to be handled in a unified manner.

In step S4, the metadata encoder 35 encodes the metadata. In this encoding, the time series data and the metadata are learned simultaneously. This allows the meaning of the time series data to be understood from the text information of the metadata. In addition, the relationships between the time series data can be understood.

In step S5, the fault information estimation unit 36 estimates fault information such as the status and cause of a fault occurring in the fault monitoring system based on the encoding result of the time-series data encoder 34 and the encoding result of the metadata encoder 35. The fault information estimation unit 36 also creates a fault report based on the estimation result.

In step S6, the fault report output unit 37 receives a fault report from the fault information estimation unit 36 and outputs the fault report.

[effect]
In the embodiment, the pruning unit 33 extracts metrics data related to a failure from among the multiple metrics data acquired by the data acquisition unit 31. This reduces the amount of metrics data used in the analysis work to estimate the failure information, and shortens the time required for the analysis work to estimate the failure information.

In addition, the time series data encoder 34 converts timestamps that represent absolute times into timestamps that represent relative times, and calculates vector expressions from the timestamps that represent relative times and other data values to aggregate them. This makes it possible to handle asynchronous time series data in a unified manner without processing missing values, and to capture relationships between asynchronous metrics.

Furthermore, the metadata encoder 35 simultaneously learns the time-series data and the metadata. This allows the meaning of the time-series data to be captured from the text information of the metadata, and also allows the relationship between the time-series data to be captured. This allows the system to respond to dynamic changes in the number and type of metrics.

As a result, the scope of monitoring targets of the monitoring system will be expanded, leading to reduced development costs. In addition, the accuracy of fault detection using metrics will be improved.

Note that the present invention is not limited to the above-described embodiments, and can be modified in various ways in the implementation stage without departing from the gist of the invention. The embodiments may also be implemented in appropriate combination, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from the multiple constituent elements disclosed. For example, if the problem can be solved and an effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiments, the configuration from which these constituent elements are deleted can be extracted as an invention.

LIST OF SYMBOLS 10 Node 11 Application 12 Monitoring agent 13 Data recording unit 20 Monitoring system 30 Fault information estimation device 31 Data acquisition unit 33 Pruning unit 34 Time-series data encoder 35 Metadata encoder 36 Fault information estimation unit 37 Fault report output unit 41 Input device 42 CPU
43: control device 44: arithmetic unit 45: storage device 46: main storage device 47: auxiliary storage device 48: output device 49: bus

Claims (6)

a data acquisition unit that acquires data having time series data and metadata of a plurality of metrics of a plurality of monitored targets in a monitored system;
a pruning unit that extracts metrics data related to a fault from the plurality of metrics data;
a fault information estimation unit that estimates fault information of a monitoring target in which a fault has occurred based on the data extracted by the pruning unit;
a time series data encoder that converts a timestamp representing an absolute time of the time series data extracted by the pruning unit into a timestamp representing a relative time within a time window;
Fault information estimation device. The pruning unit extracts data by calculating an anomaly score for the time series data using a one-dimensional time series anomaly detection model.
The fault information estimation device according to claim 1 . The time series data encoder further calculates a vector representation for each metric from a timestamp representing a relative time and other data values and aggregates them.
The fault information estimation device according to claim 1 or 2. The fault information estimation device according to any one of claims 1 to 3, further comprising a metadata encoder that simultaneously learns, for each metric, the time-series data supplied from the time-series data encoder and the metadata supplied from the pruning unit. Obtaining data having time series data and metadata for a plurality of metrics of a plurality of monitored targets in a monitored system;
extracting metric data related to a fault from the plurality of metric data;
estimating fault information of a monitoring target in which a fault has occurred based on the metric data related to the fault;
converting timestamps representing absolute times of the metric time series data related to the faults into timestamps representing relative times within a time window;
Fault information estimation method. A fault information estimation program that causes a computer to execute the functions of each component of the fault information estimation device according to any one of claims 1 to 4 .

Images