JP7564447B2 - Method and program for determining cause of abnormality - Google Patents

Description

The present invention relates to an abnormality factor determination method and an abnormality factor determination program.

Technology that uses a monitoring device to monitor the operating status of an information processing system and detect the occurrence of an abnormality is widely used. One method of detecting the occurrence of an abnormality is, for example, to use metrics that indicate the usage status of resources included in the information processing system. Furthermore, with such anomaly detection technology, when an abnormality is detected, it is necessary to determine the cause of the abnormality. One method of determining the cause of the abnormality is, for example, to analyze the log of events executed on the information processing system.

Furthermore, the following technologies have been proposed for monitoring information processing systems and analyzing the causes of anomalies. For example, a fault analysis system has been proposed that continuously acquires monitoring data from a monitored system to create a behavior model that models the system's behavior, estimates the period during which the behavior changed based on the difference between the successively created behavior models, and notifies the user. Another proposed anomaly analysis support system generates variable relation information that indicates the correspondence between the input/output of equipment in a system and the variables of an application program, and, when an equipment anomaly is detected, identifies variables related to the input/output of the equipment based on the variable relation information, and extracts and displays information on events related to the identified variables from the event information that has occurred.

International Publication No. WO2014/184934 JP 2017-227973 A

When a monitoring device such as the one described above detects an abnormality in an information processing system, it is considered that the device acquires a log of an event executed on the information processing system and determines the cause of the abnormality based on the contents of the acquired log. Usually, when an abnormality is detected, the event that may have been the cause of the abnormality is often executed immediately before the detection time. However, there are also cases where it takes a long time from when an abnormality occurs due to the execution of an event until the abnormality is detected. In such cases, it is necessary to set the search period for searching the database for logs of events that may be the cause of the abnormality to a long period ending at the time the abnormality was detected, otherwise the appropriate event log cannot be acquired. However, the longer the search period, the greater the number of logs to be searched, and the longer the time required for the search, which results in a problem of longer time required for determining the cause of the abnormality.

In one aspect, the present invention aims to provide an abnormality cause determination method and an abnormality cause determination program that can shorten the time required to determine the cause of an abnormality.

In one proposal, a method for determining the cause of an anomaly is provided, in which, when an anomaly is detected at a first time based on a first metric among a plurality of metrics indicating the usage status of resources included in an information processing system, a computer identifies one or more second metrics from among the plurality of metrics excluding the first metric, which indicate that the corresponding resource was unused immediately before the first time, and, based on the usage status indicated by each of the one or more second metrics, identifies a second time at which the corresponding resource changes from an unused state to a used state by going back from immediately before the first time for each of the one or more second metrics, specifying a search period from a third time, which is the oldest of the identified second times, to the first time, and acquiring logs of candidate events that were executed during the search period and are candidate causes of the anomaly based on the first metric from a database in which logs of events executed on the information processing system are accumulated, and determining the cause of the anomaly based on the first metric based on the candidate events indicated by the acquired logs.

In one proposal, an abnormality cause determination program is provided that causes a computer to execute processing similar to the abnormality cause determination method described above.

On the one hand, it can shorten the time it takes to determine the cause of an abnormality.

1 is a diagram illustrating an abnormality factor determination device according to a first embodiment; FIG. 13 illustrates an example of a configuration of an information processing system according to a second embodiment. FIG. 2 illustrates an example of a hardware configuration of a monitoring device. 2 is a diagram illustrating an example of a configuration of processing functions included in an operation management apparatus and a monitoring apparatus; FIG. 2 illustrates an example of a data structure of a metric database; FIG. 4 illustrates an example of a data structure of a determination rule database; FIG. 13 is a diagram illustrating an example of a data configuration of a determination result database. FIG. 11 is a first diagram showing a comparative example of a process for determining the cause of an abnormality; FIG. 2 is a second diagram showing a comparative example of the process for determining the cause of an abnormality. FIG. 11 is a diagram illustrating a process for determining the cause of an abnormality in the second embodiment. 13 is a flowchart illustrating an example of a processing procedure of a monitoring device according to a second embodiment. 13A and 13B are diagrams illustrating a process for determining the cause of an abnormality in a modified example. 13 is a flowchart illustrating an example of a processing procedure of a monitoring device according to a modified example.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First Embodiment
Fig. 1 is a diagram showing an abnormality factor determination device according to a first embodiment. The abnormality factor determination device 1 shown in Fig. 1 is a device that monitors the operating status of an information processing system (not shown) and, when an abnormality is detected, determines the cause of the abnormality. The abnormality factor determination device 1 is realized as a computer such as a server device or a personal computer. In this case, the processing of the abnormality factor determination device 1 described below is realized, for example, by a processor included in the abnormality factor determination device 1 executing a predetermined program.

The abnormality cause determination device 1 is capable of acquiring metrics from a metric database (DB) 2. In the metric database 2, a number of metrics indicating the usage status of the resources included in each of the above-mentioned information processing systems are successively collected from the information processing systems and stored. For example, if the corresponding resource is a CPU (Central Processing Unit), the metrics include CPU usage rate and CPU wait time. If the corresponding resource is memory, the metrics include memory usage and memory swap-out amount. If the corresponding resource is a network interface, the metrics include network usage and number of packet losses.

The abnormality cause determination device 1 periodically acquires a specific metric from among multiple metrics stored in the metric database 2, and can detect an abnormality in the information processing system based on the value of the acquired metric. In addition, when the abnormality cause determination device 1 detects an abnormality, it can also acquire other metrics in the metric database 2 to determine the cause of the abnormality.

In addition, the event log database (DB) 3 accumulates logs of events that have been executed on the information processing system. In order to determine the cause of a detected abnormality, the abnormality factor determination device 1 can specify search conditions and obtain logs of events that match the search conditions from the event log database 3. Note that the search process for the event log database 3 itself may be executed by the abnormality factor determination device 1, or may be executed by another device connected externally to the abnormality factor determination device 1.

On the other hand, time chart 4 shown on the right side of FIG. 1 shows an example of the status of other metrics and events when an abnormality is detected based on a certain metric. Below, the processing of the abnormality cause determination device 1 will be explained using the example shown in time chart 4.

The abnormality cause determination device 1 determines whether or not an abnormality exists in the information processing system based on one or more specific metrics among the metrics stored in the metric database 2. As an example, the determination process for determining whether or not an abnormality exists based on a specific metric is executed at each determination time at a predetermined time interval. In this case, the determination process at a certain determination time is executed based on the metrics stored in the metric database 2 during the period from the previous determination time to the current determination time.

The time chart 4 in FIG. 1 shows an example in which the presence or absence of an abnormality is determined based on metric M1 (first metric). The abnormality cause determination device 1 detects no abnormality at times T1, T2, T3, and T4 among the above determination times from metric M1, but detects an abnormality at time T5 (first time) (step S1).

Then, the abnormality cause determination device 1 identifies one or more metrics (second metrics) that indicate that the corresponding resource is unused immediately before time T5 from among the multiple metrics stored in the metric database 2, except for metric M1. In the time chart 4 of FIG. 1, it is assumed that metrics M2 and M3 that indicate that the corresponding resource is unused at time T4, which is the determination time immediately before time T5, are identified from among metrics M2 to M4, except for metric M1 (step S2).

Next, based on the usage status indicated by each of the identified metrics M2 and M3, the abnormality cause determination device 1 identifies, for each of the metrics M2 and M3, the time (second time) at which the corresponding resource changes from an unused state to a used state, going back from just before time T5 (here, time T4) (step S3).

In time chart 4 of FIG. 1, for metric M2, the corresponding resource changes to a used state in the period from time T2 to time T1. Therefore, time T1 is identified as the above time for metric M2. Also, for metric M3, the corresponding resource changes to a used state in the period from time T3 to time T2. Therefore, time T2 is identified as the above time for metric M3.

Next, the abnormality cause determination device 1 selects the oldest time T1 from the times T1 and T2 identified in step S3, and specifies the search period from the selected time T1 to the time T5 at which the abnormality was detected. Then, the abnormality cause determination device 1 obtains from the event log database 3 a log of candidate events that were executed during the specified search period and are candidates for the cause of the abnormality based on the metric M1 (step S4). Here, the candidate events that are candidates for the cause of the detected abnormality are predetermined, for example, according to the metric that was the basis for the abnormality detection.

In step S4, the event log database 3 is searched using the search period from time T1 to time T5 and the candidate event as search conditions, and logs of candidate events that match the search conditions are obtained. As described above, the search process of the event log database 3 itself may be executed by the abnormality factor determination device 1, or may be executed by another device connected to the outside of the abnormality factor determination device 1.

In the time chart 4 of FIG. 1, an event that caused an abnormality was executed in the period from time T2 to time T3, and a log 5 corresponding to this event was registered in the event log database 3. In this case, in step S4, the log 5 is acquired as a log of a candidate event. The abnormality cause determination device 1 then determines the cause of the occurrence of the abnormality based on the metric M1, based on the candidate event indicated by the log 5 acquired in step S4 (step S5).

When an abnormality is detected in the information processing system, the event that may have caused the abnormality is often executed immediately before the time of detection. In order to obtain logs of such events from the event log database 3, it is sufficient to set the log search period to a period equivalent to the abnormality determination cycle.

On the other hand, in the example shown in time chart 4 of Figure 1, it takes a long time from when an abnormality occurs due to the execution of the event shown in log 5 until the abnormality is detected. In order to obtain such an event log from event log database 3, the log search period needs to be made longer. However, the longer the log search period, the greater the number of logs to be searched, and the longer the time required for the search. As a result, it takes longer to determine the cause of the abnormality.

Cases in which it takes a long time for an abnormality to be detected include cases where an abnormality occurs related to a resource during a period in which the resource is not being used. More specifically, there are cases in which an abnormality occurs related to a resource due to the execution of a certain event, but the resource is not being used at the time, and then when the resource begins to be used again, an abnormal event appears and the abnormality is detected.

In the example shown in time chart 4 of Figure 1, when the event shown in log 5 was executed, the resource related to that event was not in use, and then just before time T5, the resource started to be used, which is why an abnormality was detected at time T5.

The abnormality factor determination device 1 then identifies, from among the metrics other than metric M1, metrics M2 and M3 that indicate that the corresponding resource was unused immediately before time T5. Next, for each of the identified metrics M2 and M3, the abnormality factor determination device 1 identifies times T1 and T2 at which the corresponding resource changed from an unused state to an used state by going back from immediately before time T5. The abnormality factor determination device 1 then determines the oldest of the identified times T1 and T2 as the start time of the log search period.

Through this processing, the start time of the search period is determined so that all event logs related to resources that were unused until just before the time T5 when the abnormality was detected can be included in the search. This allows the search period to be set to the minimum length necessary. This increases the possibility of obtaining logs of candidate events that may be the cause of the detected abnormality while keeping the length of the search period down. This reduces the time required to search the event log database 3, thereby shortening the time required from when the abnormality cause determination device 1 detects an abnormality to when it determines the cause of the abnormality, while improving the accuracy of the determination.

Second Embodiment
2 is a diagram showing an example of the configuration of an information processing system according to the second embodiment. The information processing system shown in FIG.

The operation management device 100 manages the operation of an ICT (Information and Communication Technology) infrastructure 110. Hereinafter, the ICT infrastructure will be abbreviated as "ICT infrastructure." The ICT infrastructure 110 includes various information processing devices such as computers and network devices. For example, if the ICT infrastructure 110 provides a cloud service, the ICT infrastructure 110 includes server devices that operate as cloud servers and network devices that connect the server devices.

The operation management device 100 executes various events (operation events) related to operation management for each information processing device included in the ICT infrastructure 110. Operation events are processes that perform various configuration changes and setting changes in the ICT infrastructure 110, such as creating, deleting, and migrating virtual machines running on a server device, and updating programs such as drivers. The monitoring device 200 executes operation events and records logs related to the executed operation events in a database.

The operation management device 100 also monitors the operating status of each information processing device included in the ICT infrastructure 110, and collects resource-related metrics from each information processing device. Metrics are information that indicate the operating status of monitored resources such as processors and memory, and provide, for example, a scale for evaluating the operating status of resources.

The monitoring device 200 monitors the operating status of the ICT infrastructure 110 via the operation management device 100, and if an abnormality is detected, analyzes the cause of the abnormality. Specifically, the monitoring device 200 acquires metrics collected by the operation management device 100 and determines the normality of operation. If an abnormality is detected, the monitoring device 200 acquires an operation event log from the operation management device 100 and identifies the operation event that may be the trigger for the abnormality, and determines the cause of the abnormality based on the identified operation event.

3 is a diagram showing an example of a hardware configuration of the monitoring device 200. The monitoring device 200 is realized as a computer as shown in FIG.
The monitoring device 200 shown in FIG. 3 includes a processor 201, a random access memory (RAM) 202, a hard disk drive (HDD) 203, a graphics processing unit (GPU) 204, an input interface (I/F) 205, a reading device 206, and a communication interface (I/F) 207.

The processor 201 performs overall control of the entire monitoring device 200. The processor 201 is, for example, a CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). The processor 201 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

The RAM 202 is used as the main storage device of the monitoring device 200. The RAM 202 temporarily stores at least a portion of the OS (Operating System) program and application programs to be executed by the processor 201. The RAM 202 also stores various data necessary for processing by the processor 201.

The HDD 203 is used as an auxiliary storage device for the monitoring device 200. The HDD 203 stores the OS program, application programs, and various data. Note that other types of non-volatile storage devices, such as a solid state drive (SSD), can also be used as the auxiliary storage device.

A display device 204a is connected to the GPU 204. The GPU 204 displays an image on the display device 204a in accordance with an instruction from the processor 201. Examples of the display device include a liquid crystal display and an organic EL (ElectroLuminescence) display.

The input interface 205 is connected to the input device 205a. The input interface 205 transmits a signal output from the input device 205a to the processor 201. Examples of the input device 205a include a keyboard and a pointing device. Examples of the pointing device include a mouse, a touch panel, a tablet, a touch pad, and a trackball.

A portable recording medium 206a is detachably attached to the reading device 206. The reading device 206 reads data recorded on the portable recording medium 206a and transmits it to the processor 201. Examples of the portable recording medium 206a include an optical disk and a semiconductor memory.

The communication interface 207 transmits and receives data to and from other devices such as the operation management unit 100 via a network 207a.
The above hardware configuration can realize the processing functions of the monitoring device 200. The operation management device 100 can also be realized as a computer having a configuration as shown in FIG.

FIG. 4 is a diagram illustrating an example of the configuration of processing functions provided in the operation management device and the monitoring device.
First, the operation management apparatus 100 includes an event execution unit 101, an event log search unit 102, and a metric collection unit 103. The processing of the event execution unit 101, the event log search unit 102, and the metric collection unit 103 is realized, for example, by a processor (not shown) included in the operation management apparatus 100 executing a predetermined program. Also, an event log database (DB) 104 and a metric database (DB) 105 are stored in a storage device (e.g., RAM) (not shown) of the operation management apparatus 100.

The event execution unit 101 executes operation events for each information processing device included in the ICT infrastructure 110. The event execution unit 101 registers a log of the executed operation events in the event log database 104. The operation event log includes information indicating the content of the executed process, information indicating whether the execution was successful, information indicating the time of execution, etc.

The event log search unit 102 searches the event log database 104 in response to a search request from, for example, the monitoring device 200, and returns the searched operational event log.
The metric collector 103 collects metrics from each information processing device included in the ICT infrastructure 110, and registers the collected metrics in the metric database 105. As the metrics, for example, CPU waiting time, CPU usage rate, memory swap-out amount, packet loss count, network usage rate, etc. in the server device are collected.

Next, the monitoring device 200 includes a metric acquisition unit 211, a normality determination unit 212, and a factor determination unit 213. The processing of the metric acquisition unit 211, the normality determination unit 212, and the factor determination unit 213 is realized, for example, by the processor 201 included in the monitoring device 200 executing a predetermined program. In addition, a metric database (DB) 221, a determination rule database (DB) 222, and a determination result database (DB) 223 are stored in a storage device (for example, RAM 202) of the monitoring device 200.

The metric acquisition unit 211 acquires metrics registered in the metric database 105 of the operation management device 100 , and registers them in the metric database 221 .
The normality determining unit 212 periodically executes normality determination processing for metrics based on the metrics registered in the metric database 221. This normality determination processing is executed using metrics collected by the operation management device 100 within a fixed period of time in the past. When an abnormality in a metric is detected, the normality determining unit 212 notifies the cause determining unit 213 of the metric (abnormality detection metric).

In the judgment rule database 222, anomaly detection metrics, operational events that may be the cause of an anomaly in the metric, and anomaly occurrence causes are registered in advance in association with each other. The cause determination unit 213 identifies an operational event (causing event) that may be the cause of an anomaly in the anomaly detection metric notified by the normality determination unit 212 based on the judgment rule database 222.

The cause determination unit 213 requests the event log search unit 102 to search the event log database 104 for logs of cause events that were executed during the period from the current time to a time a predetermined time before. When a log of a cause event is searched for in the event log database 104, the cause determination unit 213 extracts, from the determination rule database 222, the abnormality occurrence cause corresponding to the operation event indicated by the searched log, and registers the determination result of the abnormality occurrence cause in the determination result database 223.

Figure 5 is a diagram showing an example of the data structure of a metric database. Figure 5 shows the metric database 221 of the monitoring device 200, but the metric database 105 of the operation management device 100 also has a similar data structure.

In the metric database 221, metric values for each metric type (monitoring item) are registered in association with the collection time when the metric was collected. In the example of FIG. 5, the following metric items are registered: CPU usage of host #1, network usage of NIC (Network Interface Card) #1 of host #1, and network usage of NIC #2 of host #1. In this example, it is assumed that host #1, which is a server device on which a virtual machine runs, is equipped with at least a CPU and two NICs #1 and #2.

Figure 6 is a diagram showing an example of the data configuration of the judgment rule database. The judgment rule database 222 is a database that is referenced to estimate the cause of an anomaly from a metric in which an anomaly has been detected (anomaly detection metric). In the judgment rule database 222, a causal event, which is an operational event that may be the cause of an anomaly, and the cause of the anomaly are registered in association with each anomaly detection metric. This information is registered in advance in the judgment rule database 222.

In the example of Figure 6, when the anomaly detection metric is CPU wait time, the migration of a virtual machine (VM) is considered as a causal event, and it is registered that CPU contention due to the migration may be a cause of the anomaly. Also, when the anomaly detection metric is memory swap-out amount, the migration of a virtual machine is considered as a causal event, and it is registered that memory contention due to the migration may be a cause of the anomaly.

Furthermore, when the anomaly detection metric is the number of packets lost, it is registered that a virtual machine migration may be a contributing event, and that network contention due to this migration may be a contributing factor in the occurrence of an anomaly. As another example, when the anomaly detection metric is the number of packets lost, it is registered that a NIC driver update may be a contributing event, and that a malfunction of the NIC driver may be a contributing factor in the occurrence of an anomaly.

Figure 7 is a diagram showing an example of the data configuration of the judgment result database. In the judgment result database 223, the abnormality detection time, monitoring host name, monitoring location, abnormality detection metric, and cause are registered in association with each other as information indicating the judgment result. The abnormality detection time indicates the time when the abnormality was detected. The monitoring host name indicates the host to be monitored. The monitoring location indicates the location to be monitored on that host. The abnormality detection metric indicates the metric at which the abnormality was detected. The cause indicates the determined cause of the abnormality.

Next, a comparative example of the process for determining the cause of an abnormality will be described with reference to FIGS.
FIG. 8 is a first diagram showing a comparative example of a process for determining the cause of an abnormality.
The normality determination unit 212 of the monitoring device 200 determines the normality of the operating status of the ICT infrastructure 110 based on the metrics collected by the operation management device 100. Such normality determination times are set at regular time intervals, and the normality determination unit 212 determines the normality based on the metrics collected during a regular time period immediately prior to the determination time. In Fig. 8, as an example, the normality determination times are set at 3-minute intervals.

Among the multiple metrics collected, one or more specific metrics to be used for normality determination are predetermined. In FIG. 8, CPU usage, memory swap-out amount, and number of packet losses are exemplified as metrics to be used for normality determination. The memory swap-out amount indicates the amount of data evacuated from memory to a HDD or SSD during a certain period (the period from the previous determination time to the current determination time), and the number of packet losses indicates the number of packet losses that occurred during the certain period.

For example, based on a judgment threshold set for each metric, the normality determination unit 212 determines that an abnormality has been detected for that metric when the metric value exceeds the judgment threshold or falls below the judgment threshold. For example, in the case of the CPU usage rate, memory swap-out amount, and number of packet losses shown in FIG. 8, an abnormality is determined to have been detected when the value exceeds the judgment threshold. Note that normality (and abnormality detection) may actually be determined based on the values of multiple metrics that are related to each other. For example, normality or abnormality may be determined based on the correlation between the number of packet losses in a certain period of time and the number of transmitted packets in a certain period of time.

When the normality determination unit 212 detects an abnormality in a certain metric, the cause determination unit 213 refers to the determination rule database 222 to identify an operational event (causing event) that may be the cause of the abnormality in the metric in which the abnormality was detected. In the example of FIG. 8, it is assumed that an abnormality in the number of packet losses was detected at 10:09. Here, in the example of the determination rule database 222 shown in FIG. 6, VM migration and NIC driver update are registered as causing events for the number of packet losses. Therefore, in the example of FIG. 8, VM migration and NIC driver update are identified as causing events.

The cause determination unit 213 also requests the event log search unit 102 of the operation management device 100 to search for logs of cause events that were executed during the period from the current determination time to the previous determination time (the period from 10:06 to 10:09). In the example of FIG. 8, it is assumed that a log LG1 indicating that the NIC driver has been updated is searched for. In this case, the cause determination unit 213 extracts the cause of the abnormality associated with the number of packet losses and the NIC driver update from the determination rule database 222. When based on the determination rule database 222 of FIG. 6, a malfunction of the NIC driver is extracted as the cause. The cause determination unit 213 registers the determination result of such an abnormality cause in the determination result database 223.

Anomalies occurring in the ICT infrastructure 110 are often triggered by configuration changes or setting change events (operational events) executed in the operational management of the ICT infrastructure 110. According to the above process, the cause of the anomaly is determined based on the log of the operational event related to the metric in which the anomaly was detected, thereby improving the accuracy of determining the cause.

However, in the above method, in a case such as the example shown in FIG. 9, it is not possible to obtain a log of an appropriate cause event by search, and therefore it is not possible to accurately determine the cause of the abnormality.
9 is a second diagram showing a comparative example of the process of determining the cause of an anomaly. Among anomalies, there are some in which, when an anomaly occurs with the execution of a causal event, the anomaly is not detected immediately, but is detected after some time has passed. One such example is when an anomaly occurs in a certain resource due to the execution of a causal event, but the resource is not being used at that time, and the anomaly is detected when the resource is used thereafter.

In the example of Figure 9, let us say that a causal event to update the driver of NIC#1 was executed between 10:09 and 10:12, which caused an abnormality in the operation of the driver of NIC#1 (or NIC#1). However, let us say that the driver of NIC#1 was not being used at this point (no communication was taking place via NIC#1). In this case, no packet loss occurs in the communication via NIC#1, so no abnormality is detected from the metric of the number of packet losses.

However, suppose that communication using NIC #1 is subsequently started between 10:15 and 10:18. Because an abnormality has occurred in the driver for NIC #1 (or NIC #1), packet loss occurs in the communication started by NIC #1. For this reason, the abnormality is detected from the number of packet losses in the normality determination process at 10:18. In this way, there are cases where an abnormality is detected a long time after the execution of the causal event.

As explained in FIG. 8, the period during which the event log database 104 is searched for logs of causal events among operational events is set to a time period corresponding to the normality determination cycle. In this case, when an abnormality is detected from the number of packet losses at 10:18 in FIG. 9, the log search period (P1) is the three minutes immediately preceding that time. However, during search period P1, log LG2 indicating the driver update for NIC#1 cannot be obtained, and therefore the cause of the abnormality cannot be determined.

A possible method for solving such a problem is to lengthen the search period for logs of causal events. For example, as shown in FIG. 9, by setting a longer search period P2, it becomes possible to obtain log LG2 indicating a driver update for NIC#1. However, the longer the search period, the greater the number of event logs to be searched in the event log database 104, and it becomes necessary to search for logs of causal events that match the search conditions from among a large number of event logs. This increases the time required for search processing in the operation management device 100, which in turn increases the time required for the entire process of determining the cause of the abnormality by the monitoring device 200. Furthermore, an increase in the search processing load in the operation management device 100 may, in some cases, cause problems in the execution processing of operation events by the operation management device 100.

Figure 10 is a diagram showing the process of determining the cause of an abnormality in the second embodiment. In this embodiment, the cause determination unit 213 of the monitoring device 200 determines the search period for the cause event log in the following procedure. In this Figure 10, it is assumed that an abnormality caused by a driver update of NIC#1 has been detected from the number of packet losses, similar to Figure 9.

When an abnormality is detected from the packet loss count at 10:18, the cause determination unit 213 sets that time as the end time Te of the search period for the cause event log. The cause determination unit 213 also refers to the metric database 221 and identifies a metric that indicates that the corresponding resource was not in use at the immediately preceding normality determination time from among other metrics other than the packet loss count. In the example of FIG. 10, it is assumed that the other metrics include CPU utilization and network utilization, which are metrics that indicate the amount of resource usage. These metrics indicate that the resource is not in use when the numerical value is 0. For this reason, in the example of FIG. 10, the network utilization of NIC #1 and the network utilization of NIC #2 are identified as metrics with a numerical value of 0.

Next, the cause determination unit 213 retrieves the numerical values for each of the identified metrics going back in time, and identifies the time when the numerical values became greater than 0. This identifies the end of the period during which the resource corresponding to the metric was in use. In the example of FIG. 10, the network usage rate of NIC #1 changed from 0% to 30% at 10:06, and the network usage rate of NIC #2 changed from 0% to 20% at 10:09. Therefore, 10:06 is identified as the time when the numerical values became greater than 0 for the network usage rate of NIC #1, and 10:09 is identified as the time when the numerical values became greater than 0 for the network usage rate of NIC #2.

The cause determination unit 213 identifies the oldest time from among the times identified in this manner, and sets this time as the start time Ts of the search period for the cause event log. In the example of FIG. 10, 10:06, which is the time for the network usage rate of NIC #1, is identified as the start time Ts of the search period. As a result, the period from the start time Ts to the aforementioned end time Te is determined as the search period. By searching for the cause event log from such a search period, the cause determination unit 213 can obtain log LG2 indicating a driver update for NIC #1, and can accurately determine the cause of the abnormality.

As mentioned above, if it takes a long time from the occurrence of an abnormality related to a certain resource to its detection, the abnormality may have been triggered by an operation event executed during a period when the resource was not in use. In the above process, the oldest time among the times at which the metric value turned to a value greater than 0 is set as the start time of the search period. This makes it possible to include all logs of operation events related to resources that were not in use until just before the abnormality was detected in the search target. In other words, the search period for the cause event log can be set to the minimum required length. This increases the possibility of obtaining the log of the operation event that triggered the occurrence of the detected abnormality while suppressing the length of the search period. Therefore, the search processing time in the operation management device 100 can be shortened, thereby shortening the time required for the monitoring device 200 to determine the cause of the abnormality while improving the accuracy of the determination. In addition, the search processing load in the operation management device 100 can be suppressed while improving the accuracy of the determination of the cause of the abnormality.

In addition, FIG. 10 shows an example in which it takes time from the occurrence of an anomaly to its detection, and in response to the detection of an anomaly in the number of packet losses, the numerical change in network utilization is analyzed as another metric. As another example, when an anomaly is detected from the CPU wait time as a metric, the numerical change in CPU usage is analyzed as another metric.

11 is a flowchart showing an example of a processing procedure of a monitoring device according to the second embodiment. The processing in FIG. 11 is executed every time normality is determined.
[Step S<b>11 ] The metric acquisition unit 211 acquires metrics collected during the period from the current judgment time to the previous judgment time from the metric database 105 of the operation management device 100 , and registers them in the metric database 221 .

[Step S12] The normality determination unit 212 determines the normality of the ICT infrastructure 110 based on one or more predetermined metrics from the metrics registered in step S11. If an abnormality is detected based on the metrics, the process proceeds to step S13. In this case, the normality determination unit 212 notifies the cause determination unit 213 of the metric in which the abnormality was detected. Then, the processes of steps S13 to S17 are executed for each notified metric. On the other hand, if no abnormality is detected from any of the metrics, the process of FIG. 11 is terminated.

[Step S13] The cause determination unit 213 identifies a cause event (an operational event that is a candidate for the cause of the abnormality) corresponding to the metric in which the abnormality was detected, based on the determination rule database 222.

[Step S14] The cause determination unit 213 identifies, from among metrics other than the metric in which the abnormality was detected, a metric that indicates that the corresponding resource was unused at the normality determination time immediately prior to the abnormality detection time. For example, from among metrics that indicate the amount of resource usage, it identifies a metric whose numerical value is 0 at the normality determination time immediately prior to the abnormality detection time.

[Step S15] The factor determination unit 213 retrieves numerical values from the metric database 221 going back in time for each metric identified in step S14. Then, for each metric, the factor determination unit 213 identifies the time when the resource usage state changed from an unused state to a used state. In the case of a metric that indicates the amount of resource usage as described above, the time when the metric value changed from 0 to a value greater than 0 is identified.

When a metric indicating resource usage is used, in steps S14 and S15, the criterion used is whether the metric value is 0 or greater, but a judgment threshold greater than 0 may be used as the judgment criterion. For example, the judgment threshold may be 0.01, and in step S14, metrics whose numerical values are 0.01 or less may be identified, and in step S15, the time when the metric value changes from 0.01 or less to exceed 0.01 may be identified.

[Step S16] The cause determination unit 213 identifies the oldest time from among the times identified in step S15, and determines that time as the start time Ts of the search period for the cause event log.

[Step S17] The cause determination unit 213 sets the period from the current determination time (end time Te) to the above-mentioned start time Ts as the search period, and requests the operation management device 100 to search for event logs by specifying this search period and the identification information of the cause event identified in step S13 as arguments. The event log search unit 102 of the operation management device 100 extracts the log of the specified cause event from the logs of operation events collected during the specified search period, and returns it to the monitoring device 200. The cause determination unit 213 receives and acquires the log of the extracted cause event.

[Step S18] The cause determination unit 213 refers to the determination rule database 222 and obtains the cause associated with the metric in which the abnormality was detected (anomaly detection metric) and the cause event indicated by the log obtained in step S17. The cause determination unit 213 determines that the obtained cause is the abnormality occurrence cause and outputs the determination result. For example, the determination result is registered in the determination result database 223 as a combination of the abnormality detection time, monitoring host name, monitoring location, anomaly detection metric, and the obtained cause.

The monitoring host name and monitoring location are identified from at least one of the anomaly detection metric, the content of the causal event indicated by the log acquired in step S17, and a combination of two or more of the causes of the anomaly based on these. For example, if the causal event is a NIC driver update, the NIC corresponding to the updated NIC driver is identified as the monitoring location, and the name of the host (server device) on which that NIC is installed is identified as the monitoring host name. Also, if the anomaly detection metric is CPU wait time and the causal event is VM migration, the monitoring location is identified as the CPU that is the target of CPU wait time detection, and the name of the host on which that CPU is installed is identified as the monitoring host name.

If the search in step S17 returns logs of multiple causal events, in step S18, the causes of the abnormality based on each causal event may be output as possible causes of the abnormality.

[Modification of the second embodiment]
A part of the processing of the monitoring device 200 in the second embodiment may be modified as follows.

Figure 12 is a diagram showing the process of determining the cause of an abnormality in a modified example. In this Figure 12, it is assumed that an abnormality caused by a driver update of NIC#1 is detected from the number of packet losses, as in Figures 9 and 10.

In cases where it takes time from the occurrence of an abnormality to its detection, as in Figures 9, 10, and 12, an abnormality occurs with respect to a resource that is not in use, and then the abnormality is detected when the use of that resource begins. Thus, when an abnormality in a metric is detected, the cause determination unit 213 identifies, from among other metrics that are different from the metric, a metric for which use of a resource corresponding to the period from the immediately preceding normality determination time to the current determination time, based on the metric value. The cause determination unit 213 then obtains a numerical value going back in time for the identified metric, and based on the obtained numerical value, identifies the end of the period when the corresponding resource was in use, and determines the start time of the search period for the cause event log.

In the example of FIG. 12, when an abnormality is detected from the number of packet losses at 10:18, the cause determination unit 213 identifies, from among other metrics that indicate resource usage other than the number of packet losses, a metric whose value was 0 at the immediately preceding normality determination time and whose value exceeds 0 at the current determination time. In FIG. 12, the network utilization rate of NIC#1 is identified as such a metric. Then, the cause determination unit 213 retroactively obtains the identified network utilization rate value and identifies the time when the value changed from 0 to a value greater than that. In FIG. 12, the network utilization rate of NIC#1 changes from 0% to 30% at 10:06, and 10:06 is identified as the time when the value changed to a value greater than 0, and this time is determined as the start time Ts of the search period.

The above process makes it possible to narrow down the metrics to be analyzed for changes in numerical values to identify the end of the period when the corresponding resource was in use, reducing the processing load for determining the start time of the search period and shortening the processing time. Also, by identifying the metrics for which the corresponding resource began to be used immediately before the time the anomaly was detected, it is possible to narrow down the metrics to be analyzed for changes in numerical values to only those that are likely to be related to the detected anomaly. This makes it possible to shorten the processing time for determining the search period without reducing the accuracy of determining the cause of the anomaly, and as a result, the overall process for determining the cause of the anomaly can be shortened.

Figure 13 is an example of a flowchart showing the processing procedure of a monitoring device in a modified example. In this modified example, among the processing steps of the flowchart shown in Figure 11, the processing of step S14 is changed to the processing of the next step S14a.

[Step S14a] The cause determination unit 213 identifies, from among other metrics other than the metric in which the abnormality was detected, a metric whose corresponding resource was in an unused state at the normality determination time immediately prior to the abnormality detection time and whose corresponding resource changed to a used state at the abnormality detection time. For example, from among the metrics indicating the amount of resource usage, a metric whose numerical value was 0 at the normality determination time immediately prior to the abnormality detection time and whose numerical value is greater than 0 at the abnormality detection time is identified.

In the next step S15, each metric identified in step S14a becomes a target for numerical value acquisition. This narrows down the metrics for which numerical values are to be acquired compared to the second embodiment.

The processing functions of the devices (e.g., the anomaly factor determination device 1, the operation management device 100, and the monitoring device 200) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical disks, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical disks include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

When distributing a program, for example, portable recording media such as DVDs and CDs on which the program is recorded are sold. The program can also be stored in a storage device of a server computer, and the program can be transferred from the server computer to other computers via a network.

A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. Note that the computer can also read a program directly from a portable recording medium and execute processing according to that program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network.

1 Anomaly factor determination device 2 Metric database 3 Event log database 4 Time chart 5 Log M1-M4 Metric S1-S5 Step T1-T5 Time

Claims (6)

The computer
when an abnormality is detected at a first time based on a first metric among a plurality of metrics indicating usage status of resources included in the information processing system, identifying one or more second metrics from among the plurality of metrics excluding the first metric, the second metrics indicating that a corresponding resource was unused immediately before the first time;
Identifying, for each of the one or more second metrics, a second time at which a corresponding resource changes from an unused state to a used state, going back from just before the first time to a past time based on the usage status indicated by each of the one or more second metrics;
specifying a search period from a third time, which is the oldest of the identified second times, to the first time, and acquiring, from a database in which logs of events executed in the information processing system are accumulated, logs of candidate events that are candidates for the cause of anomaly based on the first metric and that were executed during the search period;
determining a cause of the anomaly based on the first metric on the basis of the candidate event indicated by the acquired log;
Method for determining abnormality causes. In identifying the one or more second metrics, a metric whose value is equal to or less than a predetermined value immediately before the first time is identified as the one or more second metrics from among the plurality of metrics excluding the first metric;
In identifying the second time, a time when a value of each of the one or more second metrics went from being equal to or less than the predetermined value to exceeding the predetermined value going back to a time immediately before the first time is identified as the second time.
The method for determining an abnormality cause according to claim 1. The computer further identifies one or more third metrics among the one or more second metrics, the third metrics indicating that a corresponding resource has changed from an unused state to a used state at the first time;
In the determination of the second time, the second time is determined for each of the one or more third metrics.
The method for determining an abnormality cause according to claim 1. In identifying the one or more second metrics, a metric whose value is equal to or less than a predetermined value immediately before the first time is identified as the one or more second metrics from among the plurality of metrics excluding the first metric;
In identifying the one or more third metrics, a metric whose value has changed from less than or equal to the predetermined value to more than the predetermined value at the first time is identified as the one or more third metrics from among the one or more second metrics;
In identifying the second time, a time when a value of each of the one or more third metrics went from being equal to or less than the predetermined value to exceeding the predetermined value going back to a time immediately before the first time is identified as the second time.
The method for determining an abnormality cause according to claim 3. the computer executes a determination process for determining the presence or absence of an abnormality based on each of a plurality of specific metrics among the plurality of metrics at each determination time at a predetermined time interval;
In identifying the one or more second metrics, when an abnormality is detected based on the first metric of the identified metrics at the first time among the determination times, the one or more second metrics indicating that a corresponding resource is in an unused state at a determination time immediately before the first time among the determination times are identified from among the metrics excluding the first metric among the plurality of metrics.
The method for determining an abnormality cause according to claim 1 . On the computer,
when an abnormality is detected at a first time based on a first metric among a plurality of metrics indicating usage status of resources included in the information processing system, identifying one or more second metrics from among the plurality of metrics excluding the first metric, the second metrics indicating that a corresponding resource was unused immediately before the first time;
Identifying, for each of the one or more second metrics, a second time at which a corresponding resource changes from an unused state to a used state, going back from just before the first time to a past time based on the usage status indicated by each of the one or more second metrics;
specifying a search period from a third time, which is the oldest of the identified second times, to the first time, and acquiring, from a database in which logs of events executed in the information processing system are accumulated, logs of candidate events that are candidates for the cause of anomaly based on the first metric and that were executed during the search period;
determining a cause of the anomaly based on the first metric on the basis of the candidate event indicated by the acquired log;
An abnormality cause determination program that executes processing.

Images