JP5467936B2 - Fault monitoring apparatus, method and program for distributed / parallel processing system - Google Patents

Description

  The present invention relates to a technology for supporting the operation of a client / server computer system that constitutes a distributed / parallel processing system, and in particular, fault monitoring of a data center or the like that operates and manages a server computer (hereinafter simply referred to as a server). The present invention relates to a technique suitable for efficiently detecting a failure from an event occurring in a server constituting a distributed / parallel processing system.

  In a distributed / parallel processing system composed of a plurality of servers, the performance of the entire server group can be improved by increasing the number of servers constituting the system. Hereinafter, this is referred to as “scale-out”. On the other hand, reducing the system scale by reducing the number of servers is hereinafter referred to as “scale-in”.

  Conventionally, for example, as described in Patent Document 1, a standby server for scale-out prepared in advance is started only when the server group is in a high load state, and the server group waits when the load on the server group becomes low There is a technique for automatically stopping a server.

  With this technology, for example, you can scale out by starting a standby server only when high performance is needed temporarily, such as when web access is concentrated, and when high performance is not needed, you can scale in by stopping the standby server. And wasteful power consumption can be reduced. Hereinafter, this function is called “auto scale-out” and “auto scale-in”, and a system having this function is called “auto scale-out system”.

  On the other hand, there is a technique for monitoring a server for stable operation of a business processing computer system in a business operation of the business processing computer system. For example, by constantly monitoring the life and death of a server that is performing business processing, when an operating server stops due to some kind of failure, this can be detected and notified to the operation manager as a failure. Upon receiving the notification, the operation manager can restart the business processing computer system by identifying and removing the cause of the failure.

  However, when the scale-out server is started by automatic scale-out and then stopped again by automatic scale-in, the monitoring system detects the server stop and notifies the operation administrator as a failure.

  When the server is stopped during operation, it is necessary to notify the operation administrator as a failure. However, the stop due to automatic scale-in does not need to be notified as a failure because of normal operation. In conventional server monitoring, it is not possible to distinguish between a server stop due to a failure and a server stop due to automatic scale-in. Therefore, when the operation administrator receives a server stop failure notification, the operation administrator checks whether the stopped server is a scale-out server or whether the server has stopped due to automatic scale-in. It is necessary to judge whether.

JP 2008-225642 A

  The problem to be solved is that in the conventional technology, the operation administrator who has received the server stop failure notification indicates that the stopped server is a scale-out server and the server has been stopped by automatic scale-in. It is a point that cannot be judged.

  An object of the present invention is to solve these problems of the prior art, for example, in the operation of a business processing computer system that constitutes a distributed / parallel processing system, in a fault monitoring operation of a data center or the like that operates and manages a server. It is possible to efficiently detect a failure from an event that has occurred in a server constituting a parallel processing system.

  In order to achieve the above object, in the present invention, server usage information indicating whether the servers (103 to 105) to be monitored always operate or only operate during scale-out, and each server waits. Operation status information indicating whether it is in operation or in operation is stored and managed (111, 112), and the server usage information and operation status information of the server that generated the event are confirmed for the event detected by the monitoring system (107) By doing so, it is determined whether the event has occurred due to a failure or whether it has occurred due to automatic scale-out and automatic scale-in (108, 110). The operating state information indicates whether the standby server for scale-out is on standby or is operating, and is changed as appropriate in accordance with execution of automatic scale-out or automatic scale-in (109).

  According to the present invention, the normal operation of the standby server for automatic scale-out and scale-out by automatic scale-in is not erroneously notified to the operation administrator as a failure, and only the abnormal operation that should be notified as a failure is a failure. Therefore, the operation manager does not need to check the usage and operating state of the server to determine whether there is a failure as in the conventional case.

It is a block diagram which shows the structural example of the distributed and parallel processing system which provided the failure monitoring apparatus which concerns on this invention. It is a block diagram which shows the structural example of the failure monitoring apparatus which concerns on this invention. It is explanatory drawing which shows the structural example of the server management table recorded and managed by the configuration management database in FIG. It is a flowchart which shows the 1st processing operation example of the failure monitoring apparatus which concerns on this invention. It is a flowchart which shows the 2nd processing operation example of the failure monitoring apparatus which concerns on this invention. It is explanatory drawing which shows the example of a failure determination operation | movement of the failure determination part in FIG.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In FIG. 1, 101 is a client machine, 102 is a load balancer, 103 and 104 are always-on servers, 105 is a scale-out server, 106 is an automatic scale-out system, 107 is a monitoring system, and 108 is a fault monitoring apparatus of the present invention. 1 is a configuration management system, and 113 is an operation notification system, each of which includes a CPU (Central Processing Unit), a main memory, a display device, an input device, and an external storage. A computer having a computer configuration, etc., and after installing a program or data recorded in a storage medium such as a CD-ROM via an optical disk drive or the like into the external storage device, the CPU reads from the external storage device into the main memory By processing with each The processing unit is executed.

  In addition, the automatic scale-out failure determination system 108 includes a configuration information change function 109 and a failure determination function 110 as functions for executing programmed computer processing. Further, as shown in FIG. 109 includes an automatic scale-out execution content notification receiving unit 201 and a configuration information changing unit 202. The failure determination function 110 includes a monitoring event reception unit 203, a configuration information acquisition unit 204, a failure determination unit 205, and a failure notification unit 206. have.

  With this configuration, the automatic scale-out-compatible failure determination system 108 according to the present invention performs distributed / parallel processing by a plurality of server computers, and also performs distributed scale / in parallel processing systems that perform automatic scale-out and automatic scale-in of server computers. It is possible to efficiently monitor the occurrence of a failure event.

  That is, when the monitoring system 107 detects the occurrence of an event in a server computer such as the always-on servers 103 and 104 or the scale-out server 105, the automatic scale-out failure determination system 108 detects the event detected by the monitoring system 107. It is received, and it is determined as follows whether the event is an event caused by a failure or an event caused by automatic scale-out or automatic scale-in.

  The automatic scale-out failure determination system 108 uses the server usage information collection unit and the configuration information change function 109 as the operation state information collection unit according to the present invention, so that each server such as the always-on servers 103 and 104 and the scale-out server 105 is used. Collects server usage information that indicates whether a computer is a server for automatic scale-out and automatic scale-in, and collects operating status information that indicates whether each server computer is operating or stopped. And stored in the configuration management database 112 via the configuration management system 111.

  Further, the automatic scale-out system 106 receives the automatic scale-out execution content notification in the configuration information change function 109 that automatic scale-out or automatic scale-in of the server computer for automatic scale-out and automatic scale-in has been executed. Upon confirmation by 201, the configuration information changing unit 202 changes the operating state information stored in the configuration management database 112 regarding the server computer via the configuration management system 111.

  For the event detected by the monitoring system 107, the failure determination function 110 receives event information by the monitoring event reception unit 203, and accesses the configuration management database 112 via the configuration management system 111 by the configuration information acquisition unit 204. The server usage information and the operating state information of the server computer that generated the event are read, and the failure determination unit 205 refers to the information to determine whether the event has occurred due to a failure or has occurred due to automatic scale-in. Determine if it is a thing.

  When the failure determination unit 205 determines that the event is caused by a failure, the failure notification unit 206 is activated to notify the operation notification system 113 of the failure event, and the event is automatically scaled. If it is determined that the failure has occurred, the failure notification unit 206 is not activated.

  Hereinafter, a distributed / parallel processing system configuration will be described in detail as an example.

  FIG. 1 shows an example of the overall configuration in the operation business of a business processing computer system including the system according to the present invention. That is, the always-on servers 103 and 104 and the scale-out server 105 are a group of servers that perform business processing and have a distributed / parallel processing system configuration.

  The request transmitted from the client machine 101 is distributed to each server group by the load balancer 102 and transmitted.

  The always-on servers 103 and 104 are servers that are always running and perform business processing. However, the scale-out server 105 is normally stopped, and the always-on servers 103 and 104 are changed due to an increase in the number of requests from the client machine 101. When the load becomes high, the automatic scale-out system 106 starts up and performs business processing (automatic scale-out).

  When the number of requests from the client machine 101 decreases and high performance is not required, the scale-out server 105 is stopped by the automatic scale-out system 106 (automatic scale-in).

  The monitoring system 107 is a system that performs alive monitoring of the always-on servers 103 and 104 and the scale-out server 105.

  Note that the number of client machines 101, load balancers 102, always-on servers 103 and 104, and scale-out servers 105 varies depending on the operation mode.

  When the automatic scale-out system 106 executes automatic scale-out or automatic scale-in, the automatic scale-out server 105 starts or stops the scale-out server 105 after notifying the automatic scale-out failure determination system 108 of the execution contents.

  The execution content to be notified is the server name of the server that is the target of the start or stop operation by scale-out or scale-in, and the operation content indicating whether the operation to be executed is start or stop.

  When the always-on servers 103 and 104 or the scale-out server 105 is started or stopped, the monitoring system 107 detects the event, and a monitoring event is transmitted from the monitoring system 107 to the automatic scale-out failure determination system 108.

  In the automatic scale-out failure determination system 108, the configuration management system 111 manages in order to determine whether the cause of the start or stop is due to a failure or the automatic scale-out or automatic scale-in of the automatic scale-out system 106. The server usage and operating state of the monitoring event generation source server are acquired from the configuration management database 112.

  If it is determined from the server usage and the operating state that the monitoring event is caused by a failure, the failure notification system 113 is notified of the failure. When a failure is notified to the operation notification system 113, the operation manager 114 detects the failure and takes action.

  Next, details of the automatic scale-out failure determination system 108 will be described with reference to FIG.

  As shown in FIG. 2, the automatic scale-out failure determination system 108 has two functions, a configuration information change function 109 and a failure determination function 110. The configuration information change function 109 receives automatic scale-out execution content notifications. The failure determination function 110 includes a monitoring event reception unit 203, a configuration information acquisition unit 204, a failure determination unit 205, and a failure notification unit 206.

  The configuration information change function 109 receives the execution content notified when the automatic scale-out system 106 executes automatic scale-out or automatic scale-in by the automatic scale-out execution content notification receiving unit 201. The contents of the notification are the server name of the server that is the target of the start or stop operation and the operation contents of start or stop.

  Then, the configuration information changing unit 202 receives the configuration information of the operation target server in the configuration management database 112 via the configuration management system 111 according to the content of the notification received by the automatic scale-out execution content notification receiving unit 201. change.

  Note that the configuration management database 112 manages the server name, server usage, and operating state of each server that performs business processing, as indicated by the server management table 301 in FIG.

  The processing operation of the configuration information changing function 109 will be described based on the flowchart shown in FIG. First, a notification of execution contents of automatic scale-out or automatic scale-in from the automatic scale-out system 106 is received (step 401).

  Next, the server name of the operation target server is acquired from the execution content (step 402), and it is determined whether the operation content is start or stop (step 403).

  When the operation content is activated (step 404), the operation state of the operation target server in the server management table 301 in FIG. 3 on the configuration management database 112 is changed to “in operation” (step 405).

  When the operation content is stopped (step 404), the operating state of the corresponding server in the server management table 301 is changed to “standby” (step 406).

  The failure determination function 110 in FIG. 2 receives a monitoring event transmitted from the monitoring system 107, determines whether or not there is a failure, and notifies the operation notification system 113. First, the monitoring event reception unit 203 A monitoring event transmitted from the monitoring system 107 is received. Note that the monitoring event content is the name of the server that detected the start or stop and the detected content (start or stop).

  The failure determination function 110 that has received the monitoring event acquires, from the server management table 301 in FIG. 3 on the configuration database 112, the server usage and operating status of the server that has detected the start or stop by the configuration information acquisition unit 204.

  Furthermore, the failure determination unit 205 determines whether the monitoring event is caused by a failure from the monitoring event content received by the monitoring event reception unit 203 and the server usage and operating state acquired by the configuration information acquisition unit 204, or is automatically scaled. Determine if it was caused by out or automatic scale in.

  When the failure notification unit 206 determines that the monitoring event has occurred due to a failure, the failure determination unit 205 notifies the operation notification system 113 of the failure.

  The processing operation of the failure determination function 110 will be described with reference to the flowchart shown in FIG. First, a monitoring event transmitted from the monitoring system 107 in FIG. 2 is received (step 501).

  Next, from the server management table 301 on the configuration database 112, the server usage and operating status of the server that is the source of the monitoring event is acquired (step 502). It is determined whether or not (step 503).

  At this time, the determination result determined from the monitoring event contents, the server usage, and the operating state is shown in the failure determination table 601 of FIG.

  When it is determined that the monitoring event is a failure (step 504), a failure notification is sent to the operation notification system (step 505), and when it is determined that there is no failure (step 504), no failure notification is performed (step 504). 506).

  As described above with reference to FIGS. 1 to 6, in the fault monitoring apparatus of the distributed / parallel processing system of this example, each server to be monitored always operates or only operates at the time of scale-out. Server usage information indicating whether each server is in standby or operating, and managing and managing server status information indicating whether each server is in standby or operating, and the server usage information and operating status of the server that generated the event By checking the information, it is determined whether the event has occurred due to a failure or whether it has occurred due to automatic scale-out and automatic scale-in. The operating state information indicates whether the scale-out standby server is on standby or “in operation”, and is appropriately changed according to execution of automatic scale-out or automatic scale-in.

  As described above, according to the failure monitoring apparatus and method of the distributed / parallel processing system of this example, the event that occurs when the scale-out server is started and stopped by automatic scale-out and automatic scale-in, and the event that occurs while the business system is running The failure determination can be performed by distinguishing the events to be performed, and this prevents the operation administrator from notifying the normal operation of the standby server for scale-out by automatic scale-out and automatic scale-in as a failure, and is notified as a failure. Since only an abnormal operation to be performed is notified as a failure, the operation manager does not need to check the use and operating state of the server to determine whether or not there is a failure.

  In addition, this invention is not limited to the example demonstrated using FIGS. 1-6, In the range which does not deviate from the summary, various changes are possible. For example, in this example, the automatic scale-out system 106, the monitoring system 107, the automatic scale-out failure determination system 108, the configuration management system 111, and the operation notification system 113 are configured as individual computer devices. Arbitrary combinations may be configured on one computer apparatus.

  Further, regarding the computer configuration example of each system, a computer configuration without a keyboard or optical disk drive may be used. In this example, an optical disk is used as a recording medium. However, an FD (Flexible Disk) or the like may be used as a recording medium. As for the program installation, the program may be downloaded and installed via a network via a communication device.

  101: client machine, 102: load balancer, 103: always-on server, 104: always-on server, 105: scale-out server, 106: automatic scale-out system, 107: monitoring system, 108: automatic scale-out failure determination System: 109: Configuration information change function, 110: Fault determination function, 111: Configuration management system, 112: Configuration management database, 113: Fault notification system, 114: Operation manager, 201: Receiving part of automatic scale-out execution content notification 202: configuration information change unit, 203: monitoring event reception unit, 204: configuration information acquisition unit, 205: failure determination unit, 206: failure notification unit, 301: server management table, 601: failure determination table.

Claims (3)

A device that monitors the occurrence of a failure event in a distributed / parallel processing system that performs distributed / parallel processing by a plurality of server computers, and performs automatic scale-out and automatic scale-in of server computers,
Monitoring means for detecting the occurrence of an event in the server computer;
Receiving an event detected by the monitoring means, and determining means for determining whether the event is an event caused by a failure or an event caused by automatic scale-out or automatic scale-in,
The determination means includes
Server usage information collection means for collecting server usage information indicating whether each server computer is a server for the automatic scale-out and the automatic scale-in and storing it in a storage device;
Operating state information collecting means for collecting operating state information indicating whether each server computer is operating or stopped and storing it in a storage device;
Changing means for changing the operating state information relating to the server computer when automatic scale-out or automatic scale-in of the server computer for automatic scale-out and automatic scale-in is executed,
A failure notification means for notifying a failure;
Regarding the event detected by the monitoring means, referring to the server usage information and the operating state information of the server computer that generated the event, the event occurred due to a failure or occurred due to automatic scale-in If the event is determined to be caused by a failure, the failure notification means is activated to notify the failure, and the event is caused by automatic scale-in. A failure monitoring device for a distributed / parallel processing system, comprising: failure determination means that does not activate the failure notification means .   The program for functioning a computer as each means in the failure monitoring apparatus of the distributed / parallel processing system according to claim 1. A failure monitoring method for a device that monitors the occurrence of a failure event in a distributed / parallel processing system that performs distributed / parallel processing by a plurality of server computers and performs automatic scale-out and automatic scale-in of server computers by programmed computer processing Because
As means for executing programmed computer processing, it has monitoring means, server usage information collection means, operating state information collection means, change means, failure determination means , failure notification means ,
The monitoring means detects the occurrence of an event in the server computer,
The server usage information collection means collects server usage information indicating whether each server computer is a server for the automatic scale-out and the automatic scale-in, stores the server usage information in a storage device,
The operating state information collecting means collects operating state information indicating whether each server computer is operating or stopped, and stores it in a storage device.
The change means changes the operating state information related to the server computer when the automatic scale-out or the automatic scale-in of the server computer for the automatic scale-in is executed.
The failure notification means notifies a failure,
For the event detected by the monitoring means, the failure determination means refers to the server usage information and the operating state information of the server computer that generated the event, and automatically determines whether the event has occurred due to a failure. If it is determined that the event has occurred due to scale-in, and it is determined that the event has occurred due to a failure, the failure notification means is activated to notify the failure, and the event is automatically scaled in. The failure monitoring method for a distributed / parallel processing system, wherein the failure notification unit is not activated when it is determined that the failure has occurred .

Images