JP6073211B2 - Server monitoring method and server monitoring system - Google Patents

Description

  The present invention relates to a server monitoring method and a server monitoring system, and more specifically, to a technology that can accurately detect a non-response failure in a server system having a cluster configuration.

  In view of its importance, a computer system operated by a financial institution is often a mission critical system that does not allow an easy system down. Therefore, such a computer system is required to have a high level of excellent fault tolerance and quick and accurate failure response. Thus, various techniques for monitoring the occurrence of a failure and promptly responding thereafter have been proposed.

  As such a technique, for example, the following has been proposed. That is, an interface between a status database, a status collector that establishes a first connection with a monitored device, receives a status item from the monitored device, and stores the status item in the status database, and a remote device And a command dispatcher that establishes a third connection with the monitored device and transmits a command to the monitored device (see Patent Document 1).

Special table 2010-537563

  On the other hand, in the computer system as described above, there are many cases in which a non-response failure (hang or slowdown) cannot be detected by constructing a working / standby configuration of various servers using an HA (High Availability) cluster or the like. . Under such circumstances, the switching operation to the standby node accompanying the detection of the failure occurrence is not executed and the service is often stopped as it is. Concerns about service interruptions in mission-critical computer systems are a major problem, and it is necessary to prepare an environment that can accurately detect non-response failures in the system and quickly perform recovery work after the failure occurs. .

  SUMMARY OF THE INVENTION An object of the present invention is to provide a technique capable of accurately detecting a non-response failure in a server system having a cluster configuration.

  In the server monitoring method of the present invention that solves the above problem, in a first server included in a server system having a cluster configuration, a message requesting a predetermined response is transmitted to a second server accessed by the first server. A process of measuring an elapsed time from the time of transmission of the electronic message, and a first time when an elapsed time until receiving a predetermined response from the second server in response to the transmission of the electronic message exceeds a first reference time A process of monitoring an event for each transmission of the message, and notifying a predetermined terminal that the second server is in a slowdown state when the number of consecutive occurrences of the first event exceeds a predetermined reference number; A second event in which a predetermined response cannot be received from the second server within a second reference time that is equal to or greater than the first reference time from the transmission time of the message is monitored for each transmission of the message. The occurrence of the second event When knowledge, the second server and the client performs a process of notifying the predetermined terminal to the effect that a hung up state.

  In the server monitoring system of the present invention, in the first server included in the clustered server system, a message requesting a predetermined response is transmitted to the second server accessed by the first server, and the message A process of measuring an elapsed time from the transmission time of the first event, and a first event in which an elapsed time until receiving a predetermined response from the second server in response to the transmission of the message exceeds a first reference time, A process of monitoring each time a message is transmitted, and notifying a predetermined terminal that the second server is in a slowdown state when the number of consecutive occurrences of the first event exceeds a predetermined reference number; The second event in which a predetermined response has not been received from the second server within a second reference time that is equal to or greater than the first reference time from the time of transmission is monitored for each transmission of the message. Detected the occurrence of two events The case is configured to include an arithmetic unit for executing processing for notifying that the second server is hung up state to a predetermined terminal, and wherein the.

  According to the present invention, it is possible to accurately detect a non-response failure in a server system having a cluster configuration.

It is a network block diagram containing the server monitoring system of this embodiment. It is a figure which shows the hardware structural example of AP server in this embodiment. It is a figure which shows the hardware structural example of DB server in this embodiment. It is a figure which shows the hardware structural example of the monitoring console in this embodiment. It is a figure which shows the structural example of the table for health checks of this embodiment. It is a flowchart which shows the process sequence example of the server monitoring method in this embodiment. It is explanatory drawing which shows the example of a monitoring concept in this embodiment.

--- System configuration ---
Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a network configuration diagram including a server monitoring system 10 of the present embodiment. A server monitoring system 10 illustrated in FIG. 1 is a computer system that can accurately detect a non-response failure in a server system 20 having a cluster configuration. As a cluster system server system 20 assumed in the present embodiment, a backbone system operated in a financial institution is assumed as an example. Therefore, the server monitoring system 10 according to the present embodiment is a system that monitors a non-response failure in a backbone system in a financial institution. Of course, the non-response failure monitoring target in the server monitoring system 10 is not limited to a system in a financial institution, but can be applied to various server systems (cluster configurations) in other industries.

  A server system 20 illustrated in FIG. 1 is connected via a firewall 21 to an external network 120 connected to a customer terminal of a financial institution. In addition, the server system 20 distributes the processing request received from the customer terminal through the firewall 21 to the AP server 100 by the load balancer 300. The cluster configuration of the server device in the server system 20 is associated with the two AP servers 100A and 100B, which are processing request distribution destinations by the load distribution device 300, and the AP servers 100A and 100B. One DB server 200 that is a target of data acquisition and writing by 100B and the like is configured to have two systems, an active system and a standby system. As the standby AP server 100, AP servers 100C and 100D are also shown in FIG. 1 for reference.

  Note that the two AP servers 100A and 100B in FIG. 1 correspond to the first server in the present invention. In the following description, these two AP servers 100A and 100B are not distinguished from each other, and are described as the AP server 100 when the description is made as the first server. The DB server 200 corresponds to the second server in the present invention. In the present embodiment, a configuration in which two AP servers 100A and 100B are associated with one DB server 200 is illustrated. However, the present invention is not limited to this configuration, and there are three for one DB server 200. Even if the AP servers 100A and 100B are associated with each other, the server monitoring method of the present embodiment can be applied without any particular change.

  The AP consoles 100A and 100B are connected to a monitoring console 400 used by a system person in charge of a financial institution, and the processing result of the server monitoring method in the present embodiment is output to the monitoring console 400. .

  Next, an example of the hardware configuration of each device constituting the server monitoring system 10 of the present embodiment will be described. FIG. 2 is a diagram illustrating a hardware configuration example of the AP server 100 according to the present embodiment. The hardware configuration of the AP server 100 described above is as follows. The AP server 100 reads out a storage device 101 composed of a suitable non-volatile storage device such as a hard disk drive, a memory 103 composed of a volatile storage device such as a RAM, and a program 102 held in the storage device 101 to the memory 103. Are executed to perform overall control of the device itself and perform various determinations, computations, and control processes, and the like, and a communication process with other devices (the load distribution device 300 and the DB server 200) connected to the internal network 130. The communication device 105 is responsible for.

  In addition, in the storage device 101, there is an OS (not shown) that is naturally provided as an information processing device, and a program 102 that implements functions necessary for the AP server 100 in the server monitoring system 10 of the present embodiment. At least remembered. In the present embodiment, the health check shell 110 is implemented when the AP server 100 executes the program 102. The health check shell 110 is a shell that resides in each of the AP servers 100A and 100B and realizes the main functions of the server monitoring system 10 in the present embodiment. The health check shell 110 issues a very simple SQL statement that requests only a simple value according to the resident AP server 100, and in response to the response result obtained from the DB server 200, a predetermined result on the AP server 100 is displayed. Monitoring application 150 (existing one that collects application and system information comprehensively using SNMP network monitoring information), and the monitoring application 400 receives no response failure via the monitoring application 150. It will be to present information about.

  Next, a hardware configuration example in the DB server 200 will be described. FIG. 3 is a diagram illustrating a hardware configuration example of the DB server 200 in the present embodiment. Similar to the AP server 100 described above, the DB server 200 has a configuration as a general server device, and holds a health check table 225 shared among the DB servers 200 in the storage device 201. The health check table 225 is a table in which predetermined values are associated with the AP server 100. Details of the health check table 225 will be described later.

  Next, a hardware configuration example of the monitoring console 400 will be described. FIG. 4 is a diagram illustrating a hardware configuration example of the monitoring console 400 according to the present embodiment. The monitoring console 400 accepts key input and voice input from a predetermined user such as a person in charge of a system in a financial institution in addition to a hardware configuration as a general information processing apparatus in the AP server 100 and the DB server 200 described above. An input device 405 and an output device 406 such as a display for displaying a notification from the health check shell 110 of the AP server 100 are provided.

  Then, the function with which the server monitoring system 10 of this embodiment is provided is demonstrated. As described above, the functions described below can be said to be functions that are implemented by executing a program included in an appropriate information processing apparatus constituting the server monitoring system 10, for example. In the case of this embodiment, for example, the health check shell 110 in the AP server 100 takes the lead, and the DB server 200 and the monitoring console 400 cooperate with each other. When only the AP server 100 is assumed as the server monitoring system 10, the function of the health check shell 110 becomes the function of the server monitoring system 10. As the server monitoring system 10, in addition to the AP server 100, the DB server 200 and the monitoring console When a configuration including 400 is assumed, the functions of the health check shell 110, the DB server 200, and the monitoring console 400 become functions of the server monitoring system 10.

  Here, the health check shell 110 resident in each AP server 100 is in contrast to the DB server 200 accessed by the resident destination AP server 100 (which is simply the DB server 200 to which the resident destination AP server 100 can communicate). For example, a message requesting a predetermined response is transmitted at intervals of 60 seconds, for example, and a function of measuring an elapsed time from the transmission time of each message is provided.

  In addition, each health check shell 110 described above is a first event in which the elapsed time until a predetermined response is received from the DB server 200 in response to the transmission of the above-described message exceeds, for example, 2 seconds (first reference time). Is monitored every time the above-mentioned message is transmitted, and the monitoring console indicates that the corresponding DB server 200 is in a slow-down state when the number of consecutive occurrences of the first event exceeds, for example, two times (predetermined reference number). 400 has a function of notifying the user.

  In addition, each health check shell 110 indicates a second event in which a predetermined response has not been received from the DB server 200 within 35 seconds (second reference time), for example, from the transmission time of the above-described message. It has a function of monitoring each transmission and notifying the monitoring console 400 that the corresponding DB server 200 is in a hang-up state when the occurrence of the second event is detected.

  In addition, each health check shell 110 requests a message to respond to a predetermined value corresponding to the AP server 100 in which the health check shell 110 resides as a message to be transmitted to the above-described DB server 200 at intervals of 60 seconds, for example. The function to transmit. This message preferably includes only the ID (identification information) of the corresponding AP server 100 in a format corresponding to the communication protocol between the servers. Here, the predetermined value for requesting a response is a single-digit numerical value determined in advance for each AP server 100A, 100B.

  In this case, the DB server 200 defines the correspondence between the IDs of the AP servers 100A and 100B and the single-digit numerical values indicating the corresponding AP servers 100A and 100B in the health check table 225 in the storage device 201. (See FIG. 5). Therefore, when the DB server 200 receives the above-described message transmitted from the health check shell 110, the DB server 200 uses the health check to identify the identification information of the corresponding AP servers 100A and 100B (the resident location of the health check shell 110) indicated by the message. And a function of identifying a single digit value corresponding to the AP server 100 where the health check shell 110 is resident and returning this to the health check shell 110 of the corresponding AP server 100 as a response. ing.

  On the other hand, the monitoring console 400 that receives the notification regarding the slow-down state or the hang-up state from the health check shell 110 of the AP server 100 receives the slow-down state or the state from each health check shell 110, that is, each of the AP servers 100A and 100B. When the notification about the hang-up state is received, the health check shell 110 of all the AP servers 100A and 100B associated with the corresponding DB server 200 for one DB server 200 is in a slow-down state or hang-up state. Is received, a function for displaying on the output device 406 that the DB server 200 is in a slow-down state or a hang-up state is provided.

  Further, the monitoring console 400 indicates that only one AP server 100A or AP server 100B associated with the corresponding DB server 200 is in a slow-down state or a hang-up state with respect to one DB server 200. When the notification is received, the output device 406 has a function of displaying that the AP server 100 that has not transmitted the notification is in a slowdown state or a hangup state.

  In addition, when the monitoring console 400 receives a notification indicating that it is in a slowdown state or a hangup state from all the AP servers 100 associated with the corresponding DB server 200 with respect to one DB server 200, The DB server 200 has a function of displaying on the output device 406 that a slow down state or a hang-up state has occurred and a message prompting the user to execute a login process to the DB server 200 to check the state.

--- Processing procedure example ---
Hereinafter, the actual procedure of the server monitoring method in the present embodiment will be described with reference to the drawings. Various operations corresponding to the server monitoring method described below are realized by each health check shell 110 in each AP server 100A, 100B included in the server monitoring system 10. The health check shell 110 is composed of codes for performing various operations described below.

  FIG. 6 is a flowchart illustrating an example of a processing procedure of the server monitoring method in the present embodiment, and FIG. 7 is an explanatory diagram illustrating an example of a monitoring concept in the present embodiment. Here, first, the health check shell 110 resident in each AP server 100 is connected to the DB server 200 accessed by the resident AP server 100 (which is simply the DB server 200 to which the resident AP server 100 can communicate). Then, a message requesting a predetermined response is transmitted at intervals of, for example, 60 seconds, and the elapsed time from the transmission time of each message is measured (s100). The above-mentioned message contains only the ID (identification information) of the AP server 100 where the corresponding health check shell 110 resides in a format corresponding to the communication protocol between the servers, and the AP server where the corresponding health check shell 110 resides. This is a message requesting to respond only to a single digit value corresponding to 100.

  Next, each health check shell 110 described above has elapsed time from receiving a one-digit numerical value (predetermined response) corresponding to the resident AP server 100 from the DB server 200 in response to the transmission of the above-described message. For example, the first event exceeding 2 seconds is monitored every time the above-mentioned message is transmitted, and the number of occurrences is counted (s101). As a result of this count, when the number of consecutive occurrences of the first event reaches, for example, more than two and reaches three (s102: Y), the corresponding health check shell 110 indicates that the corresponding DB server 200 is in a slow-down state. The monitoring console 400 is notified of this fact (s103). Although the first event is counted up to, for example, twice consecutively according to the above-described count, the health check shell 110 counts up to that time when a response within 2 seconds is received from the DB server 200 in the next message transmission, for example. The count value of “2 times” (held in the memory 103 of the AP server 100 or the like) is cleared, and the continuous occurrence count of the first event is started again from the time when the consecutive count is “0”.

  In addition, the health check shell 110 detects the second event in which the above-described one-digit numerical value has not been received from the corresponding DB server 200 within, for example, 35 seconds from the transmission time of the above-described message, for each transmission of the above-described message. When the occurrence of the second event is detected (s104: Y), the monitoring console 400 is notified that the corresponding DB server 200 is in the hang-up state (s105). The monitoring concept in this embodiment is as shown in FIG.

  On the other hand, the monitoring console 400 receives the notification regarding the slowdown state or the hangup state from the health check shell 110 of the AP server 100 (s106), and the received notification is either the slowdown state or the hangup state. Is determined (s107). When the received notification is related to the slowdown state (s107: slowdown), the monitoring console 400 further relates all the AP servers 100 associated with the corresponding DB server 200 with respect to one DB server 200, that is, APs. It is determined whether a notification indicating the slowdown state has been received from the health check shells 110 of the server 100A and the AP server 100B (s108).

  The monitoring console 400 determines that a notification indicating that it is in a slowdown state has been received from all AP servers 100 associated with one DB server 200, that is, the health check shells 110 of the AP server 100A and the AP server 100B. In the case (s108: all), the output device 406 displays that the DB server 200 is in a slowdown state (s109).

  On the other hand, when it is determined that only some of the AP servers 100 associated with one DB server 200, for example, only the health check shell 110 of the AP server 100A has received a notification indicating that it is in a slowdown state. (S108: Part), a message indicating that a slow-down state has occurred in the corresponding AP server 100A is displayed on the output device 406 (s110).

  If the received notification is related to the hang-up state in step s107 described above (s107: hang-up), the monitoring console 400 is further associated with the corresponding DB server 200 with respect to one DB server 200. It is determined whether a notification indicating that the state is in the hang-up state has been received from each of the health check shells 110 of all the AP servers 100, that is, the AP server 100A and the AP server 100B (s111).

  The monitoring console 400 receives a notification that only a part of the AP servers 100 associated with one DB server 200 is in a hang-up state, for example, only from the health check shell 110 of the AP server 100A. If it is determined (s111: part), the output device 406 displays that the corresponding AP server 100A is hung up (s112).

  On the other hand, when one of the health check shells 110 of each AP server 100 associated with the corresponding DB server 200, that is, the AP server 100A and the AP server 100B, has been notified that it is in a hang-up state. When the determination is made (s111: all), the monitoring console 400 displays a message prompting the user to check the status by executing the login process to the corresponding DB server 200 (s113), and ends the process.

  If such a process is performed by the monitoring console 400, it is possible to identify the failure occurrence source in which server in the cluster-structured server system 20 and present it to the user. Since this user can surely recognize the failure source and the failure content, it becomes easy to execute the failure response to be performed subsequently with high accuracy. In addition, when the DB server 200 hangs up, basically, it is determined which DB controller in the corresponding server is suspected of failure by confirming whether or not the login process is possible. Can be encouraged. Therefore, the user specifically recognizes the failure response to be performed and easily executes it quickly.

  Although the best mode for carrying out the present invention has been specifically described above, the present invention is not limited to this, and various modifications can be made without departing from the scope of the invention.

  According to this embodiment, it becomes possible to accurately detect a non-response failure in a server system having a cluster configuration.

  At least the following will be clarified by the description of the present specification. That is, in the first server of the server monitoring system of the present embodiment, as the message for the second server, a message requesting to respond with a predetermined value corresponding to the first server is transmitted, and the first server In the second server, a table in which a predetermined value is associated with the first server is held in a storage device, and the first server indicated by the corresponding message when the message is received from the first server The identification information may be collated with the table, a predetermined value corresponding to the first server may be specified, and a process of returning the predetermined value as a response to the first server may be executed.

  According to this, when monitoring unresponsive failures such as slowdowns and hangups, it is often not necessary to take part in the excessive amount of processing data that causes such failures, but simply from a table with a fixed and simple data structure. Server processing can be performed quickly and accurately with extremely low-load processing, such as obtaining and returning a single message.

  In the server system having the cluster configuration, it is assumed that a plurality of the first servers are associated with each second server. Each of the first servers in the server monitoring system in this case transmits the message to the second server associated with the first server itself, and in the second server, the first server The message is received from each of the servers, the identification information of the corresponding first server indicated by the corresponding message is collated with the table, a predetermined value corresponding to the corresponding first server is specified, and the predetermined value is used as a response. The first server sends back to the corresponding first server, and each of the first servers, when the number of consecutive occurrences of the first event exceeds a predetermined reference number, the second server associated with the first server itself Is notified to the predetermined terminal that it is in a slow-down state, and the second server is associated with the first server itself within a second reference time that is equal to or greater than the first reference time from the time of transmission of the message. Predetermined from 2 servers The second event for which an answer could not be received is monitored each time the message is transmitted, and when the occurrence of the second event is detected, the second server associated with the first server itself hangs The predetermined terminal is notified of being in the up state, and the predetermined terminal receives a notification from the first server that the second server is in the slow-down state or the hang-up state. When notification of the slowdown state or the hangup state is received from all the first servers associated with the corresponding second server with respect to one second server , A message indicating that a slowdown state or a hang-up state has occurred in the second server is displayed on the output device, and one second server is associated with the corresponding second server. When a notification indicating that it is in the slowdown state or in the hangup state is received only from some of the first servers, a slowdown state is sent to the first server that has not transmitted the notification. Or it is good also as displaying on the output device that the said hang-up state has arisen.

  According to this, in the server system with the cluster configuration, it is possible to identify the failure occurrence source in which server and identify the failure occurrence source and present it to the user side (predetermined terminal). Since the user can surely recognize the source of the failure and the content of the failure, it becomes easy to perform the failure response to be performed subsequently with high accuracy and speed.

  In addition, the predetermined terminal in the above-described case receives a notification that it is in the hang-up state from all the first servers associated with the corresponding second server with respect to one second server. In such a case, the output device may display a message that the second server is in the hang-up state and a message prompting the user to execute a login process to the second server to check the state. .

  When a hang-up occurs in a DB server (second server), you can basically determine which DB controller in the server is suspected of failure by checking whether login processing is possible. Can prompt the user. Therefore, the user specifically recognizes the failure response to be performed and easily executes it quickly.

10 server monitoring system 20 cluster system server system 100 AP server (first server)
101 storage device 102 program 103 memory 104 arithmetic device 105 communication device 110 health check shell 120 network 200 DB server (second server)
225 Health check table
300 Load balancer 400 Monitoring console (predetermined terminal)

Claims (5)

In the first server included in the clustered server system,
A process of transmitting a message requesting a predetermined response to the second server accessed by the first server, and measuring an elapsed time from the transmission time of the message;
A first event in which an elapsed time until receiving a predetermined response from the second server in response to the transmission of the message exceeds a first reference time is monitored for each transmission of the message, and the first event A process of notifying a predetermined terminal that the second server is in a slow-down state when the number of consecutive occurrences exceeds a predetermined reference number;
A second event in which a predetermined response cannot be received from the second server within a second reference time that is equal to or greater than the first reference time from the transmission time of the message is monitored for each transmission of the message. A process for notifying a predetermined terminal that the second server is in a hang-up state when the occurrence of the second event is detected;
The server monitoring method characterized by performing. In the first server,
As a message to the second server, a message requesting to respond with a predetermined value corresponding to the first server is transmitted,
In the second server,
A table in which a predetermined value is associated with the first server is held in a storage device;
When the electronic message is received from the first server, the identification information of the first server indicated by the electronic message is checked against the table, a predetermined value corresponding to the first server is specified, and the predetermined value A process of returning the response to the first server as a response,
The server monitoring method according to claim 1, wherein: In the server system having the cluster configuration, a plurality of the first servers are associated with each second server.
Each of the first servers
The telegram is transmitted to the second server associated with the first server itself,
In the second server,
The message is received from each of the first servers, the identification information of the corresponding first server indicated by the corresponding message is checked against the table, a predetermined value corresponding to the corresponding first server is specified, and the predetermined value As a response to the corresponding first server,
Each of the first servers
If the number of consecutive occurrences of the first event exceeds a predetermined reference number, notify the predetermined terminal that the second server associated with the first server is in a slowdown state,
A second event in which a predetermined response cannot be received from the second server associated with the first server within a second reference time that is equal to or greater than the first reference time from the time of transmission of the message. To the predetermined terminal that the second server associated with the first server itself is in a hang-up state when the occurrence of the second event is detected. Notify
In the predetermined terminal,
Receiving a notification from the first server that the server is in the slowdown state or the hangup state with respect to the second server,
When all the first servers associated with the second server with respect to one second server receive notification that the slow server is in the slowdown state or the hang-up state, Display on the output device that the second server is in a slowdown state or a hangup state,
Only one of the first servers associated with the second server with respect to one second server has been notified that it is in the slow-down state or the hang-up state. A message indicating that a slowdown state or a hangup state has occurred in the first server that has not transmitted the notification on the output device;
The server monitoring method according to claim 2, wherein: In the predetermined terminal,
When the notification of being in the hang-up state is received from all of the first servers associated with the corresponding second server with respect to one second server, the hang-up is sent to the second server. A message indicating that an up state has occurred and a message prompting the user to perform status check by executing login processing to the second server are displayed on the output device;
The server monitoring method according to claim 3. In the first server included in the clustered server system,
A process of transmitting a message requesting a predetermined response to the second server accessed by the first server, and measuring an elapsed time from the transmission time of the message;
A first event in which an elapsed time until receiving a predetermined response from the second server in response to the transmission of the message exceeds a first reference time is monitored for each transmission of the message, and the first event A process of notifying a predetermined terminal that the second server is in a slow-down state when the number of consecutive occurrences exceeds a predetermined reference number;
A second event in which a predetermined response cannot be received from the second server within a second reference time that is equal to or greater than the first reference time from the transmission time of the message is monitored for each transmission of the message. When the occurrence of the second event is detected, a configuration is provided that includes an arithmetic device that executes processing for notifying a predetermined terminal that the second server is in a hang-up state.
A server monitoring system characterized by that.