JP7585659B2 - Monitoring system, monitoring method, program, and fault-tolerant server - Google Patents

Description

This disclosure relates to a monitoring system, etc.

One method for identifying hard disks that make up a RAID (Redundant Arrays of Inexpensive Disks) system and that have not yet failed but have degraded performance is to measure the response time of the hard disk. With this method, a hard disk whose response time is equal to or exceeds a predetermined threshold is determined to have degraded performance. However, a response time exceeding the threshold may be caused by accidental factors such as temporary load. Therefore, even when there is actually no problem with the hard disk, it may be determined to be abnormal, leading to unnecessary disk replacement.

Patent document 1 discloses a method for detecting storage devices whose disk load is below a threshold and whose response time is above a threshold in order to discover storage devices in a potentially faulty state in a storage device. The method in patent document 1 determines that a storage device may be faulty even if the response time exceeds the threshold due to an accidental factor other than the disk load.

After detecting an abnormal trend in a hard disk, a diagnosis of the hard disk may be performed to confirm the abnormal state. For example, one method is to perform similar measurements multiple times and confirm that the performance of only a specific hard disk has decreased in all measurements, and then diagnose that the hard disk is abnormal. Another method is to compare average response times and diagnose that a hard disk whose average response time exceeds a threshold is abnormal.

Patent document 2 discloses that when a disk error such as a timeout occurs, the disk is put into a provisionally degraded state and four types of disk diagnostic processing are performed.

JP 2019-036163 A JP 2002-108573 A

Because hard disk diagnostic processing and business processing are performed in parallel, performing hard disk diagnostic processing without disconnecting the hard disk can lead to a decrease in performance of the entire system, which could cause problems with business processing. Furthermore, repeating diagnostic input/output and measuring response time for a hard disk whose performance may be degraded places an additional load on the system. In other words, the more detailed the hard disk diagnosis, the greater the load on the system, which could cause problems with business processing.

The diagnostic process in Patent Document 2 is performed in parallel with commands from the host computer, so it must be performed in parallel with business processing on the same disk array device. Therefore, the diagnostic method in Patent Document 2 may affect the performance of business processing due to the diagnostic process.

One of the objectives of this disclosure is to provide a monitoring system for a storage device that avoids impacts on business processing.

The monitoring system according to the present disclosure comprises a processing means for issuing an IO (Input Output) request to each of the storage devices included in the first and second synchronized subsystems, a determination means for determining an abnormal tendency of the storage device having a longer response time based on the difference in response times to the IO requests, and a synchronization control means for releasing the synchronization of the first and second subsystems and controlling the first and second subsystems to operate independently when an abnormal tendency is determined, and the processing means performs a diagnostic process for determining whether or not the storage device has an abnormality in the subsystem including the storage device determined to have an abnormal tendency.

The monitoring method according to the present disclosure issues an IO (Input Output) request to each of the storage devices included in the first and second synchronized subsystems, and determines whether the storage device with the longer response time is prone to an abnormality based on the difference in response time to the IO request. If an abnormality is determined, the synchronization of the first and second subsystems is released, and the first and second subsystems are controlled so that they can operate independently. In the subsystem including the storage device determined to be prone to an abnormality, a diagnostic process is performed to determine whether the storage device has an abnormality.

The program disclosed herein causes a computer to execute the following processes: issuing an IO (Input Output) request to each of the storage devices included in the synchronized first and second subsystems; judging an abnormal tendency of the storage device with the longer response time based on the difference in response time to the IO request; when an abnormal tendency is judged, releasing the synchronization of the first and second subsystems and controlling the first and second subsystems so that they can operate independently; and diagnosing the presence or absence of an abnormality in the storage device in the subsystem that includes the storage device judged to have an abnormal tendency.

The fault-tolerant server according to the present disclosure comprises a first subsystem having a first storage device, a first CPU module, and a first fault-tolerant (FT) controller, and a second subsystem having a second storage device, a second CPU module, and a second FT controller, and the first and second storage devices each respond to an IO (Input Output) request issued by a monitoring system, and when a tendency for the first storage device to be abnormal is determined based on the difference in response time to the IO request, the first FT controller controls the desynchronization of the first and second subsystems, diagnoses the abnormality of the first storage device in the first subsystem, and performs other processing in the second subsystem.

This disclosure makes it possible to provide a storage device monitoring system that avoids impacts on business processing.

2 is a diagram showing a configuration of an FT server 1 according to the present disclosure. FIG. 1 is a diagram showing the FT server 1 in a synchronized state. FIG. 13 is a diagram showing the FT server 1 in a released synchronization state. 1 is a block diagram illustrating a configuration of a monitoring system 50 according to a first embodiment. 4 is a flowchart showing an example of the operation of the monitoring system 50 according to the first embodiment. 5 is a flowchart showing an abnormal trend detection process according to the first embodiment. 11 is a flowchart showing the operations of the system 10 and the system 20 when releasing synchronization. 4 is a flowchart showing a diagnosis process according to the first embodiment. FIG. 11 is a block diagram illustrating a configuration of a monitoring system 50 according to a second embodiment. 10 is a flowchart showing an abnormal trend detection process according to the second embodiment. 10 is a flowchart showing a diagnosis process according to a second embodiment. 11 is a flowchart showing the operations of the system 10 and the system 20 when releasing synchronization. FIG. 2 is a block diagram showing an example of a hardware configuration of a monitoring system 50.

In mission-critical systems, the continuity of service is required even when a failure occurs, so fault-tolerant technology is introduced. A fault-tolerant server (FT server) is known as a computer that employs fault-tolerant technology.

The hardware components that make up the FT server are duplicated. If a failure occurs in one of the hardware components, the part where the failure occurred is logically isolated. The part that is operating normally continues processing, so duplication improves fault tolerance.

The monitoring system according to the present disclosure can be used, for example, to monitor the hard disk of an FT server. FIG. 1 is a diagram showing the configuration of an FT server 1 according to the present disclosure. The FT server 1 according to the present disclosure has two subsystems, system 10 and system 20.

The system 10 has a CPU module 11, an FT controller 12, and an IO module 13. The CPU module 11 has a CPU (Central Processing Unit) and memory. The IO module 13 has each IO (Input Output) device, including a NIC (Network Interface Card) and a hard disk 14.

The system 20 has a CPU module 21, an FT controller 22, and an IO module 23. The CPU module 21 has a CPU and memory. The IO module 23 has each IO device including a NIC and a hard disk 24.

The CPU module 11 and the CPU module 21 are controlled by the FT controller 12 and the FT controller 22 so that they operate synchronously with the same clock. Each IO device of the IO module 13 and the IO module 23 realizes a redundant configuration by software. The NIC realizes a redundant configuration by using teaming technology, and the hard disk realizes a redundant configuration by using mirroring technology.

Figure 2 is a diagram showing FT server 1 in a synchronized state. As shown in Figure 2, when FT server 1 is operating in a synchronized state, CPU module 11 and CPU module 21 are controlled by FT controller 12 and FT controller 22 so that they operate synchronously with the same clock. An OS (Operating System) 30 runs on FT server 1, and business processing is performed on OS 30. If a hardware failure occurs in one of the systems, the part where the failure occurred is logically separated, allowing the system operating normally to continue processing.

If FT Server 1 is operating in a synchronized state without any hardware failure, it is possible to split FT Server 1 into two systems that operate independently by intentionally temporarily disabling the synchronization of FT Server 1.

Figure 3 is a diagram showing the FT server 1 in a state where synchronization is released. As shown in Figure 3, when synchronization is intentionally released, the CPU module 11 operates in the system 10, and IO processing corresponding to each IO device of the IO module 13 is performed. The OS 31 runs on the system 10. The CPU module 21 operates in the system 20, and IO processing corresponding to each IO device of the IO module 23 is performed. The OS 32 runs on the system 20.

[First embodiment]
[composition]
Fig. 4A is a block diagram illustrating a configuration of a monitoring system 50 according to the first embodiment. The monitoring system 50 is connected, for example, by wire or wirelessly to the FT server 1 shown in Fig. 1, and monitors the hard disks 14 and 24 of the FT server 1. The monitoring system 50 includes a processing unit 51, a recording unit (not shown), a determination unit 53, and a synchronization control unit 54. The processing unit 51, the determination unit 53, and the synchronization control unit 54 are embodiments of the processing means, the determination means, and the synchronization control means according to the present disclosure, respectively. The hard disks 14 and 24 are embodiments of the storage devices according to the present disclosure, respectively.

The processing unit 51 issues IO requests to the storage devices of the two synchronized subsystems (hereinafter, this may simply be referred to as issuing IOs or issuing IOs). Specifically, the processing unit 51 periodically issues IOs to, for example, the hard disk 14 and the hard disk 24. The processing unit 51 issues IOs to measure the response time from when the IO request is sent until when a response to the IO request is received. The processing unit 51 then records the measured response time in the recording unit. Furthermore, the processing unit 51 performs a diagnostic process to diagnose the presence or absence of an abnormality in the storage device in the subsystem that includes the storage device that has been determined to have an abnormal tendency.

The recording unit is, for example, an auxiliary storage device such as a RAM (Random Access Memory). The recording unit may also be realized by a storage device such as a hard disk.

The determination unit 53 determines whether the storage device with the longer response time is prone to an abnormality based on the difference in response times to IO requests. Specifically, the determination unit 53 monitors the response times of each hard disk recorded in the recording unit, for example, and determines whether the difference in response times is equal to or greater than a threshold. If the difference in response times exceeds the threshold, the determination unit 53 sends a notification to the synchronization control unit 54 that desynchronization of the FT server 1 is necessary, along with information on which hard disk is prone to an abnormality. Furthermore, the determination unit 53 makes a determination to determine whether the hard disk in question is prone to an abnormality that reduces performance, based on the results of the hard disk diagnostic process.

When the synchronization control unit 54 determines that one of the storage devices is showing an abnormal trend, it releases the synchronization of the two subsystems and controls so that systems 10 and 20 can operate independently. Specifically, the synchronization control unit 54 controls the synchronization of the FT server 1, for example, based on a notification from the determination unit 53. When releasing the synchronization, it releases the synchronization with the system on the hard disk side that is not showing an abnormal trend (short response time) as the business continuation side, and the system on the hard disk side that is showing an abnormal trend (long response time) as the hard disk diagnosis side. Note that the synchronization of the FT server 1 is controlled via the FT controller 12 and the FT controller 22.

[Action]
4B is a flowchart showing an example of the operation of the monitoring system 50 according to the first embodiment. First, the processing unit 51 issues an IO request to each of the storage devices included in the two synchronized subsystems, the system 10 and the system 20 (step S101). The determination unit 53 determines the abnormal tendency of the storage device with the longer response time based on the difference in response time to the IO request (step S102). When the synchronization control unit 54 determines the abnormal tendency of the storage device, it releases the synchronization between the system 10 and the system 20 and controls the two subsystems so that they can operate independently (step S103). The processing unit 51 diagnoses the presence or absence of an abnormality in the storage device in the subsystem including the storage device determined to have an abnormal tendency (step S104).

A specific example of the processing flow according to the first embodiment will be described with reference to Figures 5 to 7.

Figure 5 is a flowchart showing the process (check C1) for detecting a hard disk with an abnormal tendency according to the first embodiment. First, the processing unit 51 checks whether the FT server 1 is operating in a synchronized state. If it is not operating in a synchronized state (step S1; NO), the processing unit 51 notifies the display unit (not shown) of an error indicating that the FT server 1 is not operating in a synchronized state (step S2). If it is operating in a synchronized state (step S1; YES), the processing unit 51 periodically issues IO to the hard disks 14 and 24 (step S3). The processing unit 51 measures the response time of each and records it in the recording unit (step S4).

Next, the determination unit 53 performs a response time monitoring process. Specifically, the determination unit 53 monitors whether the difference in response time between the hard disks exceeds a threshold value. To monitor for an abnormal trend in hard disk performance degradation, the determination unit 53 may, for example, calculate the difference in response time between the hard disks and monitor whether the difference exceeds a threshold value (M ms (ms: milliseconds)). The determination unit 53 may also, for example, calculate the ratio of the response times between the hard disks and monitor whether the ratio exceeds a threshold value (N times) (1 < N). Other indicators may be used as long as the purpose of monitoring hard disk performance degradation can be achieved. If the difference in response time exceeds the threshold value, the determination unit 53 detects that one of the hard disks has an abnormal trend in performance degradation.

If the difference in response time does not exceed the threshold (step S5; NO), the monitoring system 50 checks the synchronization status of the FT server 1 again, and the determination unit 53 waits for the next IO to be issued.

If the difference in response time exceeds the threshold (step S5; YES), the judgment unit 53 judges which hard disk has a longer response time. When the judgment unit 53 detects an abnormal trend of performance degradation, it notifies the synchronization control unit 54 that it is necessary to desynchronize the FT server 1. If the response time of the hard disk 24 is longer (step S6; YES), the synchronization control unit 54 desynchronizes the FT server 1 with the system 10 as the business continuation side and the system 20 as the hard disk diagnosis side (step S7). If the response time of the hard disk 14 is longer (step S6; NO), the synchronization control unit 54 desynchronizes the FT server 1 with the system 20 as the business continuation side and the system 10 as the hard disk diagnosis side (step S8).

Figure 6 is a flowchart showing the operation of system 10 and system 20 when canceling synchronization. Note that Figure 6 uses as an example a case where an abnormal trend of performance degradation is detected in the hard disk 24 on the system 20 side.

First, systems 10 and 20 of FT server 1 are operating in a synchronized state (step S11). Based on the control of synchronization control unit 54, FT controller 12 instructs FT controller 22 to cancel synchronization (step S12). This instruction causes FT controller 22 to cancel synchronization (step S16), and FT server 1 is divided into systems 10 and 20 that operate independently. System 10 continues operations (step S13), and system 20 stops operations (step S17). At this time, in order to continue operations on the system 10 side, the IP address that was used during the synchronous operation of FT server 1 continues to be used. On the system 20 side, a different IP address is temporarily assigned.

In the system 20, a hard disk diagnostic process A1 is executed for the hard disk 24 in which an abnormal trend has been detected, in order to determine whether or not the hard disk is actually abnormal (step S18). If the result of the hard disk diagnostic process A1 is that the hard disk 24 is determined to be normal (step S19; NO), the system 20 notifies the system 10 via the network that the hard disk diagnostic process has ended (step S20). When the system 10 receives the notification (step S14), the FT controller 12 instructs the FT controller 22 to synchronize (step S15). The system 20 receives the synchronization instruction and performs synchronization based on the system 10 that was continuing business processing (step S21). When the synchronization process is completed, the FT server 1 returns to a synchronized state (step S23).

If the result of the hard disk diagnostic process A1 indicates that there is an abnormality in the hard disk 24 (step S19; YES), the system 20 notifies the display unit (not shown) of an error indicating that the hard disk needs to be replaced (step S22).

Figure 7 is a flowchart showing the diagnostic process (hard disk diagnostic process A1) according to the first embodiment. The hard disk diagnostic process A1 is performed on a hard disk 24 that has been determined to have an abnormal trend of performance degradation in order to confirm whether the abnormal trend has been detected due to an accidental factor or whether an abnormality of performance degradation has actually occurred.

The hard disk diagnostic process A1 can be performed independently of the system on the business continuation side by releasing the synchronization of the FT server 1, making it possible to perform a detailed diagnosis without affecting business operations. In the first embodiment, a diagnostic method is shown in which an IO similar to check C1 is issued multiple times and judged based on the average response time, but the diagnostic method is not limited to this method. For example, it is also possible to perform a more detailed diagnosis, such as a full read and full write test of the hard disk.

In the hard disk diagnostic process A1, first, the processing unit 51 issues an IO to the hard disk 24 (step S31), measures the response time, and records it in the recording unit (step S32). The monitoring system 50 repeats the processes of steps S31 and S32 until the specified number of measurements (X times) is completed (step S33; NO). After the specified number of measurements is completed, the monitoring system 50 transitions to the determination process by the determination unit 53 (step S33; YES).

The determination unit 53 calculates the average response time of the hard disk 24 from the response times of the specified number of times (X times) of the hard disk 24 (step S34). The determination unit 53 also calculates the average response time of the hard disk 24 from the response times of the most recent X times of the hard disk 14 recorded in the recording unit (step S35).

The determination unit 53 compares the average response time of the hard disk 14 and the hard disk 24. If the average response time of the hard disk 24 is longer and the difference exceeds a threshold (step S36; YES), the determination unit 53 determines that the hard disk 24 is abnormal (step S37). If this condition is not met (step S36; NO), the determination unit 53 determines that the hard disk 24 is normal. Note that the threshold used here may be the same as the threshold used in check C1, but a threshold smaller than that of check C1 may be used to perform a more accurate diagnosis.

[effect]
According to the monitoring system 50 of the first embodiment, it is possible to avoid the impact on business processing due to a decrease in performance of one of the storage devices included in each of the two synchronized subsystems. This is because, in the monitoring system 50, the processing unit 51 issues an IO request to each storage device, the determining unit 53 determines whether the storage device is prone to an abnormality based on the difference in response time, and the synchronization control means releases the synchronization of the two subsystems. In addition, the subsystems whose synchronization has been released become capable of operating independently, and the processing unit 51 performs a diagnostic process for determining whether or not the storage device is abnormal in the subsystem including the storage device determined to be prone to an abnormality.

According to the first embodiment, it is possible to avoid a performance drop in the entire FT server 1 and the impact on business processing due to a performance drop in one of the hard disks. This is because the monitoring system 50 measures the response time of the IO requests of each hard disk, monitors for abnormal trends in performance drop, and releases synchronization of the FT server 1 if an abnormal trend is detected.

In addition, according to the first embodiment, by diagnosing a hard disk that shows signs of abnormality, it is possible to confirm whether or not an abnormality has actually occurred in the hard disk, thereby making it possible to avoid unnecessary replacement of the hard disk.

Furthermore, the business system can concentrate on business, and the hard disk diagnostic system can perform more detailed diagnosis of the hard disk without affecting business. This is because FT Server 1 is desynchronized and divided into System 10 and System 20, which operate independently.

[Second embodiment]
In the first embodiment, a case has been described in which, if an abnormal trend of performance degradation in the hard disk is detected at the check C1 stage, synchronization of the FT server 1 is cancelled. This is because priority is given to preventing degradation of hard disk performance from interfering with business processing (performance priority mode).

In the second embodiment, a case will be described in which synchronization is released while monitoring the load status of systems 10 and 20, such as CPU load and IO load. In the second embodiment, even if the monitoring system 50 detects an abnormal trend of performance degradation in the hard disk, if the system load is lower than the threshold and the impact on business processing is minor, the monitoring system 50 does not release synchronization of FT server 1 and continues diagnosing the hard disk. By releasing synchronization of FT server 1 only when the load exceeds the threshold, it is possible to prioritize maintaining the synchronized state of FT server 1 (synchronization priority mode).

[composition]
Fig. 8 is a block diagram illustrating a configuration of a monitoring system 50 according to a second embodiment. In Fig. 8, the monitoring system 50 is connected by wire or wirelessly to two subsystems, a system 10 and a system 20, which are capable of operating synchronously. Note that, regarding the monitoring system 50 according to the second embodiment, a description of the same configuration as that of the monitoring system 50 according to the first embodiment will be omitted. The monitoring system 50 according to the second embodiment differs from the monitoring system 50 according to the first embodiment in that it includes a recording unit 52 and further includes a load measuring unit 55.

The load measurement unit 55 measures the system load, such as the CPU load or IO load. If the system load does not exceed a predetermined threshold when an abnormal trend is determined, the synchronization control unit 54 does not release the synchronization of the subsystem, and the processing unit 51 performs diagnostic processing of the storage device in the synchronized subsystem. If the system load exceeds the predetermined threshold, the synchronization control unit 54 releases the synchronization of the subsystem.

In the second embodiment, the determination unit 53 monitors the load status in the load measurement unit 55. When the storage device shows an abnormal tendency of performance degradation and the load exceeds a threshold, the determination unit 53 transmits to the synchronization control unit 54 a notification that the subsystem needs to be desynchronized, together with information on which storage device shows an abnormal tendency.

[Action]
The process flow according to the second embodiment for monitoring the FT server 1 will be described with reference to the flowcharts of FIG. 9 to FIG.

Figure 9 is a flowchart showing the processing of the monitoring system 50 for detecting a hard disk with an abnormal tendency of performance degradation according to the second embodiment. First, the processing unit 51 checks whether the FT server 1 is operating in a synchronized state. If it is not operating in a synchronized state (step S41; NO), the processing unit 51 notifies the display unit (not shown) of an error that the FT server 1 is not operating in a synchronized state (step S42). If it is operating in a synchronized state (step S41; YES), the processing unit 51 periodically issues IO to the hard disks 14 and 24 (step S43), measures the response time of each, and records it in the recording unit 52 (step S44).

The determination unit 53 monitors whether the difference in response time between the hard disks exceeds a threshold value. The threshold value for monitoring the performance degradation of the hard disks may be the same as that in the first embodiment. If the difference in response time exceeds the threshold value, the determination unit 53 detects that one of the hard disks has an abnormal tendency for performance degradation.

If the difference in response time does not exceed the threshold (step S45; NO), the process returns to step S41 to check the synchronization state, and the determination unit 53 waits for the next IO to be issued.

If the difference in response time exceeds the threshold (step S45; YES), the determination unit 53 monitors whether the load exceeds the threshold. The load to be monitored may be selected according to the characteristics of the business, such as the CPU load or IO load, which is necessary for monitoring the degree of impact on business processing. There may be one monitored object, or a combination of necessary objects may be monitored.

If the load exceeds the threshold (step S46; YES), the judgment unit 53 judges which hard disk has a longer response time. If the response time of the hard disk 24 is longer (step S47; YES), the synchronization control unit 54 releases the synchronization of the FT server 1 with the system 10 as the business continuation side and the system 20 as the hard disk diagnosis side (step S48). If the response time of the hard disk 14 is longer (step S47; NO), the synchronization control unit 54 releases the synchronization of the FT server 1 with the system 20 as the business continuation side and the system 10 as the hard disk diagnosis side (step S49).

If the load does not exceed the threshold (step S46; NO), the FT server 1 performs the hard disk diagnostic process A2 while remaining in a synchronized state (step S50). If the result of the hard disk diagnostic process A2 indicates that there is an abnormality in the hard disk (step S51; YES), the FT server 1 separates the abnormal hard disk from the mirroring and notifies an error that the hard disk needs to be replaced (step S52). At this time, the CPU module of the FT server 1 operates while maintaining a synchronized state. Furthermore, necessary hardware components other than the CPU module may also operate while maintaining a synchronized state.

If the result of the hard disk diagnosis process A2 is that the hard disk is determined to be normal (step S51; NO), the process returns to step S41 to check the synchronization state and then waits for the next IO to be issued.

Figure 10 is a flowchart showing the diagnostic process (hard disk diagnostic process A2) according to the second embodiment. The hard disk diagnostic process A2 is performed when it is determined that one of the hard disks has an abnormal tendency of performance degradation, in order to check whether this is due to accidental factors or whether an abnormality in performance degradation has actually occurred. The hard disk diagnostic process A2 is performed when the FT server 1 is synchronized.

In the hard disk diagnostic process A2, first, IO is issued to the hard disk 14 and the hard disk 24, and the response time is measured and recorded (steps S61 and S62). Next, it is monitored whether the load exceeds a threshold value.

If the load exceeds the threshold (step S63; YES), a determination is made as to which hard disk has a longer response time. If the response time of hard disk 24 is longer (step S71; YES), the synchronization of FT server 1 is released with system 10 as the business continuation side and system 20 as the hard disk diagnosis side (step S72). If the response time of hard disk 14 is longer (step S71; NO), the synchronization of FT server 1 is released with system 20 as the business continuation side and system 10 as the hard disk diagnosis side (step S73).

If the load does not exceed the threshold (step S63; NO), this process is repeated until the specified number of measurements (X times) is completed (step S64; NO), and after the specified number of measurements is completed, the process proceeds to the judgment process step (step S64; YES).

In the judgment process, first, the average response time of the hard disk 14 and the hard disk 24 for a specified number of times (X times) is calculated (step S65). The average response times of the hard disk 14 and the hard disk 24 are compared, and if the difference does not exceed the threshold value (step S66; NO), the hard disk is judged to be normal (step S70).

The average response times of hard disk 14 and hard disk 24 are compared, and if the difference exceeds a threshold (step S66; YES), a comparison is made to see which hard disk has the longer average response time. If the average response time of hard disk 14 is longer (step S67; NO), the hard disk 14 is determined to be abnormal (step S68). If the average response time of hard disk 24 is longer (step S67; YES), the hard disk 24 is determined to be abnormal (step S69).

Figure 11 is a flowchart showing the operations performed by system 10 and system 20 when releasing synchronization of FT server 1 when the difference in response time between the hard disks exceeds a threshold and the load exceeds a threshold. Note that the following describes an example in which an abnormal trend of performance degradation is detected in the hard disk 24 on the system 20 side.

First, systems 10 and 20 of FT server 1 are operating in a synchronized state (step S81). When an abnormal trend of performance degradation is detected in the hard disk 24, the load exceeds a threshold, and a notification is received that FT server 1 needs to be desynchronized, FT controller 12 issues a desynchronization instruction to FT controller 22 (step S82). This instruction causes FT server 1 to be divided into systems 10 and 20, which operate independently (step S86). System 10 continues operations (step S83), and system 20 stops operations (step S87). At this time, in order to continue operations on the system 10 side, the IP address used during the synchronous operation of FT server 1 continues to be used. A different IP address is temporarily assigned on the system 20 side.

The system 20 executes a hard disk diagnostic process A1 for the hard disk 24 for which an abnormal trend has been detected, to determine whether or not the hard disk is actually abnormal. If the result of the hard disk diagnostic process A1 is that the hard disk 24 is determined to be normal (step S89; NO), the system 10 is notified via the network that the hard disk diagnostic process has ended (step S90). The system 10 that receives the notification (step S84) issues a synchronization instruction from the FT controller 12 to the FT controller 22 (step S85). The system 20 receives the synchronization instruction and performs synchronization based on the system 10 that was continuing business processing (step S91). When the synchronization process is completed, the FT server 1 returns to a synchronized state (step S92).

If the result of the hard disk diagnostic process A1 is that the hard disk 24 is determined to be abnormal (step S89; YES), the FT server 1 notifies the system 10 of an error that the hard disk is abnormal and needs to be replaced (step S93). At this time, the hard disk 24 determined to be abnormal is placed in an unsynchronized state (step S94), and the system 10 is notified via the network that the hard disk diagnostic process has ended (step S90). The system 10 that receives the notification (step S84) issues a synchronization instruction from the FT controller 12 to the FT controller 22 (step S85). The system 20 receives the synchronization instruction and performs synchronization based on the system 10 that was continuing business processing (step S91). When the synchronization process is completed, the hardware components of the FT server 1 other than the hard disk are restored to a synchronized state (step S92).

[effect]
According to the second embodiment, the present disclosure can also be applied when it is desired to give priority to the synchronization state of the FT server 1. The reason for this is that the FT server 1 performs a hard disk diagnosis while maintaining the synchronization of the FT server 1 until the load exceeds a threshold value. If it is determined that there is an abnormality in a hard disk, only that hard disk is separated from the mirroring.

In the second embodiment, when the load exceeds a threshold, the FT server 1 cancels synchronization of the FT server 1 and diagnoses the hard disks. The FT server 1 makes the hard disks that are determined to have an abnormality by the diagnosis unsynchronizable, and returns the other hardware components to a synchronized state.

[Hardware configuration]
In each of the above-described embodiments, each component of the monitoring system 50 is shown as a functional block. A part or all of each component of the monitoring system 50 may be realized by any combination of a computer 500 and a program. FIG. 12 is a block diagram showing an example of a hardware configuration of the monitoring system 50. Referring to FIG. 12, the computer 500 includes, for example, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input/output interface 511, and a bus 512.

The program 504 includes instructions for implementing each function of the monitoring system 50. The program 504 is stored in advance in the ROM 502, the RAM 503, or the storage device 505. The CPU 501 implements each function of the monitoring system 50 by executing the instructions included in the program 504. For example, the CPU 501 of the monitoring system 50 implements the functions of the monitoring system 50 by executing the instructions included in the program 504. The RAM 503 may also store data to be processed in each function of the monitoring system 50. For example, the response time to an IO request may be stored in the RAM 503 of the computer 500.

The drive device 507 reads and writes data from the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or a keyboard, and accepts information input from a user. The output device 510 is, for example, a display, and outputs (displays) information to a user. The input/output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the CPU 501 via a communication network, or may be stored in advance in the recording medium 506, read by the drive device 507, and supplied to the CPU 501. For example, the computer 500 and the subsystem in the above-mentioned embodiment may be connected via a communication network, or may be connected via the input/output interface 511.

Note that the hardware configuration shown in FIG. 12 is an example, and other components may be added, or some components may not be included.

[Modification]
There are various modified examples of the method of realizing the monitoring system 50. The monitoring system 50 may be realized by using the resources of the FT server 1. For example, the program of the monitoring system 50 may be loaded onto each of the hard disks 14, 24. Also, for example, the monitoring system 50 may be realized by any combination of a different computer and program for each component. Also, multiple components included in the monitoring system 50 may be realized by any combination of a single computer and program.

Furthermore, some or all of the components of the monitoring system 50 may be realized by general-purpose or dedicated circuits including a processor, etc., or a combination of these. These circuits may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of the monitoring system 50 may be realized by a combination of the above-mentioned circuits, etc., and a program.

In addition, when some or all of the components of the monitoring system 50 are realized by multiple computers, circuits, etc., the multiple computers, circuits, etc. may be centrally or decentralized.

Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with each other without departing from the scope of the present disclosure.

REFERENCE SIGNS LIST 1 FT server 10, 20 Subsystem 11, 21 CPU module 12, 22 FT controller 13, 23 IO module 14, 24 Hard disk 50 Monitoring system 51 Processing unit 52 Recording unit 53 Determination unit 54 Synchronization control unit 55 Load measurement unit

Claims (5)

A monitoring system for monitoring a fault-tolerant server having a first subsystem and a second subsystem, comprising:
a processing means for issuing an IO (Input Output) request to a storage device included in each of the first and second subsystems which perform business processing in synchronization with each other;
a determination means for determining an abnormal tendency of the storage device having a longer response time based on a difference in the response time to the I/O request;
a load measuring means for measuring a load of the first and second subsystems;
a synchronization control means for releasing the synchronization between the first and second subsystems and controlling the first and second subsystems to be able to operate independently when the load exceeds a predetermined threshold value when the determination means determines that there is an abnormal tendency,
The processing means includes:
When the load exceeds a predetermined threshold value when the abnormal tendency is determined, a diagnosis process is performed on the storage device in either the first or second subsystem, in which synchronization is released , the storage device being determined to have the abnormal tendency , to determine whether or not the storage device has an abnormality;
When the abnormal tendency is determined, if the load does not exceed a predetermined threshold, the first and second subsystems are synchronized and the diagnostic process is performed.
Surveillance system. the processing means determines whether the first and second subsystems are operating in synchronization, and when it is determined that the first and second subsystems are operating in synchronization, periodically issues the I/O requests, and measures and records the response times of the I/O requests;
The determining means determines whether the load exceeds a predetermined threshold value when the difference in the response times exceeds a threshold value.
The monitoring system of claim 1 . 1. A method for monitoring a fault-tolerant server having a first subsystem and a second subsystem, comprising:
issuing an IO (Input Output) request to a storage device provided in each of the first and second subsystems which perform business processing in synchronization with each other ;
determining an abnormal tendency of the storage device having a longer response time based on the difference in response time to the IO request;
Measuring loads on the first and second subsystems;
when the load exceeds a predetermined threshold when the abnormal tendency is determined, the synchronization of the first and second subsystems is released, and the first and second subsystems are controlled so as to be able to operate independently, and in the subsystem including the storage device determined to have the abnormal tendency, a diagnosis process is performed to determine whether or not there is an abnormality in the storage device;
When the load does not exceed a predetermined threshold value when the abnormal tendency is determined, the synchronization between the first and second subsystems is not released, and the diagnosis process is performed in a state in which the first and second subsystems are synchronized.
Monitoring methods. A program for monitoring a fault-tolerant server having a first subsystem and a second subsystem, comprising:
a process of issuing an IO (Input Output) request to a storage device included in each of the first and second subsystems which perform business processing in synchronization with each other;
A process of determining an abnormal tendency of the storage device having a longer response time based on a difference in the response time to the I/O request;
A process of measuring loads of the first and second subsystems;
a process of releasing synchronization between the first and second subsystems and controlling the first and second subsystems to be able to operate independently when the load exceeds a predetermined threshold when an abnormal trend is determined;
a diagnostic process for diagnosing whether or not there is an abnormality in the storage device in either the first or second subsystem in which synchronization has been released, the storage device being determined to have the abnormal tendency ;
a diagnostic process for diagnosing the presence or absence of an abnormality in the storage device while the first and second subsystems are in a synchronized state, without releasing the synchronization between the first and second subsystems if the load does not exceed a predetermined threshold when the abnormality tendency is determined;
A program that causes a computer to execute the following. a first subsystem having a first storage device, a first CPU module, and a first FT ( Fault Tolerant ) controller;
a second subsystem having a second storage device, a second CPU module, and a second FT controller;
the first and second storage devices respectively included in the first and second subsystems which perform business processing in synchronization with each other respond to an IO (Input Output) request issued from a monitoring system;
an abnormality tendency of the first storage device is determined based on the difference in response time to the I/O request , and when the load of the first and second subsystems exceeds a predetermined threshold , the first FT controller controls the release of synchronization of the first and second subsystems, and a diagnosis of an abnormality of the first storage device is performed in the first subsystem from which synchronization has been released ;
An abnormal tendency of the first storage device is determined based on the difference in response time to the I/O request, and if the load of the first and second subsystems does not exceed a predetermined threshold, a diagnosis of an abnormality of the first storage device is performed in a state in which the first and second subsystems are synchronized.
Fault-tolerant servers.

Images