JP3825640B2 - Fault-tolerant system and fault isolation method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a node system of a communication network that requires continuous operation for 24 hours, a fault-tolerant system used for servers in a company, and a fault isolation method thereof.
[0002]
[Prior art]
FIG. 7 is a diagram showing an example of a conventional fault-tolerant system of this type, and more particularly an example of a triple system that aims to improve efficiency and ensure fault isolation.
[0003]
As shown in FIG. 7, this fault-tolerant system includes three information processing apparatuses 1010, 1020, and 1030 having the same specifications. Each of the information processing apparatuses 1010, 1020, and 1030 includes a central processing unit 1011, 1021, 1031, main storage devices 1012, 1022, 1032, I / O devices 1013, 1023, 1033, and buses 1014, 1024, 1034 connecting them. Etc.
[0004]
A verification circuit 1040 is connected to the buses 1014, 1024, 1034 of the information processing apparatuses 1010, 1020, 1030. The collation circuit 1040 draws information flowing in each of the buses 1014, 1024, and 1034 and takes the majority of the three pieces of information.
[0005]
If the three pieces of information match, it is determined that there is no failure and the service continues. When the three pieces of information do not match, the collation circuit 1040 determines that one information processing apparatus that has output different information is a failure. The information processing apparatus determined to be faulty is disconnected from the fault-tolerant system 1050, and the service is continued with two normal information processing apparatuses.
[0006]
FIG. 8 is a diagram showing another example of a conventional fault-tolerant system, and is a diagram showing an example of a duplex system that avoids the overhead of hardware increase in the triple system.
[0007]
As shown in FIG. 8, this fault tolerant system includes two information processing apparatuses 1060 and 1070 having the same specifications. Each of the information processing apparatuses 1060 and 1070 includes a central processing unit 1061 and 1071, main storage devices 1062 and 1072, I / O devices 1063 and 1073, buses 1064 and 1074 for connecting them, and the like.
[0008]
A verification circuit 1080 is connected to the buses 1064 and 1074 of the information processing apparatuses 1060 and 1070. The collation circuit 1080 draws information flowing in each of the buses 1064 and 1074 and compares the two pieces of information.
[0009]
If the two pieces of information match, it is determined that there is no failure and the service continues. If the two pieces of information do not match, the collation circuit 1080 determines that an abnormality has occurred in one of the information processing apparatuses, and notifies that to both information processing apparatuses 1060 and 1070 (through an interrupt notification line (not shown)). Notice. Each of the information processing apparatuses 1060 and 1070 that received the interrupt suspends the service, performs failure processing including various tests such as comparison of register contents, and confirmation of normality of access to registers, memories, I / O devices, etc. Is determined, and the verification result is notified to the collation circuit 1080. The verification circuit 1080 isolates the suspicious information processing apparatus from the information, disconnects the apparatus from the fault-tolerant system 1090, and resumes the service with the normal information processing apparatus.
[0010]
[Problems to be solved by the invention]
As described above, isolation of failed information processing devices in a conventional fault-tolerant system by triple processing is realized by performing exactly the same processing with three information processing devices and making a majority decision with certain information of each information processing device It had been. In addition, in the conventional fault-tolerant system by duplication, the faulty information processing device is isolated when two information processing devices perform exactly the same processing, compare information of each information processing device, and a mismatch occurs. In order to identify which information processing device is out of order, analysis of failure history information, analysis of status information, comparison of register contents, confirmation of normality of access to registers, memory, I / O devices, etc. It was realized by conducting various tests.
[0011]
Therefore, the following points have been problems with conventional fault-tolerant systems.
(1) When a triple system is used, three information processing devices are required for identifying a faulty device, resulting in an increase in cost.
(2) In the case of a duplex system, a lot of processing and time are required to isolate the failed information processing apparatus, so that the service interruption time becomes longer and the reliability of the system is lowered.
[0012]
The present invention has been made in view of the above circumstances, and an object of the present invention is to solve the above-described problems and to isolate a failed information processing apparatus in a short time in a duplex system so that the service interruption time is as short as possible. It is an object of the present invention to provide a fault-tolerant system that can be shortened and a fault isolation method thereof.
[0013]
[Means for Solving the Problems]
In order to achieve the above object, in the present invention, two information processing apparatuses each having a central processing unit are connected to each information processing apparatus, and verification is performed by verifying the identity of processing in each information processing apparatus and detecting a failure. In the fault-tolerant system, the information processing apparatus and the collating means sequentially generate the same fault detection information, and the collating means generates the fault detection information generated in each information processing apparatus and the collating means, respectively. We propose a system characterized in that the occurrence of a failure in an information processing device is detected and the information processing device in which the failure has occurred is identified.
[0014]
According to the present invention, since the same failure detection information is sequentially generated in each information processing apparatus and collation means, it can be determined that no failure has occurred if the failure detection information matches each other. it can. When the pieces of failure detection information do not match each other, the occurrence of a failure can be detected and the information processing apparatus in which the failure has occurred can be identified from the combination. In this way, by determining the failure detection information, it is possible to easily detect the occurrence of a failure and identify the information processing apparatus in which the failure has occurred, in a short time, and to minimize service interruption. Can be suppressed.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
A fault tolerant system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a system configuration diagram of a fault-tolerant system. FIG. 2 is a diagram illustrating an example of a register configuration.
[0016]
As shown in FIG. 1, the fault-tolerant system 40 includes information processing devices 10 and 20 that perform the same processing, and a verification device 30 that connects the information processing devices 10 and 20.
[0017]
Each of the information processing apparatuses 10 and 20 includes buses 11 and 21, and central processing units (hereinafter referred to as CPU (Central Processing Unit)) 12 and 22 connected to the buses 11 and 21, main storage devices 13 and 23, I / O, respectively. O devices 14 and 24 are provided.
[0018]
The collation device 30 includes a collation circuit 31, and the collation circuit 31 is controlled by a farm engine 32. The firmware engine 32 includes a CPU, a memory, and the like (not shown) for operating the firmware. The collation circuit 31 includes failure detection registers 33a and 33b provided corresponding to the information processing devices 10 and 20, respectively, and further includes a failure detection information flag register 34.
[0019]
As shown in FIG. 2, the failure detection information flag register 34 has 1-bit failure detection information arrival flags 34a and 34b corresponding to the failure detection registers 33a and 33b, respectively. When new data other than “0” is written in the failure detection registers 33a and 33b, “1” is set in the failure detection information arrival flags 34a and 34b, respectively. When "0" is written in the failure detection registers 33a and 33b, the failure detection information arrival flags 34a and 34b are reset to "0", respectively.
[0020]
Each information processing apparatus 10 and 20 has a failure detection process 15 and 25. The failure detection processes 15 and 25 generate failure detection data using an algorithm that generates pseudo-random numbers by the joint multiplication method each time it is activated. The algorithm generates a different value every time it is executed, but if the seed is the same, the value always has a characteristic that is uniquely determined. The failure detection processes 15 and 25 usually sleep and are periodically started by a timer interrupt so as not to increase the load on the CPUs 12 and 22. Each time the failure detection processes 15 and 25 are started, pseudo random numbers are written as failure detection data in the failure detection registers 33a and 33b held by the verification circuit 31. Here, the pseudo-random numbers written to the failure detection registers 33a and 33b are numbers other than “0”. That is, when the result of the congruent multiplication method is “0”, data is generated again by the congruent multiplication method and repeated until data other than “0” is obtained.
[0021]
The farm engine 32 also has an algorithm for generating the pseudo-random number, and the same data string is generated by giving the same seed to the failure detection processes 15 and 25 and the farm engine 32.
[0022]
In this fault-tolerant system 40, it is possible to determine a fault in a short time by taking a majority decision between the contents of the fault detection registers 33a and 33b held by the verification circuit 31 and the pseudo random number data generated by the farm engine 32. it can.
[0023]
Next, the operation of the fault-tolerant system according to the present embodiment will be described with reference to FIGS. FIG. 3 is a diagram for explaining the operation of the fault-tolerant system, and FIG. 4 is a flowchart for explaining the operation of the fault-tolerant system.
[0024]
[Step SA1]
When the failure detection process 15 is activated, pseudo-random failure detection data is generated and a write request is generated to the failure detection register 33a. After acquiring the bus right of the bus 11, the CPU 12 issues a write request to the failure detection register 33a. The failure detection information arrival flag 34a is set to “1”. Similarly, when the failure detection process 25 is activated, pseudo-random failure detection data is generated and a write request is generated to the failure detection register 33b. After acquiring the bus right of the bus 21, the CPU 22 issues a write request to the failure detection register 33b. The failure detection information arrival flag 34b is set to “1”.
[0025]
When the failure detection information arrival flags 34a and 34b are set to "1", the firm engine 32 knows the arrival of failure detection data by raising an interrupt to the farm engine 32.
[0026]
If there is no write in the failure detection register to be collated from the other CPU 12 or 13 within a certain time, it is determined that a mismatch has occurred, and a message to that effect is sent to each of the information processing devices 10 and 20. Notice.
[0027]
[Step SA2]
The farm engine 32 reads the failure detection data, which is the contents of the failure detection registers 33a and 33b, and then writes "0" as data in the failure detection registers 33a and 33b, thereby causing a failure detection information arrival flag. 34a and 34b are reset to "0".
[0028]
Next, the collation circuit 31 takes a majority decision between the failure detection data in the failure detection registers 33a and 33b and failure detection data that is a pseudo-random number generated by the farm engine 32. It is determined that there is no, and no particular processing is performed.
[0029]
If the failure detection data in the failure detection register 33a does not match the other failure detection data as a result of the majority decision, it is determined that a failure has occurred in the information processing device 10, and the information processing device 10 and the verification device 30 Disconnect the connection. Then, the service is continued with the normal information processing apparatus 20.
[0030]
On the other hand, if the failure detection data in the failure detection register 33b does not match the other failure detection data as a result of the majority decision, it is determined that a failure has occurred in the information processing device 20, and the information processing device 20 and the verification device 30 Disconnect the connection. Then, the service is continued with the normal information processing apparatus 10.
[0031]
If the failure detection data generated by the farm engine 32 does not match the other failure detection data as a result of the majority decision, it is determined that a failure has occurred in the collation device 30. In the case of the present embodiment, since the collation device 30 is not multiplexed, the failure of the information processing devices 10 and 20 cannot be detected, and the service is continued only by the operation of the primary system that has been determined in advance. The case where the collation devices are multiplexed will be described later.
[0032]
FIG. 4 shows a flowchart of the above processing.
[0033]
As described above, in the fault-tolerant system 40 according to the present embodiment, the three fault detection data generated by the joint multiplication method in each of the information processing apparatuses 10 and 20 and the farm engine 32 are periodically compared with each other, and a majority decision is made. Thus, the occurrence of a failure in the information processing device is detected and the failed information processing device is identified. As a result, the service can be continued in the normal system after a short interruption of the service.
[0034]
As mentioned above, although one embodiment of the present invention was illustrated, the present invention is not limited to this. The scope of the present invention is defined by the terms of the claims, and all modifications that come within the meaning of the claims are intended to be embraced by the present invention.
[0035]
For example, in the above embodiment, the fault detection data is generated using the congruent multiplication method, but other algorithms may be used. Also, unique data may be used without using an algorithm.
[0036]
Moreover, in the said embodiment, although the failure detection processes 15 and 25 are regularly started by the timer interruption, you may start at another opportunity. For example, it may be the timing after the I / O access. Furthermore, as shown in FIG. 5, when the failure detection mechanism 35 detects a failure, the failure detection mechanism 35 raises an interrupt to each of the information processing devices 10 and 20, and the failure detection processes 15 and 25 are triggered by this. It may be activated. In the case of this example, the failure detection mechanism 35 compares the information flowing in the buses 11 and 21 of the information processing apparatuses 10 and 20 to determine the presence or absence of a failure. Further, the failure detection processes 15 and 25 may be executed at all times. In FIG. 5, the same reference numerals are given to the same components as those in the above embodiment.
[0037]
Furthermore, although only one collating device 30 is provided in the above embodiment, a plurality of (two in FIG. 6) collating devices 30 may be connected as shown in FIG. By configuring in this way, fault tolerance is further improved. In FIG. 6, the same reference numerals are assigned to the same components as those in the above embodiment.
[0038]
Furthermore, in the above embodiment, the failure detection registers 33 a and 33 b and the failure detection information flag register 34 are external to the verification circuit 31, but may be provided inside the verification circuit 31.
[0039]
Furthermore, in the above embodiment, the verification circuit 31 is controlled by the firmware 32, but may be controlled by wired logic.
[0040]
【The invention's effect】
As described above in detail, according to the present invention, by determining the failure detection information, it is possible to easily and quickly determine the detection of the failure and the identification of the information processing apparatus in which the failure has occurred, Service interruptions can be minimized.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram of a fault-tolerant system. FIG. 2 is a diagram illustrating an example of a register configuration. FIG. 3 is a diagram illustrating an operation of the fault-tolerant system. 5] System configuration diagram of a fault-tolerant system according to another example [Fig. 6] System configuration diagram of a fault-tolerant system according to another example [Fig. 7] System configuration diagram of a conventional fault-tolerant system [Fig. System configuration diagram of the fault-tolerant system [Explanation of symbols]
10, 20 ... Information processing device, 11, 21 ... Bus, 12, 22 ... Central processing unit, 13, 23 ... Main storage device, 14, 24 ... I / O device, 15, 25 ... Failure detection process, 30 ... Verification Device 31 ... Collation circuit 32 ... Farm engine 33a, 33b ... Fault detection register 34 ... Fault detection information flag register 34a, 34b ... Fault detection information arrival flag 35 ... Fault detection mechanism 40 ... Fault tolerance system

Claims (2)

In a fault-tolerant system having two information processing apparatuses each having a central processing unit, and a collating unit that connects each information processing apparatus and collates the identity of processing in each information processing apparatus and detects a failure,
Wherein each of the information processing apparatus,
Means for realizing a service that an information processing apparatus originally performs as a business;
Separately from the means for realizing the service that the information processing apparatus originally performs as a business, an algorithm for sequentially generating first failure detection information used only for failure detection is executed, and the generated first detection information is provided with a <br/> the first fault detection information generating means for transmitting to said comparing means,
The verification means includes
Second failure detection information generating means for sequentially generating second failure detection information with the same algorithm as the algorithm for sequentially generating the first failure detection information;
Said first and second fault detection information generating means determines the first and fault-free if the failure detection information matches all by the second fault detection information comparison generated respectively , only the first fault detection information in which one of the information processing apparatus is generated when matching the second failure detection information, and a detection means for identifying that a failure in the other information processing apparatus is generated A fault tolerant system characterized by that. In a fault isolation method of a fault-tolerant system, comprising two information processing apparatuses each having a central processing unit, and a verification means for connecting each information processing apparatus and verifying the identity of processing in each information processing apparatus and detecting a fault ,
Each of the information processing devices
A procedure for realizing a service that an information processing apparatus originally performs as a business;
Separately from the procedure for realizing the service that the information processing apparatus originally performs as a business, an algorithm for sequentially generating first failure detection information used only for failure detection is executed, and the generated first detection information is A first failure detection information generation procedure to be transmitted to the verification means;
And having
The verification means includes
A second failure detection information generation procedure for sequentially generating second failure detection information with the same algorithm as the algorithm for sequentially generating the first failure detection information;
By comparing the first and second failure detection information, if all the failure detection information matches, it is determined that there is no failure, and only the first failure detection information generated by one information processing device A fault isolation method for a fault-tolerant system, comprising: a detection procedure for specifying that a fault has occurred in the other information processing apparatus when s matches with the second fault detection information .

Images