JP7632633B2 - Virtualization system recovery device and virtualization system recovery method - Google Patents

Description

The present invention relates to a virtualization system recovery device and a virtualization system recovery method that detects abnormalities and recovers failures in containers and applications running on containers in a computing platform based on virtual machines or containers.

The virtual machine mentioned above is a computer that realizes the same functions as a physical computer using software. A container is a virtualization technology that is created by packaging an application in an environment called a "container" and runs on a container engine. In conventional container-based technologies, anomaly detection and failure recovery of containers and applications running on containers are mainly achieved by the Liveness/Readiness Probe function (also called the probe function) of Kubernetes, which will be described later.

CubeNetes is container virtualization software that creates and clusters containers such as Docker, and is also open source software. The liveness probe function controls whether or not a container will accept requests, while the readiness probe function controls whether or not a container will accept requests. An example of this type of prior art is the technology described in Non-Patent Document 1.

"Using Liveness Probes, Readiness Probes, and Startup Probes," kubernetes, [online], [Retrieved June 10, 2021], Internet, https://kubernetes.io/ja/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

By the way, in virtualization systems, which are not limited to the containers mentioned above but are part of the virtualization technology field, manual recovery work is performed based on the alerts that are issued when faults in the virtualization system are detected. However, since manual recovery work is performed after an alert is issued, it is difficult to reduce the time from the occurrence of a fault to normalization.

When detecting and recovering from such a fault using the probe function of CubeNetes that detects anomalies and recovers from faults, the period for fault monitoring can only be set to a slow period, such as one second, which is determined in advance. This poses the problem that when the fastest possible fault detection and recovery is required, fault detection and recovery cannot be performed faster than the default CubeNetes function.

The present invention was made in consideration of the above circumstances, and its objective is to detect abnormalities and recover from failures that occur in a virtualization system more quickly than the anomaly detection and recovery functions of the container virtualization software.

In order to solve the above problems, a virtualization system recovery device of the present invention includes a computational resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers, a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers, a plurality of clusters each including the computational resource cluster and the cluster management unit, an internal anomaly detection unit that is arranged for each of the plurality of clusters and that is virtually created outside the virtually created computational resource cluster and the cluster management unit and that detects an anomaly in the container, an external anomaly detection unit that is virtually created outside the plurality of clusters and that detects an anomaly in a cluster in which an abnormal container is arranged as an anomaly when an anomaly in a container is detected by the internal anomaly detection unit, and a distribution destination switching unit that is virtually created outside the plurality of clusters and that notifies a DNS (Domain Name System) of a communication distribution stop instruction related to the cluster in which the abnormality is detected by the external anomaly detection unit, and The DNS, which manages IP addresses in association with DNS (Network Address Translation) addresses, is provided outside the servers constituting the cluster, and the DNS erases the IP address of the abnormal cluster indicated in the communication distribution stop instruction .

According to the present invention, it is possible to detect abnormalities and recover from failures that occur in a virtualization system more quickly than the abnormality detection and recovery functions inherent in container virtualization software.

1 is a block diagram showing a configuration of a virtualization system recovery device according to an embodiment of the present invention. 11 is a block diagram showing a configuration when an endpoint setting unit and a Pod are deployed in a 1:1 configuration by a failure response deployment instruction unit in the virtualization system recovery device of this embodiment. FIG. FIG. 11 is a block diagram for explaining a first abnormality detection process of a container by a Pod of the virtualization system recovery device of this embodiment. 11 is a block diagram for explaining a second abnormality detection process by a routing table provided for each worker node of the virtualization system recovery apparatus of this embodiment. FIG. FIG. 11 is a block diagram for explaining a third abnormality detection process by monitoring a daemon of a virtual switch provided for each worker node of the virtualization system recovery device of this embodiment. FIG. 11 is a block diagram for explaining a fourth abnormality detection process by monitoring a daemon of a container runtime provided for each worker node of the virtualization system recovery device of this embodiment. FIG. 13 is a block diagram for explaining a fifth abnormality detection process by monitoring each worker node of the virtualization system recovery device of this embodiment. FIG. 13 is a block diagram for explaining a sixth abnormality detection process by monitoring a DB externally attached to a cluster of a container system of the virtualization system recovery device of this embodiment. 11 is a block diagram illustrating a configuration for explaining a process of detecting an anomaly related to the occurrence of a failure in a plurality of clusters by an external anomaly detection unit. FIG. 1 is a block diagram for explaining an abnormality handling process of the virtualization system recovery device of the present embodiment. FIG. FIG. 2 is a diagram showing the correspondence between domain names in a DNS record table and resolved IP addresses. FIG. 13 illustrates how IP addresses of a failed cluster are deleted from a DNS record table. 11 is a flowchart for explaining an abnormality handling process performed by the virtualization system recovery device of the present embodiment. FIG. 11 is a block diagram showing a configuration of a virtualization system recovery device according to a first modified example of an embodiment of the present invention. FIG. 11 is a block diagram showing a configuration of a virtualization system recovery device according to a second modified example of an embodiment of the present invention. FIG. 2 is a hardware configuration diagram showing an example of a computer that realizes the functions of the virtualization system recovery device according to the present embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, in all the drawings in this specification, components having corresponding functions are given the same reference numerals, and the description thereof will be omitted as appropriate.
<Configuration of the embodiment>
FIG. 1 is a block diagram showing the configuration of a virtualization system recovery device according to an embodiment of the present invention.

The container system 20 shown in Figure 1 is a virtualization system composed of multiple clusters (in this example, a first cluster 12A and a second cluster 12B) in which containers are clustered. The first cluster 12A is composed of a cluster management unit 14A and a computational resource cluster 15A. The second cluster 12B is composed of a cluster management unit 14B and a computational resource cluster 15B.

The cluster management units 14A and 14B are configured to include a communication distribution unit 14a, a computational resource operation unit 14b, a computational resource management unit 14c, a container configuration reception unit 14d, a container placement destination determination unit 14e, and a container management unit 14f. The computational resource clusters 15A and 15B are configured to include a plurality of applications 15a and 15b.

The cluster management units 14A and 14B are also referred to as the cluster management units 14, and the computational resource clusters 15A and 15B are also referred to as the computational resource cluster 15.

The virtualization system recovery device (also referred to as the recovery device) 10 shown in Figure 1 detects anomalies and recovers from failures in containers in a container system 20. This recovery device 10 is configured with cluster management units 14A, 14B, computational resource clusters 15A, 15B, internal anomaly detection units 17A, 17B, anomaly recovery response units 18A, 18B, failure response deployment instruction units 19A, 19B, an allocation destination switching unit 21, and an external anomaly detection unit 23.

The internal anomaly detection units 17A and 17B are also referred to as the internal anomaly detection unit 17, the abnormality recovery response units 18A and 18B are also referred to as the abnormality recovery response unit 18, and the fault response deployment instruction units 19A and 19B are also referred to as the fault response deployment instruction unit 19.

Inside each cluster 12A, 12B, an internal anomaly detection unit 17, an anomaly recovery response unit 18, and a failure response deployment instruction unit 19 are provided. Outside each cluster 12A, 12B, an allocation destination switching unit 21 and an external anomaly detection unit 23 are provided. However, the internal anomaly detection unit 17, the anomaly recovery response unit 18, the failure response deployment instruction unit 19, the allocation destination switching unit 21, and the external anomaly detection unit 23 are provided outside the cluster management unit 14 and the computational resource cluster 15 that are virtually created by the container virtualization software. In addition, the internal anomaly detection unit 17, the anomaly recovery response unit 18, and the failure response deployment instruction unit 19 may be provided outside each cluster 12A, 12B, like the allocation destination switching unit 21 and the external anomaly detection unit 23.

Since the first and second clusters 12A, 12B have substantially the same configuration, the functional configuration will be explained using the first cluster 12A as a representative.

The computational resource cluster 15 is configured with multiple applications 15a and 15b. In other words, the applications 15a and 15b are Pods (see Pods 15a and 15b in FIG. 3) that serve as a management unit for a collection of one or more containers. A Pod is the smallest unit of an application that can be executed by CubeNetes (container virtualization software). In other words, containers are created and clustered with the applications 15a and 15b as Pods, and this cluster is operated on a container engine. This computational resource cluster 15 is virtually created on a physical machine by the container virtualization software, and the virtually created containers are clustered and arranged.

The cluster management unit 14 manages the control related to the placement and operation of the virtually created and clustered containers. The cluster management unit 14 is configured with a communication distribution unit 14a, a computational resource operation unit 14b, a computational resource management unit 14c, a container configuration reception unit 14d, a container placement destination determination unit 14e, and a container management unit 14f.

In the recovery device 10 configured as above, the failure response deployment instruction unit (also referred to as the deployment instruction unit) 19 performs a process of deploying (placing) the endpoint (end point) setting units 14j, 14k and the Pods 15a, 15b shown in Fig. 2 in a 1:1 configuration. The endpoint setting units 14j, 14k are associated with each of the multiple Pods 15a, 15b, and the traffic allocation ratio (%) to each of the Pods 15a, 15b is set, and the endpoint setting units 14j, 14k become the end points of communication data.

The internal anomaly detection unit 17 shown in Figure 1 detects abnormalities in Pods (applications) 15a, 15b, which are one or more containers within the container system 20.

The abnormality recovery response unit 18 changes the weight value of the deployment instruction unit 19 associated with a Pod (e.g., Pod 15a) in which an abnormality has been detected by the internal abnormality detection unit 17 to 0%, and transmits a change command to the communication distribution unit 14a to separate the abnormal Pod 15a. When restoring the separated Pod 15a, the abnormality recovery response unit 18 transmits a recovery command to the communication distribution unit 14a to gradually increase traffic to the Pod 15a to be restored up to a predetermined traffic value.

In the cluster management unit 14, the communication distribution unit 14a is a router that distributes and notifies the change commands or recovery commands from the abnormality recovery response unit 18 to the corresponding units 14b to 14f. The communication distribution unit 14a also distributes traffic to the destination endpoint setting units 14j and 14k (described later) based on a weight value (%) indicating the traffic distribution ratio set for each of the endpoint setting units 14j and 14k (described later).

The container configuration reception unit (also referred to as the reception unit) 14d receives configuration information for deploying a container to the computational resource cluster 15 from an external server, etc.

The container placement destination determination unit (also referred to as the placement destination determination unit) 14e determines which container to place on which worker node (computational resource cluster 15) based on the configuration information received by the reception unit 14d.

The container management unit 14f checks whether the container is operating normally or not.

The computational resource management unit 14c grasps and manages whether a worker node is operational, the amount of computational resources used by the servers that make up the worker node, the remaining CPU (Central Processing Unit) capacity, etc.

The computational resource operation unit 14b performs operations to allocate a certain amount of computational resources, such as a CPU, to a certain container, in other words, to allocate storage capacity, CPU time, memory capacity available to the container, etc.

Next, various abnormality detection processes (first to sixth abnormality detection processes) related to containers in the container system 20 performed by the internal abnormality detection unit 17 of the recovery device 10 will be explained with reference to Figures 3 to 8.

<First abnormality detection process>
3 is a block diagram for explaining a first abnormality detection process of a container by the Pods (applications) 15a and 15b of the virtualization system recovery device 10 of the present embodiment. Each of the Pods 15a and 15b includes one or more containers.

In FIG. 3, within the container system 20, a master node 14A, an infrastructure node 14B, and worker nodes 15A and 15B are configured by virtual machines, and each is connected by a virtual switch {OVS (Open vSwitch)} 30. However, the virtual switch may be a virtual switch other than OVS. The master node 14A and the infrastructure node 14B correspond to the cluster management units 14A and 14B (FIG. 1), and the worker nodes 15A and 15B correspond to the computational resource clusters 15A and 15B (FIG. 1).

Furthermore, a first cluster 12 is formed by the master node 14A and the worker node 15A, and a second cluster 12 is formed by the infrastructure node 14B and the worker node 15B. It is assumed that these clusters 12 form a container system 20.

An internal anomaly detection unit 17 is arranged outside the container system 20, similar to the configuration in Figure 1. In Figure 3, a total of two internal anomaly detection units 17 are shown for each worker node 15A, 15B, but there may be only one. The master node 14A, infrastructure node 14B, worker nodes 15A, 15B, and internal anomaly detection unit 17 are connected to a counterpart device 24 via a network 22. The counterpart device 24 is a communication device such as an external server that transmits request signals, etc. to the container system 20.

The internal anomaly detection unit 17 sends a predetermined command (e.g., "sudo crictl ps") to Pods 15a and 15b of worker nodes 15A and 15B by polling as indicated by the round-trip arrows Y1 and Y2, and judges whether the command is normal or abnormal based on the response returned from Pods 15a and 15b in response to the command. In this polling test, the average round-trip time when polling was performed 10 times was 0.06 seconds.

The internal anomaly detection unit 17 judges whether an abnormality has occurred by reading a character string indicating normality or abnormality that is written in the command response result returned from Pod 15a, 15b by polling. For example, the character string "Running" indicates that the operation of the container (Pod 15a, 15b) is normal, and a character string other than "Running" indicates an abnormality. For this reason, the internal anomaly detection unit 17 judges that the operation of the container (Pod 15a, 15b) is normal if the command response result contains "Running," and judges that an abnormality has occurred if the command response result contains a character string other than "Running."

<Second abnormality detection process>
Next, FIG. 4 is a block diagram for explaining a second abnormality detection process by the routing table 15c provided for each of the worker nodes 15A, 15B of the virtualization system recovery device 10 of this embodiment.

The routing table (also called table) 15c uses route information indicating the destination container of a packet sent from the opposing device 24 to Pods 15a and 15b of worker nodes 15A and 15B via network 22. If the destination management in this table 15c is incorrect, the packet will not reach the appropriate container. For this reason, the internal anomaly detection unit 17 is configured to detect whether the destination management in table 15c is normal or abnormal.

However, the routing table 15c is composed of a pair of tables, "iptables" and "nftables". Alternatively, the routing table 15c may be composed of only "iptables" or only "nftables".

The internal anomaly detection unit 17 sends a specified command to each table 15c of the worker nodes 15A and 15B by polling as indicated by the reciprocating arrows Y3 and Y4, and determines whether the table is normal or abnormal based on the response results returned from each table 15c in response to the command.

The above-mentioned specified commands are a pair of "sudo iptables -L│wc-│" and "sudo nft list ruleset." The command "sudo iptables -L│wc-│" is notified to "iptables" in table 15c, and the command "sudo nft list ruleset" is notified to "nftables." Then, each of the "iptables" and "nftables" tables sends a response according to the command back to the internal anomaly detection unit 17.

In actual polling tests using a pair of commands, the average round-trip time when polling was performed 10 times was 0.03 seconds for the command "sudo iptables -L│wc-│" and 0.08 seconds for the command "sudo nft list ruleset."

The internal anomaly detection unit 17 judges anomalies as follows: if the command response result returned from each table 15c contains destination route information, it is judged to be normal; if nothing is contained, it is judged to be abnormal.

<Third abnormality detection process>
5 is a block diagram for explaining a third abnormality detection process by monitoring a daemon of the virtual switch 30 provided for each worker node 15A, 15B of the virtualization system recovery device 10 of the present embodiment. The daemon of the virtual switch 30 is also called an OVS daemon.

The daemon is a program that manages the destination of packets in the virtual switch 30. The internal anomaly detection unit 17 monitors the OVS daemon and detects that a packet is being sent properly if it is normal, and that a packet is not being sent if it is abnormal.

The internal anomaly detection unit 17 sends a specified command (for example, "ps aux|grep ovs-vswitchd|grep "db.sock"|wc-│") to the virtual switch 30 for each worker node 15A, 15B by polling as indicated by the reciprocating arrows Y5 and Y6, and determines whether the virtual switch 30 is normal or abnormal based on the response returned from the virtual switch 30 in response to the command.

In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

The internal anomaly detection unit 17 judges an anomaly as follows: if the command response result returned from each virtual switch 30 contains, for example, a "db.sock process" related to the destination, it is judged as normal; if it does not contain such a process, it is judged as abnormal.

<Fourth abnormality detection process>
6 is a block diagram for explaining a fourth abnormality detection process by monitoring a daemon of the container runtime 15d provided for each worker node 15A, 15B of the virtualization system recovery device 10 of the present embodiment. The daemon is also called a crio daemon and is an example of the container runtime 15d. crio (cri-o) is an open source community-driven container engine used in container-based virtualization technology.

The container runtime 15d is responsible for starting the containers of Pods 15a and 15b, so by monitoring the container runtime 15d, it is possible to detect whether the containers are running normally. Therefore, the internal anomaly detection unit 17 monitors the crio daemon and detects that the container is running as normal if it is running, and that the container is not running as an anomaly.

The internal anomaly detection unit 17 sends a specified command (e.g., "systemctl│status crio|grep Active") to the container runtime 15d for each worker node 15A, 15B by polling as indicated by the reciprocating arrows Y7 and Y8, and determines whether the status is normal or abnormal based on the response results returned from each container runtime 15d in response to the command.

In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

The internal anomaly detection unit 17 judges anomalies as follows: if the command response result returned from each virtual switch 30 contains "active (running)", which indicates the startup status of the crio daemon, then it is judged to be normal; if it contains anything other than "active (running)", then it is judged to be abnormal.

<Fifth abnormality detection process>
Next, FIG. 7 is a block diagram for explaining a fifth abnormality detection process by monitoring each of the worker nodes 15A, 15B by the virtualization system recovery device 10 of this embodiment.

However, this assumes a configuration in which worker nodes 15A and 15B are created using virtualization technology (virtual machines) using a physical machine 32. In this configuration, an internal anomaly detection unit 17 exists on the physical machine 32 outside the virtual machines, and this internal anomaly detection unit 17 detects that the container is normal if the virtual machine is running, and detects that the container is abnormal if the virtual machine is not running.

The internal anomaly detection unit 17 sends a specified command (e.g., "sudo virsh list") to each worker node 15A, 15B by polling as indicated by the reciprocating arrows Y9, Y10, and determines whether the node is normal or abnormal based on the response returned from each worker node 15A, 15B in response to the command.

In this actual polling test, the average round trip time when polling was performed 10 times was 0.03 seconds.

The internal anomaly detection unit 17 judges an anomaly as follows: if the command response result returned from each worker node 15A, 15B contains "running", which indicates the startup status of the target worker node 15A, 15B, then it is judged to be normal; if it contains anything other than "running", then it is judged to be abnormal.

<Sixth abnormality detection process>
Next, FIG. 8 is a block diagram for explaining a sixth abnormality detection process by monitoring DBs (Data Bases) 26a, 26b externally attached to the cluster 12 of the container system 20 of the virtualization system recovery device 10 of this embodiment.

As an external device of each cluster 12A, 12B (FIG. 1), DBs (also called external DBs) 26a, 26b that store data related to containers are configured to be connected to the worker nodes 15A, 15B via the network 22. In this case, the internal anomaly detection unit 17 is also connected to the worker nodes 15A, 15B via the network 22.

Here, since there is a configuration in which each cluster 12A, 12B is connected to each other via a network 22, even if the internal anomaly detection unit 17 is connected to the cluster 12 via the network 22 as shown in Figure 8, it is positioned as the internal anomaly detection unit 17 within each cluster 12A, 12B, as shown in Figure 1.

The internal abnormality detection unit 17 transmits a predetermined command to the external DBs 26a and 26b via the network 22 by polling as indicated by the reciprocating arrows Y11 and Y12, and judges whether the external DBs 26a and 26b are normal or abnormal based on the response results returned from the external DBs 26a and 26b in response to the command. The command in this case depends on the type of the external DBs 26a and 26b.

Response results include results related to response/alive monitoring and results related to the number of connections exceeding the upper limit. Response/alive monitoring monitors whether the external DBs 26a, 26b are running normally. In other words, the internal anomaly detection unit 17 determines that an anomaly has occurred if the response result indicates that the external DBs 26a, 26b are not running normally.

The "Connection limit exceeded" indicates that the number of containers connected to external DBs 26a and 26b exceeds a predetermined threshold. In other words, the internal anomaly detection unit 17 determines that an anomaly has occurred if the response result indicates that the number of connected containers to external DBs 26a and 26b exceeds the threshold.

In this actual polling test, the polling round trip time depends on the type of external DB 26a, 26b.

<Multiple cluster anomaly detection 1>
Next, when a fault occurs in a plurality of clusters 12A and 12B, the process of detecting an anomaly related to the fault will be described using the external anomaly detection unit 23 shown in Fig. 9. However, it is assumed that the anomaly detection 1 for each of the clusters 12A and 12B is any one of the first to sixth anomaly detections described above.

9, the external anomaly detection unit 23 is connected to the internal anomaly detection unit 17A of the first cluster 12A and the internal anomaly detection unit 17B of the second cluster 12B. When the internal anomaly detection units 17A, 17B detect an anomaly related to any one of the first to sixth anomaly detections, the external anomaly detection unit 23 detects the cluster 12A or 12B in which the container related to the anomalous application 15a, 15b is located as abnormal, as shown by the arrow Y31a or Y31b.

<Anomaly detection for multiple clusters 2>
9 is connected to a communication distribution unit 14a of a cluster management unit 14A in the first cluster 12A and a communication distribution unit 14a of a cluster management unit 14B in the second cluster 12B. The communication distribution unit 14a is provided at a signal input portion of the cluster management unit 14, and distributes and outputs an input signal to a subsequent stage, and also returns a response in response to confirmation communication for each of the clusters 12A and 12B when the clusters 12A and 12B are normal.

As indicated by the bidirectional arrows Y33a and Y33b, the external anomaly detection unit 23 performs confirmation communication for each cluster 12A and 12B at regular intervals with the communication distribution unit 14a for each cluster 12A and 12B, and detects whether a normal response is returned. If no response is returned, it detects an anomaly in the corresponding cluster 12A or 12B.

In this multiple cluster anomaly detection 2, anomaly detection in each cluster 12A, 12B is possible without going through the internal anomaly detection units 17A, 17B.

<Anomaly detection for multiple clusters 3>
The multiple cluster anomaly detection 3 is a process in which the external anomaly detection unit 23 performs anomaly detection for each of the clusters 12A and 12B by both of the above anomaly detections 1 and 2. In this process, anomaly detection for each of the clusters 12A and 12B can be performed more appropriately.

Figure 10 is a block diagram for explaining the anomaly response processing of the virtualization system recovery device 10 of this embodiment. The anomaly detection for which the anomaly response processing is performed is any one of the anomaly detections 1 to 3 of the multiple clusters described above.

The distribution destination switching unit 21 is connected to a DNS (Domain Name System) 25 external to the recovery device 10. The DNS 25 is a server that manages the correspondence between the domains (or domain names) indicating the names of the applications 15a and 15b of each cluster 12A and 12B and the resolved IP (Internet Protocol) addresses corresponding to the addresses of the communication distribution units 14a of each cluster 12A and 12B. This DNS 25 converts between domains and IP addresses and includes a DNS record table 25a.

As shown in Figure 11, the DNS record table (also called table) 25a stores a correspondence between a domain name and a resolved IP address. In this example, in table 25a, "Svc1.net" as the domain name of application 15a for each cluster 12A, 12B is associated with "IP address of first cluster 12A" and "IP address of second cluster 12B" as resolved IP addresses of communication distribution unit 14a for each cluster 12A, 12B.

This correspondence relationship indicates that application 15a of "Svc1.net" runs in either the first cluster 12A or the second cluster 12B.

Furthermore, in table 25a, "Svc2.net" as the domain name of application 15b for each cluster 12A, 12B is associated with "IP address of first cluster 12A" and "IP address of second cluster 12B" as the resolved IP addresses of communication distribution unit 14a for each cluster 12A, 12B.

This correspondence relationship indicates that application 15b "Svc2.net" runs in either the first cluster 12A or the second cluster 12B.

When an external server (not shown) queries DNS 25, which has such a table 25a, for a resolved IP address, DNS 25 returns the IP addresses of both clusters 12A and 12B. This allows the external server to send data to either cluster 12A or 12B.

10 detects one of the anomalies 1 to 3 in the multiple clusters 12A and 12B (for example, an anomaly in the second cluster 12B). The external anomaly detection unit 23 notifies the allocation destination switching unit 21 of the anomaly detection in the second cluster 12B, as shown by arrow Y34.

The destination switching unit 21 notifies the DNS 25 of an instruction to stop the distribution of communication to the second cluster 12B (communication distribution stop instruction), as shown by arrow Y35. In response to the communication distribution stop instruction, the DNS 25 performs a process of deleting the IP address of the second cluster 12B in the resolved IP addresses associated with both the domain names "Svc1.net" and "Svc2.net" in the table 25a shown in FIG. 12.

<Operation of the embodiment>
Next, the operation of the abnormality handling process will be described with reference to the flowchart shown in FIG.

13, suppose that a fault (marked with an x) occurs in the applications 15a and 15b of the second cluster 12B, and this fault is detected by the internal fault detection unit 17B. In this case, the internal fault detection unit 17B notifies the external fault detection unit 23 of the fault in the second cluster 12B, as shown by the arrow Y31b.

In step S2, the external anomaly detection unit 23 detects an abnormality in the second cluster 12B based on the above notification and notifies the allocation destination switching unit 21, as indicated by arrow Y34.

In step S3, the distribution destination switching unit 21 notifies DNS 25 of an instruction to stop distribution of communication to the second cluster 12B, as shown by arrow Y35.

In step S4, DNS 25 erases the IP address of the second cluster 12B from the resolved IP addresses associated with both the domain names "Svc1.net" and "Svc2.net" in table 25a shown in Figure 12. As a result, the resolved IP address associated with both the domain names "Svc1.net" and "Svc2.net" in table 25a becomes only the IP address of the first cluster 12A.

Therefore, in step S5, when an external server queries DNS 25 for a resolved IP address, only the IP address of the first cluster 12A is returned from DNS 25. In other words, since the second cluster 12B where the failure occurred cannot be accessed, communication to the second cluster 12B is stopped.

Effects of the embodiment
The effects of the virtualization system recovery device 10 according to the embodiment of the present invention will be described.

(1a) The recovery device 10 includes a computational resource cluster 15, a cluster management unit 14, a plurality of clusters 12A, 12B, an internal anomaly detection unit 17, and an external anomaly detection unit 23. The computational resource cluster 15 is virtually created on a physical machine by container virtualization software, and the virtually created containers are arranged in a cluster. The cluster management unit 14 manages control related to the arrangement and operation of the virtually created and clustered containers.

Each cluster 12A, 12B is configured to have a computational resource cluster 15 and a cluster management unit 14. The internal anomaly detection unit 17 is arranged for each cluster 12A, 12B, and is virtually created outside the virtually created computational resource cluster 15 and cluster management unit 14 to detect anomalies in the container. The external anomaly detection unit 23 is virtually created outside each cluster 12A, 12B, and is configured to detect, when the internal anomaly detection unit 17 detects an anomaly in a container, the cluster in which the abnormal container is located as being abnormal.

According to this configuration, when the internal anomaly detection unit 17 for each cluster 12A, 12B detects an anomaly in a container, the external anomaly detection unit 23 detects the cluster in which the abnormal container is placed as abnormal. The internal anomaly detection unit 17 and the external anomaly detection unit 23 are not involved in the container virtualization software that virtually creates the cluster management unit 14 and the computational resource cluster 15. As a result, a failure that occurs in each cluster 12A, 12B can be detected more quickly than the anomaly detection and recovery function of the container virtualization software. This early anomaly detection allows containers etc. related to the cluster failure to be quickly recovered.

(2a) The cluster management unit 14 is provided at a signal input portion of the cluster management unit 14, and includes a communication distribution unit 14a that distributes and outputs an input signal to a subsequent stage and returns a response to a cluster confirmation communication when the cluster is normal. The external anomaly detection unit 23 is configured to perform cluster confirmation communication with the communication distribution unit 14a at a predetermined period and detect an anomaly in the cluster when no response is returned.

With this configuration, abnormalities in each cluster can be detected without going through the internal anomaly detection unit 17 for each cluster 12A, 12B.

(3a) The external anomaly detection unit 23 is configured to detect anomalies by both a process of detecting an anomaly in the cluster in which the container is placed when the internal anomaly detection unit 17 detects an anomaly in the container, and a process of sending a cluster confirmation communication to the communication distribution unit 14a at a predetermined period and detecting an anomaly in the cluster if no response is returned.

With this configuration, anomaly detection for each cluster can be performed more accurately.

(4a) A DNS 25 that associates and manages a domain name indicating the name of an application related to a container with an IP address for each cluster is provided outside the servers that constitute each cluster 12A, 12B. A distribution destination switching unit that is virtually created outside each cluster 12A, 12B and notifies the DNS 25 of a communication distribution stop instruction related to an abnormal cluster detected by the external anomaly detection unit 23 is provided for each cluster. The DNS 25 is configured to erase the IP address of the abnormal cluster indicated by the communication distribution stop instruction.

According to this configuration, the IP address of the cluster in which an abnormality has been detected by the external anomaly detection unit 23 is erased from the IP addresses of the clusters managed by DNS 25. Therefore, when an external server queries DNS 25 for the IP address of the cluster, it becomes impossible to access the IP address of the cluster in which the failure has occurred. In other words, communication to the abnormal cluster can be stopped. The external anomaly detection unit 23, the allocation destination switching unit, and DNS 25 are not involved in the container virtualization software described above. Therefore, a failure that has occurred in a cluster can be detected earlier than the anomaly detection and recovery function of the container virtualization software, and therefore containers etc. related to the failure of the cluster in which the abnormality has been detected can be quickly recovered.

<First Modification of the Embodiment>
FIG. 14 is a block diagram showing a configuration of a virtualization system recovery device 10A according to a first modified example of the embodiment of the present invention.

The recovery device 10A of variant example 1 shown in Figure 14 differs from the recovery device 10 (Figure 10) in that the communication distribution stop instruction indicated by arrow Y35 from the distribution destination switching unit 21 is notified not only to DNS 25 but also to the communication distribution units 14a of each cluster 12A, 12B.

The communication distribution unit 14a stops communication with the first cluster 12A or the second cluster 12B indicated by the notified communication distribution stop instruction. In other words, since communication to each cluster 12A, 12B is always performed via the communication distribution unit 14a on the input side, the communication function of the communication distribution unit 14a is stopped in response to the communication distribution stop instruction.

According to this configuration, a communication sorting instruction for the abnormal cluster (e.g., the second cluster 12B) can be sent to the communication sorting unit 14a of the abnormal cluster 12B to stop the communication function of the communication sorting unit 14a. This stops the communication function of the abnormal cluster 12B, making it impossible to access the abnormal cluster 12B. This makes it possible to omit queries to the DNS 25 of the external server.

<Modification 2 of the embodiment>
FIG. 15 is a block diagram showing the configuration of a virtualization system recovery device 10B according to the second modification of this embodiment.

The recovery device 10B of variant example 2 shown in Figure 15 differs from the recovery device 10 (Figure 10) in that it has an internal DNS 25A in the first cluster 12A and an internal DNS 25B in the second cluster 12B, and the communication distribution stop instruction indicated by arrow Y35 from the distribution destination switching unit 21 is notified not only to DNS 25 but also to each internal DNS 25A, 25B.

The internal DNSs 25A and 25B have a DNS record table 25a like DNS 25, but the difference is that the table 25a is stored in a cache memory. Therefore, in the internal DNSs 25A and 25B, the information in the table 25a is erased at predetermined times. However, after the information is erased, the internal DNSs 25A and 25B are able to obtain the necessary information from DNS 25 as necessary.

The internal anomaly detection unit 17A (or the internal anomaly detection unit 17B) performs a process of erasing the IP address of the second cluster 12B from the table 25a shown in FIG. 12 in response to a communication distribution stop instruction (see arrow Y35) when an abnormality is detected in the second cluster 12B.

With this configuration, an external server can query the internal DNS 25A, 25B for each cluster for the IP addresses of each cluster 12A, 12B, thereby reducing the load on the external DNS 25.

<Hardware Configuration>
Any one of the virtualization system recovery devices 10, 10A, 10B according to the above-described embodiments is realized by a computer 100 having a configuration as shown in Fig. 16. The computer 100 has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HDD (Hard Disk Drive) 104, an input/output I/F (Interface) 105, a communication I/F 106, and a media I/F 107.

The CPU 101 operates based on a program stored in the ROM 102 or the HDD 104, and controls each functional unit. The ROM 102 stores a boot program executed by the CPU 101 when the computer 100 is started, programs related to the hardware of the computer 100, and the like.

The CPU 101 controls an output device 111 such as a printer or a display, and an input device 110 such as a mouse or a keyboard, via the input/output I/F 105. The CPU 101 acquires data from the input device 110, or outputs generated data to the output device 111, via the input/output I/F 105.

The HDD 104 stores programs executed by the CPU 101 and data used by the programs. The communication I/F 106 receives data from other devices (not shown) via the communication network 112 and outputs the data to the CPU 101, and also transmits data generated by the CPU 101 to other devices via the communication network 112.

The media I/F 107 reads the program or data stored in the recording medium 113 and outputs it to the CPU 101 via the RAM 103. The CPU 101 loads the program related to the target processing from the recording medium 113 onto the RAM 103 via the media I/F 107, and executes the loaded program. The recording medium 113 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto Optical disc), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory, etc.

For example, when the computer 100 functions as any one of the virtualization system recovery devices 10, 10A, and 10B according to the embodiment, the CPU 101 of the computer 100 executes a program loaded onto the RAM 103 to realize the functions of the virtualization system recovery device 10. Furthermore, the HDD 104 stores data in the RAM 103. The CPU 101 reads and executes a program relating to a target process from the recording medium 113. Additionally, the CPU 101 may read a program relating to a target process from another device via the communication network 112.
＜Effects＞
(1) A virtualization system recovery device comprising: a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers; a cluster management unit that manages control related to the arrangement and operation of the virtually created and clustered containers; a plurality of clusters, each of which is configured to have the computing resource cluster and the cluster management unit; an internal anomaly detection unit that is arranged for each of the plurality of clusters and is virtually created outside the virtually created computing resource cluster and the cluster management unit and that detects an abnormality in the container; and an external anomaly detection unit that is virtually created outside the plurality of clusters and that, when an abnormality in a container is detected by the internal anomaly detection unit, detects as abnormal a cluster in which the abnormal container is arranged.

According to this configuration, when an anomaly is detected in a container by the internal anomaly detection unit for each of the multiple clusters, the external anomaly detection unit detects the cluster in which the abnormal container is placed as abnormal. The internal anomaly detection unit and the external anomaly detection unit are not involved in the cluster management unit and the container virtualization software that virtually creates the computing resource cluster. As a result, anomalies that occur in multiple clusters can be detected more quickly than the anomaly detection and recovery function of the container virtualization software. This rapid anomaly detection allows containers etc. related to the cluster failure to be quickly recovered.

(2) The cluster management unit is provided at a signal input portion of the cluster management unit, and distributes and outputs the input signal to a subsequent stage, and includes a communication distribution unit that returns a response when the cluster is normal in response to a confirmation communication of the cluster, and the external anomaly detection unit is a virtualization system recovery device as described in (1) above, characterized in that it performs cluster confirmation communication with the communication distribution unit at a predetermined period, and detects an abnormality in the cluster when the response is not returned.

With this configuration, abnormalities in each cluster can be detected without going through the internal anomaly detection unit for each of the multiple clusters.

(3) The external anomaly detection unit is a virtualization system recovery device described in (2) above, characterized in that it detects an anomaly by both a process of detecting an anomaly in a cluster in which a container is placed when an anomaly in the container is detected by the internal anomaly detection unit, and a process of sending a cluster confirmation communication to the communication distribution unit at a predetermined period and detecting an anomaly in the cluster when no response is returned.

With this configuration, anomaly detection for each cluster can be performed more accurately.

(4) A virtualization system recovery device as described in any one of (1) to (3) above, characterized in that a DNS (Domain Name System) that associates and manages a domain name indicating the name of an application related to the container with an IP (Internet Protocol) address for each cluster is provided outside the server that constitutes the cluster, and a distribution destination switching unit that is virtually created outside the multiple clusters and notifies the DNS of a communication distribution stop instruction related to an abnormal cluster detected by the external anomaly detection unit is provided for each cluster, and the DNS erases the IP address of the abnormal cluster indicated by the communication distribution stop instruction.

According to this configuration, the IP address of a cluster that has an abnormality detected by the external anomaly detection unit is erased from the IP addresses of the clusters managed by DNS. Therefore, when an external server queries the DNS for the IP address of a cluster, it will not be able to access the IP address of the cluster where the failure occurred. In other words, communication to the abnormal cluster can be stopped. The external anomaly detection unit, the distribution destination switching unit, and the DNS are not involved in the container virtualization software described above. Therefore, a failure that occurs in a cluster can be detected earlier than the anomaly detection and recovery function of the container virtualization software, so that containers etc. related to the failure of the cluster where the abnormality was detected can be recovered earlier.

(5) The virtualization system recovery device described in (4) above is characterized in that the distribution destination switching unit notifies the communication distribution units in the multiple clusters of an instruction to stop communication distribution related to the cluster of the abnormality detected by the external anomaly detection unit, and the communication distribution unit performs processing to stop communication function when the communication distribution stop instruction related to the cluster of the abnormality detected by the external anomaly detection unit is notified.

With this configuration, a command to stop traffic sorting for the abnormal cluster can be sent to the traffic sorting unit of the abnormal cluster, and the communication function of the traffic sorting unit can be stopped. This stops the traffic sorting function from being accessible to the abnormal cluster. This makes it possible to omit queries to the DNS of external servers.

(6) A virtualization system recovery device as described in (4) above, characterized in that for each of the multiple clusters, an internal DNS is provided which, similar to the DNS, associates and manages a domain name indicating the name of the application related to the container with an IP address for each of the clusters, and notifies the internal DNS of a communication distribution stop instruction from the distribution destination switching unit.

With this configuration, external servers can query the internal DNS for each cluster for the cluster's IP address, thereby reducing the load on the external DNS.

In addition, the specific configuration may be modified as appropriate without departing from the spirit and scope of the present invention.

10, 10A, 10B Virtualization system recovery device 12A First cluster (cluster)
12B Second Cluster (Cluster)
14A, 14B Cluster management unit 14a Communication distribution unit 14b Computational resource operation unit 14c Computational resource management unit 14d Container configuration reception unit 14e Container placement destination determination unit 14f Container management unit 15A, 15B Computational resource cluster 15a, 15b Application 17A, 17B Internal abnormality detection unit 21 Distribution destination switching unit 23 External abnormality detection unit 25 DNS
25a DNS record table 25A, 25B Internal DNS

Claims (6)

a computing resource cluster that is virtually created on a physical machine by container virtualization software and that clusters and arranges the virtually created containers;
a cluster management unit that manages control related to the placement and operation of the virtually created and clustered containers;
A plurality of clusters, each of which includes the computational resource cluster and the cluster manager;
an internal anomaly detection unit that is disposed for each of the plurality of clusters, that is virtually created outside the virtually created computing resource clusters and the cluster management unit, and that detects an anomaly in the container;
an external anomaly detection unit that is virtually created outside the plurality of clusters and detects, when an anomaly is detected in a container by the internal anomaly detection unit , a cluster in which the abnormal container is located as abnormal;
a distribution destination switching unit that is virtually created outside the plurality of clusters and notifies a DNS (Domain Name System) of a command to stop distribution of traffic related to an abnormal cluster detected by the external abnormality detection unit;
the DNS that associates and manages a domain name indicating a name of an application related to the container with an IP (Internet Protocol) address for each cluster is provided outside a server that constitutes the cluster;
The DNS deletes the IP address of the abnormal cluster indicated in the communication distribution stop instruction.
A virtualization system recovery device comprising : the cluster management unit includes a communication distribution unit that is disposed at a signal input portion of the cluster management unit, distributes and outputs an input signal to a subsequent stage, and returns a response in response to a confirmation communication of the cluster when the cluster is normal;
2. The virtualization system recovery device according to claim 1, wherein the external anomaly detection unit performs a cluster confirmation communication with the communication sorting unit at a predetermined period, and detects an anomaly in the cluster when the response is not returned. 3. The virtualization system recovery device according to claim 2, characterized in that the external anomaly detection unit detects an anomaly by both a process of detecting an anomaly in a cluster in which the container is placed when the internal anomaly detection unit detects an anomaly in the container, and a process of sending a cluster confirmation communication to the communication sorting unit at a predetermined period and detecting an anomaly in the cluster when the response is not returned. the distribution destination switching unit notifies communication distribution units in the plurality of clusters of a communication distribution stop instruction related to a cluster in which an abnormality has been detected by the external abnormality detection unit,
2. The virtualization system recovery device according to claim 1 , wherein the communication sorting unit performs processing for stopping a communication function when an instruction to stop communication sorting related to a cluster in which an abnormality has been detected by the external abnormality detection unit is notified. an internal DNS for managing, for each of the plurality of clusters, a domain name indicating a name of an application related to the container and an IP address for each of the clusters in association with each other, in the same manner as the DNS;
2. The virtualization system recovery device according to claim 1 , further comprising: a notification of a traffic sorting stop instruction from said sorting destination switching unit to said internal DNS. A method for recovering a virtualization system by a virtualization system recovery device, comprising:
Virtualization system recovery device,
A plurality of clusters in which containers virtually created by container virtualization software are clustered and arranged on a physical machine;
The virtual internal anomaly detection unit and the virtual external anomaly detection unit are provided outside the plurality of clusters ,
a DNS that associates and manages a domain name indicating a name of an application related to the container with an IP address for each cluster, the DNS being provided outside the server that constitutes the cluster;
Virtualization system recovery device,
the internal anomaly detection unit detecting an anomaly in a container for each of the plurality of clusters;
a step in which the external anomaly detection unit detects, when the internal anomaly detection unit detects an anomaly in a container, a cluster in which the abnormal container is located as an anomaly ;
notifying the DNS of a command to stop traffic sorting related to the cluster of anomalies detected by the external anomaly detection unit;
The DNS executes a step of deleting the IP address of the abnormal cluster indicated in the communication sorting stop instruction.
A virtualization system recovery method comprising :

Images