JP5876425B2 - Virtual machine live migration scheduling method and virtual machine system - Google Patents

Description

  The present invention relates to virtual machine technology, and in particular, a scheduling method for performing live migration in which a virtual machine is moved between physical machines without stopping the operation of the virtual machine having a plurality of physical machines, and the like And a virtual machine system that executes various scheduling methods.

  Virtual machine technology is attracting attention for the purpose of optimal load arrangement and power saving. A virtual machine (VM) is realized by software that emulates the operation of a computer, and on a single physical machine (computer as a physical entity) by using virtual machine technology. A different operating system (OS) may be operated to execute various application software (APL) on the OS, or software that originally operates in a different computer architecture environment may be executed. it can. Virtual machine technology is widely used in data centers, for example.

  In addition, there is a “live migration technology” that moves a virtual machine for each OS between different physical machines connected to a network without stopping the operation of the virtual machine. Live migration is realized by transmitting a copy of the memory information of the virtual machine to the migration destination physical machine via the network. With this live migration technology, for example, when a certain physical machine has a high load, the virtual machine running on the physical machine can be moved to another physical machine to eliminate the high load state. .

  FIG. 1 is a diagram for explaining live migration of a virtual machine. The migration source physical machine 11 and the migration destination physical machine 12 are connected by a network 13 and share a shared storage 14. In each physical machine 11, 12, the virtualization software 15 is operating, and a virtual machine (VM) is realized on the physical machine 11, 12 by the virtualization software 15. In the VM, an operating system OS and application software APL are executed. Live migration is achieved by memory copying from the migration source physical machine 11 to the migration destination physical machine 12 via the network 13.

  In Non-Patent Document 1, a VM management server that manages a VM is provided separately from a server that is a physical machine, and the VM management server gives a live migration execution instruction to a VM operating on the server. It is disclosed that memory information is transmitted through a network between a migration source server and a migration destination server while the operation of the VM is continued. When the memory difference information generated during transmission decreases to a predetermined amount, the VM of the migration source server is temporarily stopped, and the VM is started on the migration destination server. As a result, live migration of the VM is realized.

  FIG. 2 is a diagram for explaining the time required for live migration. When performing a memory copy for live migration, the memory information changes sequentially because the VM is operating on the migration source physical machine even during the memory copy. Therefore, after executing a memory copy corresponding to the entire memory area of the VM on the migration source physical machine, it is necessary to copy the difference regarding the memory information changed during the memory copy and send it to the migration destination physical machine. is there. Since the memory state also changes during this differential copy, it is necessary to copy the difference between them. As described above, when the memory copy and the difference copy are executed several times and the memory difference information is reduced to the above amount (the time required so far is referred to as migration time), the VM is stopped at the migration source, and finally The difference remaining in is copied, then the VM is started at the migration destination, and the processing by the VM is resumed. Here, the time required from the stop of the VM at the migration source to the restart of the VM at the migration destination is referred to as downtime. The total time required for live migration (total migration time) is represented by the sum of migration time and downtime.

  By the way, in addition to the network for VM live migration, the network constituting the virtual machine environment is synchronized with a SAN (Storage Area Network) for shared storage access and a standby VM. Various networks are required, such as a network for fault tolerance, a network for communication with the VM management server, and a network connected to a wide area network (WAN). FIG. 3 shows an example of a VM and a network environment. It is inefficient to physically separate all networks due to the limitation of the number of interfaces and cost, etc., and physical links are shared by these networks and virtual local area network (VLAN) technology is used. By dividing the network logically, resources can be used efficiently. However, during execution of VM live migration, various types of traffic other than memory information for VM live migration may flow through the network between the migration source physical machine and the migration destination physical machine. There is sex.

  FIG. 4 is a diagram illustrating an example of network traffic assumed when live migration is executed. FIG. 4 shows a virtual machine system in which a plurality of physical machines are connected via a network. Here, physical machine # 1 to physical machine # 3 are connected to switch (SW) # 1, and similarly, Physical machine # 4 to physical machine # 6 are connected to switch # 2. Switch # 1 and switch # 2 are further connected to a packet transfer network via a router. Further, the shared storage 14 and the VM management server 16 are also connected to the switch # 1. Here, it is assumed that the physical machine # 3 is for a standby VM, and one VM (VM # 1) is live migrated from the physical machine # 1 to the physical machine # 2. At this time, shared storage access traffic occurs between the physical machine # 1 and the shared storage 14, and user traffic occurs between the physical machine # 1 and the packet transfer network. The physical machine # 1 and the physical machine # 3, traffic of fault tolerance synchronization information is generated, and the bandwidth of the physical link between the physical machine # 1 and the switch # 1 is consumed by these traffics. The bandwidth for memory copy for live migration with the machine # 2 becomes extremely small. If sufficient bandwidth that can be used for live migration cannot be secured, it will take time to complete memory migration, and it will take time to complete live migration, or the memory difference information will not become too small and migration will fail. there is a possibility.

“VMware vSphereTM vSphere vMotionTM: Migrate Virtual Machines Without Downtime”, [online], [Search June 11, 2012], Internet <http://www.vmware.com/en/products/datacenter-virtualization/ vsphere / vmotion / overview.html>

  As described above, when performing live migration of a virtual machine (VM) in a conventional network environment, sufficient bandwidth for memory copy cannot be secured due to the influence of other traffic on the network, and the live migration takes time. Or live migration failure occurs.

  In order to solve the above-described problems in the conventional live migration technology, the present inventors have already disclosed in Japanese Patent Application No. 2012-176811 other traffic on the network through which VM memory copy traffic flows when VM live migration is executed. It is proposed to temporarily limit the bandwidth for and secure bandwidth for memory copy traffic. By this method, the VM live migration can be completed at high speed. FIG. 5 shows an example of a system configuration in which the bandwidth of traffic other than the VM memory copy is limited when the VM live migration is executed as described above.

  In the system shown in FIG. 5, a plurality of physical machines (physical machine # 1 to physical machine #i) are connected to each other via a switch 30, and each switch 30 has a traffic monitor for monitoring traffic in the network. A unit 33 and a bandwidth control unit 32 that controls the bandwidth for each traffic are provided. One or a plurality of VMs can be operated on each physical machine, and each physical machine is provided with a VM state monitoring unit 31 that monitors the state of the VM on the physical machine. Each switch is connected to a packet transfer network via a router, and a plurality of user terminals are connected to the packet transfer network. A management device 20 for controlling the switch is also provided. When the execution of live migration is determined, the management apparatus 20 monitors the status of each VM by the VM status monitoring unit 31 in each physical machine and information based on the result of monitoring the traffic by the traffic monitoring unit 33 of each switch. Based on the information based on the results, determine the bandwidth required for live migration, determine whether there is a free bandwidth that exceeds that bandwidth, and if the free bandwidth is insufficient, determine the bandwidth control method and control the bandwidth of each switch. Part 32.

  In the configuration shown in FIG. 5, the traffic from the user terminal to each VM (user traffic) and the traffic for performing VM memory copy at the time of live migration (VM memory copy traffic) share the network. Here, a case where a plurality of VMs operating on one physical machine (for example, physical machine # 1) are sequentially moved to another physical machine (for example, physical machine # 2 to physical machine #i). Think. In the system shown in FIG. 5, the VM memory copy traffic bandwidth is secured by temporarily restricting user traffic. However, as indicated by the thick line in FIG. If most of the link bandwidth is given, the bandwidth in which user traffic can be used becomes extremely small as shown by the thin dotted line in the figure. As a result, depending on the VM, the user traffic of the VM is band-limited for a long time, which causes the user to feel quality degradation. Here, an example in which the VM memory copy traffic and the user traffic output from the VM compete with each other is described, but the traffic that competes with the VM memory copy traffic may be traffic other than the user traffic. After all, when the bandwidth of traffic other than the VM memory copy is restricted at the time of VM live migration, the user traffic is restricted for a long time depending on the migration schedule and bandwidth control amount of the VM, and the deterioration of the user quality becomes serious. The problem of becoming.

  An object of the present invention is to provide a scheduling method that can sufficiently secure a bandwidth required for memory copy of live migration and that does not excessively restrict traffic competing with live migration, and a virtual machine system that executes such a scheduling method. It is to provide.

  The live migration scheduling method of the present invention includes a plurality of physical machines and a network that connects the physical machines, a virtual machine is operable on each physical machine, and one or more other physical machines are moved from the source physical machine. Virtual machine system in which memory copy traffic for moving virtual machines and other traffic different from memory copy traffic for moving virtual machines use a common link during live migration of multiple virtual machines to other physical machines In the method of scheduling live migration, a step of determining a live migration execution order in a plurality of virtual machines based on at least one of information on virtual machines and information on traffic on a network, Determine the amount of bandwidth control for other traffic during live migration so that the fluctuation from the other traffic bandwidth before migration is reduced, and perform physical migration of virtual machines one by one according to the execution order. A step of instructing a machine, and a step of setting a bandwidth control amount in a bandwidth control means incorporated in a physical machine or a network.

  The virtual machine system of the present invention is a virtual machine system having a plurality of physical machines and a network for connecting the physical machines, and capable of operating a virtual machine on each physical machine, and is incorporated in the physical machine or the network. A bandwidth control means for performing bandwidth control of other traffic different from the memory copy traffic for moving the virtual machine, a management device for controlling live migration of the virtual machine between physical machines and instructing bandwidth control, When performing live migration of a plurality of virtual machines from a migration source physical machine, the management device performs live migration on the plurality of virtual machines based on at least one of information on the virtual machines and information on traffic on the network. The execution order of One of the schedule determination means and the execution order determined by the schedule determination means for determining the bandwidth control amount of the other traffic during the live migration so that the fluctuation from the other traffic bandwidth before the implementation of the migration is reduced. VM management means for instructing physical machines to perform live migration of virtual machines, and bandwidth control setting means for setting the bandwidth control amount of other traffic determined by the schedule determination means in the bandwidth control means.

  In the present invention, the other traffic different from the memory copy traffic for moving the virtual machine is, for example, user traffic.

  The present invention determines the execution order of live migration in a plurality of virtual machines based on at least one of information related to virtual machines and information related to traffic on the network, and the fluctuation from the bandwidth of other traffic before live migration is small. By determining the amount of bandwidth control for other traffic bandwidth during live migration, sufficient bandwidth for memory copy of live migration can be secured without excessively restricting user traffic competing with live migration. As a result, the user quality degradation can be minimized.

It is a figure explaining the live migration of a virtual machine (VM). It is a figure explaining time required for live migration. It is a figure which shows an example of VM and a network environment. It is a figure which shows an example of the network traffic assumed at the time of execution of live migration. FIG. 3 is a block diagram illustrating an example of a system configuration in which a bandwidth of traffic other than a VM memory copy is limited when performing VM live migration. It is a block diagram explaining the live migration in the virtual machine system of one embodiment of this invention. In the virtual machine system in FIG. 6, the user traffic bandwidth is compared between when live migration is performed with a fixed bandwidth for VM memory copy and when live migration is performed based on the method of the present invention. It is a graph shown. It is a block diagram which shows the 1st structural example of a virtual machine system. It is a block diagram which shows the 2nd structural example of a virtual machine system. It is a block diagram which shows the 3rd structural example of a virtual machine system. It is a block diagram which shows the 4th structural example of a virtual machine system. FIG. 6 is a sequence diagram illustrating an operation in the first embodiment. FIG. 10 is a sequence diagram illustrating an operation in the second embodiment.

  Next, modes for carrying out the present invention will be described with reference to the drawings. FIG. 6 is a diagram illustrating live migration in the virtual machine system according to the embodiment of this invention.

  The virtual machine system shown in FIG. 6 has a configuration in which a plurality of physical machines (in the example shown, five physical machines # 1 to # 5) are connected via a network, similar to that shown in FIG. Have. Each physical machine is connected to a switch 30 in the network. Here, a plurality of switches 30 are provided, and each physical machine is connected to one of the switches 30. Each switch 30 is also connected to a packet transfer network via a router in the network. Some user terminals are also connected to the packet transfer network.

  Also in this virtual machine system, similarly to the virtual machine system shown in FIG. 5, each physical machine, each VM (virtual machine) and each switch are controlled, and live migration (movement) and bandwidth control of the VM are performed. A management device 20 is provided that limits the bandwidth of each traffic in the network when performing live migration. The virtual machine system management apparatus 20 illustrated in FIG. 6 further includes a schedule determination unit 26 as compared with the apparatus illustrated in FIG. The schedule determination unit 26, according to the VM state information stored in the data storage unit 23 of the management device 20 and the schedule determination algorithm 27, the movement order of a plurality of migration target VMs (that is, the execution order of VM live migration), In addition, the bandwidth control amount and control order of user traffic are determined. Here, the bandwidth control amount represents the bandwidth amount allocated to the user traffic under bandwidth limitation. Further, in addition to the schedule determination unit 26, the management apparatus 20 includes a packet reception unit 21 that receives a packet as an interface with the network, a packet transmission unit 22 that transmits a packet as an interface with the network, and the data storage unit 23 described above. A VM management unit 24 that manages VMs on the physical machine, an information collection unit 25 that collects necessary information and stores it in the data storage unit 23, and a user traffic bandwidth limit determined by the schedule determination unit 26 And a bandwidth control setting unit 28 that performs bandwidth control setting for each physical machine, VM, and switch. Here, the VM management unit 24 instructs each physical machine to start the VM live migration based on the result determined by the schedule determination unit 27 in particular. Such a management apparatus 20 is connected to a physical machine of the same group that executes live migration. However, the connection from the management apparatus 20 does not have to be a direct connection, and may be reachable on the network. Accordingly, in the illustrated example, the management device 20 is directly connected to each switch.

  Each physical machine is provided with a VM state monitoring unit 31 that monitors the load state of the physical machine, the CPU usage rate and the memory usage rate of each VM, the operation state of the application, and collects information related thereto. In particular, the VM state monitoring unit 31 has a function of monitoring the memory usage and the memory change rate of each VM. Here, the memory change speed indicates how much the information stored in the VM memory changes per time. If the memory change speed is high, the differential copy shown in FIG. The VM state monitoring unit 31 may be provided in the management device 20 as will be described later.

  Each switch is provided with a bandwidth control unit 32 and a traffic monitoring unit 33. The bandwidth control unit 32 has a function of performing bandwidth control of each traffic as instructed by the bandwidth control setting unit 28 of the management device 20 based on the determination by the management device 20. As will be described later, the bandwidth control unit 32 may be provided in each physical machine instead of the switch. The traffic monitoring unit 33 monitors input / output traffic of each VM. The traffic monitoring unit 33 may be provided in each physical machine.

  In the above configuration, the information collection unit 25 collects information on traffic flowing through the link between the physical machines that perform live migration of the VM from the traffic monitoring unit 33 provided in the switch or physical machine, and sends the information to the switch or physical machine. From the provided VM state monitoring unit 31, VM state information representing the load state of the physical machine, the CPU usage rate and memory usage of each VM, the memory change speed, the operation state of the application, and the like is collected.

  Next, the schedule determination algorithm 27 in this embodiment will be described. The schedule determination algorithm 27 represents a schedule determination method that defines how to determine the VM movement order, the traffic bandwidth limit amount, the movement schedule, and the like when the execution of the VM live migration is determined. Specifically, the schedule determination method is based on the VM movement order when given the number of VMs, the planned VM movement time, the memory usage of each VM, the memory change speed of each VM, the user traffic amount, and the like. This is a method for determining the bandwidth control amount and control schedule of traffic. The planned VM movement time is a time planned as the time required for the entire movement of the migration target VM when performing live migration of the VM sequentially for one or a plurality of VMs. When the execution of the VM live migration is determined, the schedule determination unit 26 reads the VM state information and the schedule determination algorithm 27 from the data storage unit 23 prior to the start of the VM migration, and determines the read VM state information and the schedule. Based on the algorithm 27, information such as the number of VMs, the planned VM movement time, the memory usage of each VM, the memory change speed of each VM, and the user traffic volume, the movement order of VMs, the bandwidth control amount and control schedule of traffic, etc. The information indicating them is output. Here, the method of determining the moving order when the VMs are sequentially moved is called a VM order determining method, and the method of determining the bandwidth control amount of the user traffic of each VM is determined as the bandwidth control method. This is called a method. As will be described below, there are several types of VM order format methods and bandwidth control method determination methods.

  In the example illustrated in FIG. 6, assuming that there are five VMs (VM # 1 to VM # 5) on the physical machine # 1, the VM # 2 is moved to the physical machine # 2 (live migration), and the VM # 3 Is moved to the physical machine # 3, the VM # 4 is moved to the physical machine # 4, and the VM # 5 is moved to the physical machine # 5. In addition, the user traffic only considers the direction from the physical machine to the user terminal. In the figure, as the VM order determination method, the movement time per VM is calculated from the VM movement plan time, and the movement is completed for each VM in the movement time per machine from the VM memory usage and the memory change speed. The required bandwidth (that is, the VM memory copy traffic bandwidth) is calculated, and the VM is moved in order from the smallest required bandwidth. In addition, as a bandwidth control method determination method, by dividing the amount required for VM migration (VM memory copy traffic bandwidth) from the physical bandwidth by the number of VMs operating on the migration source physical machine, The bandwidth control amount of user traffic for each VM for the migration source physical machine is determined. When moving a plurality of VMs, the VMs are moved one by one in order, and thus the bandwidth of user traffic in the physical machine after the movement is not limited for the VMs that have been moved. As a result, in the example shown in FIG. 6, bandwidth control is performed so that the amount of fluctuation from the user traffic bandwidth before movement becomes small.

  Here, in the case shown in FIG. 6, when the VMs # 2 to VM # 5 on the physical machine # 1 are moved to the physical machines # 2 to # 5, respectively, the memory usage and the memory for each VM The difference between the method of this embodiment and the conventional method shown in FIG. 5 when the change rate is as shown in Table 1 will be described with specific numerical values. As a physical band, 1000 Mbit / s is prepared for each physical machine, and 1 minute is assumed as a VM movement planned time for moving four VMs. Then, the movement time per VM becomes 15 seconds. Also, it is assumed that VM # 1 remains in physical machine # 1.

  In the conventional method shown in FIG. 5, if a bandwidth of 900 Mbit / s is given to the VM memory copy traffic and the VM memory copy traffic is made one by one from VM # 2, the time change of the user traffic bandwidth in each VM is as shown in FIG. It looks like the one shown in half. Here, for simplification of explanation, the time change of the user traffic band is not shown for VM # 2 and VM # 4. The time required to complete the migration of VM # 2 to VM # 5 is as short as less than 20 seconds. However, since VM # 1 does not move from the physical machine # 1 of the migration source, until the entire migration is completed, User traffic bandwidth is greatly limited. Also, since VM # 5 moves last and exists in physical machine # 1 that is the migration source of the other VMs that have been migrated, user traffic for VM # 5 also remains until the entire migration is completed. Bandwidth is greatly limited. On the other hand, if the movement of the VM # 3 is completed and the user physical traffic # 3 is received by the movement destination physical server # 3, then the VM # 3 will not be limited by the user traffic band thereafter. As a whole, in the conventional method, although all the movement target VMs have a short time to complete the movement, the user traffic bandwidth is limited to an extremely low 20 to 50 Mbit / s during the movement period of about 20 seconds. Is done.

  On the other hand, the lower half of FIG. 7 shows the time change of the user traffic band in each VM in the embodiment shown in FIG. As described above, since the movement time per VM is 15 seconds, the bandwidth (VM memory copy traffic bandwidth) required to move the VM in 15 seconds for each of VM # 2 to VM # 5 is As shown in Table 1. Since the movement order is determined in the order from the smallest band, VM # 3 is moved first, and then moved by VM # 2, VM # 5, and VM # 4. And while moving each of VM # 2-VM # 5, the bandwidth control amount allocated to the user traffic of each VM is as shown in Table 1 for each moving VM. That is, during the movement of the first moving VM # 3, 88.26667 Mbit / s is allocated as the VM memory copy traffic band for the movement, so this VM memory copy traffic band from 1000 Mbit / s of the physical band. 91.733333 Mbit / s obtained by subtracting is the bandwidth value allocated to the entire user traffic of VM # 1 to VM # 5 existing in the migration source physical machine # 1 at this time. By dividing this by 5 which is the number of VMs on the physical machine # 1, the user traffic bandwidth control value of 182.34 Mbit / s is given to each VM. Similarly, when considering VM # 2 that moves second, the VM memory copy traffic bandwidth for moving in 15 seconds is 136.5333 Mbit / s. At this time, VM # 3 has already completed the movement. Therefore, the number of VMs in the migration source physical machine # 1 is 4. Therefore, 215.88 Mbit / s obtained by subtracting the VM memory copy traffic from the physical band and divided by 4 is 215.88 Mbit / s, while VM # 2 is moving, VM # 1, VM # 2, VM # 4 and This is the user traffic bandwidth control value assigned to VM # 5. Since VM # 3 has already been moved to physical machine # 3, its user traffic control value matches the physical bandwidth (1000 Mbit / s).

  As described above, according to the embodiment shown in FIG. 6, although the time until all VMs complete the movement (this time can be set by the user as the VM movement plan time) is long, the user traffic for each VM As a band, approximately 200 Mbit / s can be maintained. Considering that the average user traffic bandwidth allocated to each VM at the time when five VMs originally exist in the physical machine # 1, is 200 Mbit / s (= (1000 Mbit / s) / 5). According to this, it is possible to minimize the amount of decrease in the user traffic bandwidth without greatly increasing the VM live migration time, and it is possible to minimize the degradation of the user quality.

  Next, variations in each of the VM movement order determination method and the bandwidth control method determination method in the schedule determination algorithm will be described. Any of the variations of the VM movement order determination method and any of the variations of the bandwidth control method determination method can be arbitrarily combined.

  The VM movement order determination method includes (1) a memory information utilization method that determines a VM movement schedule based on memory information of each VM, and (2) a VM movement schedule that is determined based on the input or output traffic volume of each VM. Traffic information utilization method.

  The bandwidth control method determination method includes (a) a VM movement time equalization method in which a bandwidth control value is determined under the condition that the movement time of each VM is equal, and (b) user traffic is restricted during VM movement. There is a user traffic bandwidth reduction minimization method that determines the bandwidth control value so that the amount of reduction in user traffic due to restriction falls within a set value and the movement of all VMs is completed in the shortest time.

  Further, as one of the variations in which the VM movement order determination method and the bandwidth control method determination method are integrated, (3) a VM whose memory change speed is 0 (that is, since the memory contents of the VM do not change during movement, the difference For VMs that do not require copying, there is a multiple VM simultaneous movement method in which movement is performed simultaneously with movement of other VMs.

  Hereinafter, these variations will be described in detail.

(1) Memory information utilization method:
The memory information utilization method is a method suitable for a case where VM user traffic is uniform, although the VM memory usage and the memory change rate differ for each VM. When the memory usage is c [bytes], the memory change rate is a [bit / s], and the bandwidth is b [bit / s], the VM movement time t [s] is t = c × 8 / (b−a In the memory information utilization method, the VM movement order is determined in ascending order of the VM movement time t. The bandwidth b is an arbitrary constant given according to the specifications of the network constituting the virtual machine system. Table 2 shows an example of determining the movement order when 10 VMs (VM # 1 to VM # 10) exist and move all of them. Here, the band b = 3 × 10 ⁸ [bit / s]. Table 2 shows the calculation result of the VM movement time t and the movement order based on it, assuming that the memory usage c and the memory change speed a are given for each VM.

  In the memory information utilization method, the resources of the migration source physical machine can be vacated at an early stage by quickly moving to another physical machine from the VM whose movement time is short, and therefore exists in the migration source physical machine at that time. There is an advantage that the limitation of the user traffic bandwidth in the VM can be reduced.

(2) User traffic information usage method:
The user traffic information utilization method is a method suitable for the case where the VM memory usage and the memory change speed are all the same, but the VM user traffic varies depending on the VM. In this method, the order of movement of VMs is determined in descending order of VM input or output user traffic volume. As to whether it is based on input traffic or output traffic, user traffic in the same direction as VM memory copy traffic output from the migration source physical machine is selected for each VM. Table 3 shows an example of determining the movement order when 10 VMs (VM # 1 to VM # 10) exist and move all of them.

  In the example shown in Table 3, the movement order is determined according to the output user traffic amount for each VM. In the user traffic information utilization method, a VM having a large output user traffic volume, that is, a VM that uses a lot of resources is moved to another physical machine, so that the resource of the migration source physical machine can be freed from an early stage. There is an advantage that the restriction of the user traffic bandwidth in the VM existing in the physical machine of the movement source at that time can be reduced.

(A) VM movement time equalization method:
In the VM movement time equalization method, first, the movement time t of each VM is
Movement time t = VM movement planned time / number of VMs to be moved. After that, for each movement VM, the bandwidth B necessary for completing the movement within the movement time t is obtained using the memory usage c [bytes] and the memory change speed a [bit / s]. When the VM is moving, the bandwidth limit value of the user traffic of each VM is as follows when the VM user traffic bandwidth is uniform in all VMs:
(Physical bandwidth-B) / (Number of VMs remaining in the migration source physical machine)
On the other hand, if the VM user traffic bandwidth is different for each VM, the remaining bandwidth represented by (physical bandwidth -B) is calculated as the mutual amount of user traffic bandwidth of each VM immediately before the start of movement of the VM. It is assumed that it is assigned to each VM at a ratio between. Example of user traffic bandwidth control value of each VM when the VM is moving for each moving VM when 10 VMs (VM # 1 to VM # 10) exist and move all of them Is shown in Table 4. Here, it is assumed that the VM user traffic bandwidth is uniform in all VMs, and the VM migration plan time is 120 seconds (therefore, the migration time t of each VM is 12 seconds).

  In the VM movement time equalization method, in order to increase the user traffic bandwidth of each VM while a certain VM is moving, the VM movement plan time may be increased. Since the user traffic bandwidth control value of each VM is set even during movement so as to be proportional to the traffic bandwidth of each VM before the start of movement, the bandwidth is reduced so that the fluctuation from the user traffic before movement is reduced. Control will be implemented.

(B) User traffic bandwidth reduction minimization method:
In the traffic bandwidth reduction minimization method, the maximum value of the amount of reduction in user traffic due to bandwidth limitation is set to x in advance. The user traffic control value y during movement of each VM is
y = (user traffic band before movement) −x
Is required. Band b that can be used to move each VM when moving each VM is
b = physical band- (sum of y of each VM remaining in the migration source physical machine)
Is required. In particular, when the user traffic bandwidth is uniform, the bandwidth b is
b = physical band− {y * (total number of VMs−movement order + 1)}
It is expressed.

The movement time t of each VM is
t = c × 8 / (ba)
Is required.

  Table 5 shows that in the user traffic bandwidth reduction minimization method, when 10 VMs (VM # 1 to VM # 10) exist and move all of them, the VM moves for each moving VM. Table 5 shows an example of the user traffic bandwidth control value of each VM and the time during which bandwidth control is performed. Here, since the physical bandwidth is 1 Gbit / s and the number of VMs is 10, each VM traffic bandwidth before the first movement is 100 Mbit / s, and the maximum reduction amount of user traffic is 5 Mbit / s. Further, it is assumed that the VM user traffic is uniform.

  In the user traffic bandwidth reduction minimization method, how much the user traffic bandwidth is restricted during movement of the VM is determined by how the maximum reduction amount is set. The user traffic bandwidth control value of each VM is set even during movement as a value obtained by subtracting the maximum amount of decrease from the user traffic band of each VM before the start of movement so that fluctuations from the user traffic before movement are reduced. Band control is performed.

(3) Multiple VM simultaneous movement method:
Next, for a VM whose memory change speed is 0, a multiple VM simultaneous movement method is described in which when a movement of another VM is executed, the movement is performed in parallel using a free band.

First, as a first step (step 1), for a VM whose memory change rate a [bit / s] is not 0, the memory usage is c [byte] and the bandwidth is b [bit / s].
VM movement time t [s] = c × 8 / (ba)
Ask for. The bandwidth b is an arbitrary constant given according to the specifications of the network constituting the virtual machine system. Then, for the VM in which a ≠ 0, the movement order is determined in the order of the shortest VM movement time t.

  On the other hand, for VMs with a memory change rate a of 0, the order of movement is determined separately in ascending order of memory usage.

Table 6 shows an example of determining the movement order when 10 VMs (VM # 1 to VM # 10) exist and move all of them. Here, the band b is assumed to be 3 × 10 ⁸ [bit / s]. Among the 10 VMs, VM # 3 and VM # 4 have a = 0, so the VM movement time t is set for other VMs as indicated by 1 to 8 in the “movement order” column of the table. The order of movement is determined accordingly. For VM # 3 and VM # 4 in which a = 0, the order of movement is determined in the order of increasing memory usage c, as indicated by AB in the “movement order” column of Table 6.

Next, as a second step, for a VM whose memory change speed a is not 0, the movement time t of the VM is calculated as VM movement time t = VM movement planned time / number of VMs to be moved. For each movement VM, a bandwidth B required to complete the movement within the movement time t is obtained using the memory usage c [bytes] and the memory change speed a [bit / s]. The bandwidth limit value of the user traffic of each VM while the VM is moving is (physical bandwidth-B) / (number of VMs remaining in the migration source physical machine)
And

  Table 7 shows that in the case shown in Table 6, the VM movement plan time is 120 seconds, the movement time of each VM (VM # 3 and VM # 4) whose memory change speed a is 0 is 15 seconds, and the user traffic is The user traffic bandwidth control value of each VM when it is assumed to be uniform is shown.

  Thereafter, in the third step, too much bandwidth is calculated when the maximum value of the user traffic bandwidth is x, and the VM (VM # 3 and VM # 4) having a memory change rate a of 0 is used by using the bandwidth. ) To schedule the move.

  If the multiple VM simultaneous migration method is used, VMs that do not change in memory are migrated in parallel with other VMs using the free bandwidth, so that the time to complete migration of all VMs can be further reduced. Can do. Note that, in the multiple VM simultaneous movement method, similarly to the above-described VM movement time equalization method, bandwidth control is performed so that fluctuations from user traffic before movement are reduced.

  Next, an example of a virtual machine system suitable for each of the above variations will be described.

  The virtual machine system shown in FIG. 8 is suitable when the memory information utilization method is used as the VM movement order determination method, or when the multiple VM simultaneous movement method is used. As the bandwidth control method determination method, either the VM moving time equalization method or the user traffic bandwidth reduction minimization method can be used.

  This virtual machine system has a configuration in which a plurality of physical machines (physical machines # 1 to #N) are connected to a network, and each physical machine is connected to a switch 30 in the network. The switch 30 is also connected to the packet transfer network. Although not shown, a plurality of user terminals are connected to the packet transfer network 30. Each physical machine is capable of operating a virtual machine (VM) on it. In the illustrated example, n VMs (VM # 1 to VM # n) are arranged on the physical machine # 1. Each physical machine is provided with a VM state monitoring unit 31 that monitors the state of each VM of the physical machine. The VM state monitoring unit 31 particularly monitors the memory usage and the memory change rate of each VM.

  Further, in this virtual machine system, a management device 20 that manages each physical machine, each VM, and the switch 30 and instructs VM movement and bandwidth control between the physical machines is connected to the switch 30. As the management device 20, the management device 20 described with reference to FIG. 6 can be used. Particularly in this example, the information collecting unit 25 collects the memory usage and the memory change rate of each VM from the VM state monitoring unit 31 of each physical machine. However, the memory usage and memory change rate of a VM may be obtained by calculation or estimation from the type of application installed in the VM and its state.

  The switch 30 is a device for connecting the physical machine and the management apparatus 20 and for connecting between the physical machines. The switch 30 is provided with a bandwidth control unit 32 that performs bandwidth control of the traffic of each VM, in particular, user traffic, in response to an instruction from the bandwidth control setting unit 28 of the management device 20.

  In the network configuration in the virtual machine system using the memory information utilization method or the multiple VM simultaneous migration method, the connectivity between the management apparatus 20 and each physical machine is maintained, and the connectivity between the physical machines is maintained. It is not limited to what was shown in FIG. Further, the VM state monitoring unit 31 can be provided in the management apparatus 20 instead of in the physical machine. When the VM state monitoring unit 31 is provided in the management apparatus 20, the VM state monitoring unit 31 is connected to each physical machine via a network and acquires information regarding the state of each VM. Further, the bandwidth control unit 32 can be mounted not in the switch 30 but in each physical machine. When mounted in a physical machine, the bandwidth control unit 32 executes a bandwidth control operation on a virtual switch that connects a VM in the physical machine. FIG. 9 shows a configuration when a bandwidth control unit 32 is provided in each physical machine.

  The virtual machine system shown in FIG. 10 is suitable when the user traffic information utilization method is used as the VM movement order determination method. As the bandwidth control method determination method, either the VM moving time equalization method or the user traffic bandwidth reduction minimization method can be used. This virtual machine system is the same as the virtual machine system shown in FIG. 8, except that each physical machine is provided with a traffic monitoring unit 33 instead of the VM state monitoring unit 31. FIG. It is different from what is shown. The traffic monitoring unit 33 monitors input / output user traffic of each VM. The information collection unit 25 in the management apparatus 20 collects information related to input / output user traffic of each VM from the traffic monitoring unit 33 of each physical machine and stores it in the data storage unit 23 as user traffic information. In response, the schedule determination unit 26 determines the VM movement order and the user traffic bandwidth control amount based on the user traffic information stored in the data storage unit and the schedule determination algorithm.

  The network configuration in the virtual machine system using the traffic information utilization method is as shown in FIG. 10 if the connectivity between the management apparatus 20 and each physical machine is maintained, and the connectivity between the physical machines is maintained. It is not limited to. Further, the traffic monitoring unit 33 can be provided in the switch 30 instead of in the physical machine. Further, the bandwidth control unit 32 can be mounted not in the switch 30 but in each physical machine. When mounted in a physical machine, the bandwidth control unit 32 executes a bandwidth control operation on a virtual switch that connects a VM in the physical machine. FIG. 11 shows a configuration in which the traffic monitoring unit 33 is provided in the switch 30 and the bandwidth control unit 32 is provided in each physical machine in the configuration shown in FIG.

  Next, some of the combinations of variations of the VM movement order determination method and the bandwidth control method determination method will be described in more detail.

[Example 1]
A case will be described in which the virtual machine system having the configuration shown in FIG. 8 is used, the memory information utilization method is used as the VM movement order determination method, and the VM movement time equalization method is used as the bandwidth control method determination method. It is assumed that a plurality of user terminals are connected to the illustrated packet transfer network. It is assumed that the number of physical machines in the virtual machine system is three (physical machine # 1 to physical machine # 3) and is connected to the same switch 30. Assuming that the physical machine # 1 has 10 VMs (VM # 1 to VM # 10), VM # 1 to VM # 5 are live migrated to the physical machine # 2, and VM # 6 to VM # 10 Are migrated live to the physical machine # 3. It is assumed that the time required for the memory copy for live migration (VM migration planned time) is 120 seconds. Further, it is assumed that the bandwidth of each physical link is 1 Gbit / s. Further, it is assumed that traffic flows from each VM to the user terminal, and the amount of user traffic is uniform in each VM.

  As shown in FIG. 8, the switch 30 is provided with a bandwidth control unit 32, and the traffic is a VM (virtual local area network), which VM traffic, and user traffic. Or VM memory copy traffic. Bandwidth control can be performed for each VLAN. The VM state monitoring unit 31 on each physical machine monitors the VM memory usage and usage rate by using, for example, the Linux (registered trademark) top command on each VM, and determines the number of paging (swap file) operations. Monitor the memory usage and memory change speed of each VM.

  The flow of operation in this case is shown in FIG.

  When the VM management unit 24 in the management apparatus 20 determines to execute live migration of VM # 1 to VM # 10 in step 101, the VM management unit 24 notifies the information collection unit 25 to that effect (step 102). The information collecting unit 25 collects VM state information from the VM state monitoring unit 31 of the physical machine # 1 that is the migration source physical machine (step 103), and stores the collected information in the data storage unit 24 (step 104). . Next, the schedule determination unit 26 acquires the VM state information and the scheduling determination algorithm from the data storage unit 23 (step 105), determines the schedule by the memory information utilization method and the VM movement time equalization method (step 106), and determines The schedule thus stored is stored in the data storage unit 24 (step 107). This schedule consists of a schedule (migration schedule) indicating the movement order of VMs and a schedule (bandwidth control schedule) indicating the bandwidth control amount of user traffic. In the case of this example, the determined schedule is as shown in Table 8, and the movement order of the VMs is the order of VM # 4 → VM # 1 → VM # 7 →.

  The schedule determination unit 26 notifies the migration management unit 24 of the migration schedule (step 108), and notifies the bandwidth control setting unit 28 of the bandwidth control schedule (step 109). Then, the bandwidth control setting unit 28 outputs, from the physical machine # 1, 91.47 Mbit / s (see Table 8) that is the user traffic bandwidth control value of each VM corresponding to the VM # 4 that is the VM that moves first. The bandwidth is set in the bandwidth control unit 32 in the switch 30 so as to be the user traffic bandwidth control value of each VM (step 110).

  Thereafter, the VM management unit 24 instructs the physical machine # 1 to perform live migration of the VM # 4 to be moved first (step 111), and thereby, from the physical machine # 1 to the physical machine # 2. Memory copy is performed for VM # 4 (step 112). When the memory copy for VM # 4 is completed, notification of completion of live migration of VM # 4 is sent from the physical machine # 2 to the VM management unit 24 (step 113), and the VM management unit 24 The bandwidth control setting unit 28 is notified of the completion of live migration (step 114). Then, the bandwidth control setting unit 28 outputs 100.52 Mbit / s (see Table 8), which is the user traffic bandwidth control value of each VM corresponding to the VM # 1 that is the second moving VM, from the physical machine # 1. The bandwidth is set in the bandwidth control unit 32 in the switch 30 so as to be the user traffic bandwidth control value of each VM (step 115).

  As a result, the VM management unit 24 instructs the physical machine # 1 to perform the live migration of the VM # 1 that is to be moved second, similarly to the movement of the VM # 4 (Step 116). As a result, the memory copy for the VM # 1 from the physical machine # 1 to the physical machine # 2 is performed (step 117). When the memory copy for VM # 1 is completed, notification of completion of live migration of VM # 1 is sent from the physical machine # 2 to the VM management unit 24 (step 118), and the VM management unit 24 sends the notification of VM # 1. The bandwidth control setting unit 28 is notified of the completion of live migration (step 119).

  Similarly, live migration processing is executed in the order of VM # 7, VM # 2, VM # 8, VM # 10, VM # 9, VM # 5, VM # 6, VM # 3 (step 120). When the live migration of VM # 3, which is the VM to be moved last, is completed, a notification of VM # 3 live migration completion is sent from the VM management unit 24 to the bandwidth setting unit 28 (step 121), and the bandwidth setting unit 28 The bandwidth control setting in the bandwidth control unit 32 within 30 is canceled (step 122).

  As described above, the series of live migration processes for VM # 1 to VM # 10 are all completed.

[Example 2]
A case will be described in which the virtual machine system having the configuration shown in FIG. 10 is used, the user traffic information utilization method is used as the VM movement order determination method, and the VM movement time equalization method is used as the bandwidth control method determination method. It is assumed that a plurality of user terminals are connected to the illustrated packet transfer network. It is assumed that the number of physical machines in the virtual machine system is three (physical machine # 1 to physical machine # 3) and is connected to the same switch 30. Assuming that the physical machine # 1 has 10 VMs (VM # 1 to VM # 10), VM # 1 to VM # 5 are live migrated to the physical machine # 2, and VM # 6 to VM # 10 Are migrated live to the physical machine # 3. It is assumed that the time required for memory copy for live migration (VM migration planned time) is 120 seconds. Further, it is assumed that the bandwidth of each physical link is 1 Gbit / s. Further, it is assumed that traffic flows from each VM to the user terminal, and the amount of user traffic is different for each VM. It is assumed that the memory usage and the memory change rate are uniform for each VM, the memory usage is 256 Mbytes, and the memory change rate is 10 Mbit / s.

  It is assumed that a traffic monitoring unit 33 is provided in the switch 30, and it is possible to identify which VM traffic is user traffic or VM memory copy traffic by VLAN. Each physical machine is provided with a bandwidth control unit 32, which can individually control the bandwidth of traffic for each VM and memory copy traffic.

  The flow of operation in this case is shown in FIG.

  When the VM management unit 24 in the management apparatus 20 determines to perform live migration of VM # 1 to VM # 10 in step 151, the VM management unit 24 notifies the information collection unit 25 to that effect (step 152). The information collecting unit 25 collects traffic information from the traffic monitoring unit 33 in the switch 30 (step 153), and stores the collected information in the data storage unit 24 (step 154). Next, the schedule determination unit 26 acquires traffic information and a scheduling determination algorithm from the data storage unit 23 (step 155), and schedules (migration schedule and bandwidth control schedule) according to the user traffic information utilization method and the VM movement time equalization method. The determined schedule is stored (step 156), and the determined schedule is stored in the data storage unit 24 (step 157). In the case of this example, the determined schedule is as shown in Table 9, and the movement order of the VMs is VM # 3 → VM # 8 → VM # 1 →... → VM # 10.

  The schedule determination unit 26 notifies the migration management unit 24 of the migration schedule (step 158), and notifies the bandwidth control setting unit 28 of the bandwidth control schedule (step 159). Then, since the first VM to be moved is VM # 3, the bandwidth control setting unit 28 sets the bandwidth control value described for each VM in the “bandwidth control value during movement of VM # 3” column of Table 9. The bandwidth control unit 32 in the physical machine # 1 is set so as to set the user traffic bandwidth control value of each VM (step 160).

  Thereafter, the VM management unit 24 instructs the physical machine # 1 to perform live migration of the VM # 3 to be moved first (step 161), and the VM # from the physical machine # 1 to the physical machine # 2 3 is performed (step 162), and when this memory copy is completed, notification of completion of live migration of VM # 3 is sent from the physical machine # 2 to the VM management unit 24 (step 163). The completion of the live migration of # 4 is notified from the VM management unit 24 to the bandwidth control setting unit 28 (step 164). Then, since the VM that moves second is VM # 8, the bandwidth control setting unit 28 sets the bandwidth control value described for each VM in the “bandwidth control value during movement of VM # 8” column of Table 9. Then, the bandwidth is set in the bandwidth control unit 32 in the physical machine # 1 so as to be the user traffic bandwidth control value of each VM (step 165). At this time, since the VM # 3 has already been moved from the physical machine # 1, the user traffic related to the VM # 3 is not subject to bandwidth control.

  Thereafter, the VM management unit 24 instructs the physical machine # 1 to perform the live migration of the VM # 8 to be moved second, similarly to the movement of the VM # 3 (step 166). Thus, the memory copy for the VM # 8 from the physical machine # 1 to the physical machine # 3 is performed (step 167). When the memory copy for VM # 8 is completed, a notification of completion of live migration of VM # 1 is sent from the physical machine # 3 to the VM management unit 24 (step 168), and the VM management unit 24 The bandwidth control setting unit 28 is notified of the completion of live migration (step 169).

  Similarly, live migration processing is executed in the order of VM # 1, VM # 4, VM # 2, VM # 7, VM # 6, VM # 5, VM # 9, and VM # 10 (step 170). When the live migration of VM # 10, which is the VM to be moved last, is completed, a notification of VM # 10 live migration completion is sent from the VM management unit 24 to the bandwidth setting unit 28 (step 171). The bandwidth control setting in the bandwidth control unit 32 in the machine # 1 is canceled (step 172).

  As described above, the series of live migration processes for VM # 1 to VM # 10 are all completed.

DESCRIPTION OF SYMBOLS 11, 12 Physical machine 13 Network 14 Shared storage 15 Virtualization software 16 VM management server 20 Management apparatus 21 Packet receiving part 22 Packet transmission part 23 Data storage part 24 VM management part 25 Information collection part 26 Schedule determination part 27 Schedule determination algorithm 28 Bandwidth control setting unit 30 Switch 31 VM state monitoring unit 32 Bandwidth control unit 33 Traffic monitoring unit

Claims (4)

A plurality of physical machines and a network connecting the physical machines, a virtual machine is operable on each physical machine, and a plurality of physical machines from the source physical machine to one or more physical machines Scheduling of live migration in a virtual machine system in which memory copy traffic for moving a virtual machine and other traffic different from the memory copy traffic for moving the virtual machine use a common link during live migration of the virtual machine A method of performing
Collecting memory usage and memory change rate of each virtual machine;
Based on the memory usage and the memory change rate for each virtual machine, the live migration time under the given bandwidth is in ascending order or the memory copy traffic bandwidth under the given moving time is in ascending order. The execution order of live migration in the plurality of virtual machines is such that live migration of the virtual machines whose memory change rate is 0 is performed in parallel with the live migration of other virtual machines. A stage for determining
Determining a bandwidth control amount of the other traffic during the live migration so that fluctuation from the bandwidth of the other traffic before the live migration is reduced;
Instructing the physical machine to perform live migration of the virtual machines one by one according to the execution order;
Setting the bandwidth control amount in a bandwidth control means incorporated in the physical machine or the network;
A scheduling method comprising: A plurality of physical machines and a network connecting the physical machines, a virtual machine is operable on each physical machine, and a plurality of physical machines from the source physical machine to one or more physical machines Scheduling of live migration in a virtual machine system in which memory copy traffic for moving a virtual machine and other traffic different from the memory copy traffic for moving the virtual machine use a common link during live migration of the virtual machine A method of performing
A stage for generating traffic information about the traffic for each virtual machine,
Based on the previous SL traffic information, in order input or output traffic is high for each virtual machine, the method comprising determining the execution order of the live migration in the plurality of virtual machines,
Determining a bandwidth control amount of the other traffic during the live migration so that fluctuation from the bandwidth of the other traffic before the live migration is reduced;
Instructing the physical machine to perform live migration of the virtual machines one by one according to the execution order;
Setting the bandwidth control amount in a bandwidth control means incorporated in the physical machine or the network;
A scheduling method comprising: A plurality of physical machines and a network connecting the physical machines, a virtual machine is operable on each physical machine, and a plurality of physical machines from the source physical machine to one or more physical machines Scheduling of live migration in a virtual machine system in which memory copy traffic for moving a virtual machine and other traffic different from the memory copy traffic for moving the virtual machine use a common link during live migration of the virtual machine A method of performing
Determining an execution order of live migration in the plurality of virtual machines based on at least one of information on the virtual machines and information on traffic on the network;
It is possible to complete live migration of all virtual machines in the shortest time under the condition that the movement time of each virtual machine is equal, or the amount of decrease in the bandwidth of the other traffic is within the set value. under the condition that it, determining a bandwidth control amount of the other traffic according to the respective virtual machine,
Instructing the physical machine to perform live migration of the virtual machines one by one according to the execution order;
Setting the bandwidth control amount in a bandwidth control means incorporated in the physical machine or the network;
A scheduling method comprising: A virtual machine system having a plurality of physical machines and a network connecting the physical machines, and capable of operating a virtual machine on each physical machine,
A bandwidth control unit that is incorporated in the physical machine or the network and performs bandwidth control of other traffic different from the memory copy traffic for moving the virtual machine;
A management device for controlling live migration of the virtual machine between the physical machines and instructing bandwidth control;
With
The physical machine or the network has traffic monitoring means for monitoring traffic related to each virtual machine,
The management device
An information collection unit that is provided in the management device and collects traffic information about traffic related to each virtual machine from the traffic monitoring unit;
When performing live migration of a plurality of virtual machines from a migration source physical machine , based on the traffic information collected by the information collection unit, in the order in which the input or output traffic for each virtual machine increases, in the plurality of virtual machines Schedule determination means for determining a live migration execution order, and determining a bandwidth control amount of the other traffic during the live migration so that a variation from the bandwidth of the other traffic before the live migration is reduced ,
VM management means for instructing the physical machine to perform live migration of the virtual machines one by one according to the execution order determined by the schedule determination means;
Bandwidth control setting means for setting the bandwidth control amount of the other traffic determined by the schedule determination means in the bandwidth control means;
A virtual machine system.

Images