JP6970337B2 - Virtual machine management program, virtual machine management method and virtual machine management device - Google Patents

Description

The present invention relates to a virtual machine management program, a virtual machine management method, and a virtual machine management device.

In recent years, multiple virtual machines have been deployed in each of multiple physical machines (or computers, information processing devices, etc.), and the virtual machines execute various business application programs (hereinafter simply referred to as business applications) to provide business systems. Is being done. Virtualization software (hereinafter referred to as a hypervisor) that runs on a physical machine is a hardware resource (hardware resource, or simply) such as the CPU and memory of the physical machine for multiple virtual machines deployed on the physical machine. Allocate a predetermined amount of resources out of (referred to as resources).

Then, in the virtual machine that executes the business application, the resource usage amount also changes according to the business amount that changes in the cycle of one day, for example. For example, a virtual machine that executes online processing that is frequently accessed during the day and a virtual machine that executes batch processing at night repeatedly change the resource usage to the same extent every day. However, if the resource usage of virtual machines increases due to changes in business volume or system revisions, the resources of physical machines may run short.

When such a resource shortage occurs, the management device that manages the virtual machine relocates (or migrates or migrates) a specific virtual machine from the current physical machine that has run out of resources to another physical machine with a large amount of free resources. ) To solve the resource shortage of the physical machine. The relocation of virtual machines is described in the following patent documents.

JP-A-2015-184965 Japanese Unexamined Patent Publication No. 2015-152984 International Publication No. 2013/099019

However, simply migrating a virtual machine from a physical machine with insufficient resources to another physical machine with a large amount of free resources causes the following problems. First, in the original physical machine in which the virtual machine has migrated to another physical machine, the resource usage may decrease depending on the time of day, and the resources may not be effectively utilized. Secondly, conversely, in the physical machine to which the virtual machine has migrated from another physical machine, the resource usage increases depending on the time zone, and the performance of the business system may deteriorate. Thirdly, in the second case above, if a sudden increase in demand for resources occurs, a resource shortage may occur.

Therefore, an object of the first aspect of the present disclosure is to provide a virtual machine management program, a virtual machine management method, and a virtual machine management device for relocating virtual machines in which resource usage is unlikely to increase extremely in each physical machine. To do.

The first aspect of the present embodiment stores the resource usage per unit time of the plurality of virtual machines deployed in each of the plurality of physical machines in the group of physical machines to be managed and the plurality of physical machines. ,
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time within the predetermined period of the first physical machine is similar. Select the first virtual machine with the second waveform of the normalized resource usage, which has a relatively high degree,
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. Select the second physical machine you have and
Relocating the first virtual machine from the first physical machine to the second physical machine.
A virtual machine management program that causes a computer to perform processing.

According to the first aspect, it is possible to relocate a virtual machine in which an extreme increase in resource usage is unlikely to occur.

It is a figure which shows the relationship between the management apparatus and a plurality of physical machines and virtual machines in this embodiment. It is a figure which shows the configuration example of the management apparatus 1 in this embodiment. It is a figure explaining the change of the resource usage of a physical machine. It is a figure explaining the relocation of a virtual machine. It is a figure which shows the problem in the physical machine after relocation. It is a figure explaining the resource usage of the target physical machine in 1st Embodiment. It is a figure which shows an example of the history of the resource usage of a virtual machine. It is a flowchart of the optimal placement process of a virtual machine in 1st Embodiment. It is a figure explaining the process of selecting a virtual machine to be relocated from a plurality of virtual machines running on a physical machine. It is a figure explaining the process of selecting the physical machine B to relocate the virtual machine a from a plurality of managed physical machines excluding the physical machine A. It is a figure explaining the Euclidean distance. It is a figure explaining the magnitude W of a waveform. It is a figure explaining the evaluation value ρ of the leveling of all physical machines. It is a figure which shows the example when there is a big bias in the resource usage of the whole virtual environment. It is a figure which shows the example of the case where the optimum arrangement of the virtual machine is performed by the 2nd Embodiment. It is a flowchart of the optimal placement process of a virtual machine in 2nd Embodiment. It is a figure explaining the process S12 which selects a virtual machine to be relocated from a plurality of virtual machines running in a physical machine. It is a figure explaining the process S13 which selects the physical machine of the relocation destination of the virtual machine to be relocated. It is a figure which shows the example of the whole flow chart of the concrete process of the 1st and 2nd Embodiment. It is a figure which shows the example of the confirmation screen for the user at the time of executing the relocation plan of a virtual machine. It is a figure which shows an example of the execution history of the relocation of a virtual machine. It is a figure which shows an example of the correspondence table 30 of a virtual machine and a physical machine in the database of a management apparatus, and the operation history table 32 of a virtual machine. It is a figure which shows the specific example of the resource usage of the virtual machine VM of the physical machines PM1 to PM4. It is a figure which shows the value and state required for each process in the optimal placement process (relocation process) of a virtual machine by a management device. It is a figure which shows the value and state required for each process in the optimal placement process (relocation process) of a virtual machine by a management device.

FIG. 1 is a diagram showing a relationship between a management device and a plurality of physical machines and virtual machines in the present embodiment. The management device 1 is a virtual machine management device that manages a plurality of virtual machine VMs operating on each physical machine PM of the physical machine group 2 to be managed. The physical machine group 2 to be managed has a plurality of physical machines PM_0, PM_1 and the like, and the plurality of physical machines PM_0, PM_1 and the like are communicably connected to the management device 1 via the network NW1. Further, the management device 1 is communicably connected to the user terminal devices 20 and 22 of the user who operates the virtual service system composed of virtual machines via the network NW2. A service user terminal device (not shown) that accesses the virtual service system and uses the service is also connected to the virtual machine VM so as to be able to communicate with the virtual machine VM via a network (Internet, intranet, not shown) (not shown). Will be done.

The management target physical machine group 2 is, for example, a data center, a server facility, or a management unit physical machine group deployed in the data center or the server facility. The virtual machine operating on the physical machine can be relocated (migrated) between a plurality of physical machines in the managed physical machine group 2, for example.

Each of the physical machines PM_0 and PM_1 executes the hypervisor HV and operates multiple virtual machines VM_0-VM_7. The hypervisor HV allocates some of the resources of the physical machine to each virtual machine. The resources of the physical machine are, for example, CPU and memory. Physical machine resources may include network bandwidth.

FIG. 2 is a diagram showing a configuration example of the management device 1 according to the present embodiment. The management device 1 may be either a physical machine or a virtual machine running on the physical machine. The management device has a CPU 10 connected via a bus 16, a main memory 11, and a network interface 12. Storages 13, 14 and 15, which are auxiliary memories such as HDDs and SSDs, are further connected to the bus 16, a virtual machine management program is stored in the storage 13, and a user accessed by the user terminal device is stored in the storage 14. Contains programs that control the portal site.

Further, the storage 15 stores a configuration file of a virtual machine, an operation history database of a virtual machine VM and a physical machine PM, and the like. The operation history includes, for example, hardware usage such as CPU usage, memory usage, and bandwidth usage per unit time.

The virtual machine management program has the following functions. First, the function to register the configuration file of the newly deployed virtual machine (VM registration), the function to start and shut off the virtual machine on the physical machine, and the function to suspend (operating) the virtual machine running on the physical machine (suspended). Ability to pause) and then resume (resum), migrate (relocate or migrate) a virtual machine from one physical machine to another (VM migration program), and make the virtual machine the best physical machine. It is a function to place and relocate to (VM optimal placement program).

Next, the process in which the management device executes the virtual machine management program and relocates the virtual machines among the physical machines will be outlined.

FIG. 3 is a diagram illustrating changes in the resource usage of the physical machine. In the figure, two graphs are shown, the horizontal axis is time, and the vertical axis is the resource usage of the physical machine by the virtual machine. A virtualized machine that executes a business application changes its resource usage as the business volume changes day by day. Then, when a plurality of virtual machines run on a physical machine, the total resource usage of the plurality of virtual machines also changes.

In the graph on the left side of FIG. 3, the dashed line is the resource capacity (or resource amount) R_MAX of the physical machine PM, and the total resource usage R_USE of multiple virtual machines running on that physical machine is the time as shown. It changes day by day with the passage of time. As long as the total resource usage R_USE does not exceed the resource capacity R_MAX of the physical machine, there is no problem with the operation of each virtual machine.

However, when the business volume of the business application executed by a virtual machine increases, the required total resource usage R_USE temporarily exceeds the resource capacity R_MAX that can be allocated by the physical machine, as shown in the graph on the right side of Fig. 3. The physical machine runs out of resources. In this case, the physical machine cannot allocate resources to the virtual machine in excess of the allottable resource capacity R_MAX, so the resource capacity allocated to the virtual machine is less than the required capacity, and the operation of the business application by the virtual machine. Decreases.

FIG. 4 is a diagram illustrating the rearrangement of virtual machines. The upper side of FIG. 4 shows a state in which a resource shortage of a physical machine occurs while a plurality of virtual machines VM1 to VM4 are operating on the physical machine PM. Similar to FIG. 3, the total resource usage R_USE required by the virtual machines VM1-VM4 running on the physical machine PM exceeds the resource capacity R_MAX of the physical machine, and a resource shortage occurs.

In this case, as shown in the lower part of FIG. 4, one virtual machine VM4 running on the physical machine PM that is running out of resources is physically relocated to another physical machine that has enough free resource capacity. Total resource usage of machine PM R_USE may decrease by the amount of resource usage of the relocated virtual machine VM4, and the resource shortage may be resolved.

FIG. 5 is a diagram showing problems in the physical machine after relocation. If any virtual machine of a physical machine that is simply running out of resources is simply relocated to another physical mine that has plenty of free resources (large free resource capacity), after relocation as shown in FIG. The following problems may occur in the physical machines (relocation source and relocation destination physical machines).

First, as shown by R1 in the figure, the resource usage of the physical machine from which the virtual machine is relocated after being relocated to another physical machine becomes extremely small depending on the time of day, and the resources of the physical machine Will not be used effectively. Secondly, as shown by R2 in the figure, in the physical machine to which the virtual machine is relocated, the resource usage may increase depending on the time zone, and the performance of the business system by the virtual machine may decrease. Thirdly, as shown by R2 in the figure, when the resource demand by the virtual machine suddenly increases, the resource demand may exceed the maximum resource capacity R_MAX and the resource usage R_USE may reach the maximum resource capacity R_MAX.

[First Embodiment]
FIG. 6 is a diagram illustrating a resource usage amount of a target physical machine in the first embodiment. In order to solve the problem described with reference to FIG. 5, in the first embodiment, as shown in FIG. 6, the resource usage amount R_USE of the target physical machine is sufficiently smaller than the maximum resource capacity R_MAX, and the time changes. Relocate the virtual machine so that it is leveled (or flattened) to a constant value as much as possible without changing significantly. Here, leveling (or flattening) to a constant value means that the total resource usage used by the virtual machine of the physical machine does not change significantly with the passage of time and changes to a constant value as much as possible. means. Then, all of the plurality of managed physical machines are made to have a flat resource usage R_USE waveform as shown in FIG.

In the first embodiment, as a premise, as shown in FIG. 1, the management device stores a plurality of physical machines to be managed and the operation history (history of resource usage, etc.) of virtual machines operating on the physical machines. It is saved in. A plurality of virtual machines operating on a plurality of managed physical machines constitute a managed virtual environment.

FIG. 7 is a diagram showing an example of a history of resource usage of a virtual machine. The resource usage history includes the resource usage used by the virtual machine every arbitrary unit time (for example, one hour). Examples of resources in FIG. 7 are CPU usage (clocks used per second, etc.) and main memory usage (bytes).

In the present embodiment, the resource usage is, for example, the average value or the maximum value of the resource usage in each unit time. If the maximum value is adopted as an example, if the unit time is 1 hour, the history of resource usage between 0 am and 1 am is 2.5 GHz to 3.0 Hz (4 cores) for the CPU, and 12 GB to 16 GB for the memory, the unit time. The resource usage in is 3.0Hz (4 cores) for the CPU and 16GB for the memory.

Historical data of resource usage is stored sequentially. If the cycle (predetermined period) of the resource usage waveform is 24 hours, the history data is stored sequentially every day, and if the history collection time is multiple days (for example, 5 days), the resource usage in each unit time. The average or maximum value of is adopted. If the memory resource usage from 0:00 to 1:00 AM for 5 days is 13,14,15,11,12GB, the memory usage during this time period will be either the average value of 13GB or the maximum value of 15GB. Will be done.

By summing up the resource usage of multiple virtual machines running on a physical machine, historical data of the resource usage of the physical machine can be acquired. Similarly, the resource usage of the entire virtual environment can be obtained by summing the resource usage of all virtual machines in the virtual environment or the resource usage of all physical machines.

FIG. 8 is a flowchart of the optimum placement process of the virtual machine according to the first embodiment. The management device executes the optimal placement program of the virtual machine and selects the physical machine A having the minimum free capacity from all the physical machines to be managed (S1). Here, the physical machine having the minimum free capacity is the physical machine having the smallest minimum value of the resource free capacity in a predetermined period (cycle of the waveform of the resource usage) for each unit time. For example, when FIG. 5 shows the resource usage of the physical machine, the minimum value of the free resource capacity for each unit time within a predetermined period on the horizontal axis is R2. Therefore, the minimum value R2 of a physical machine having the minimum free capacity is the minimum value among all physical machines.

The free resource capacity of a physical machine is the amount of resources obtained by subtracting the total resource usage R_USE of all virtual machines from the maximum value R_MAX of the resources possessed by the physical machine. The physical machine with the least free resources reduces the risk of running out of resources by relocating (migrating) any virtual machine running on that physical machine to another physical machine. can do. Therefore, the above physical machine A is selected as the target for relocating the virtual machine.

Next, the management device executes the virtual machine optimum placement program, and among the plurality of virtual machines running on the physical machine A, the waveform and the similarity of the resource usage within the predetermined period of the physical machine A are relative to each other. Select virtual machine a with a high, preferably the highest resource usage waveform within a given period (S2). When such virtual machine a is relocated (migrated) from physical machine A to another physical machine, the waveform of the resource usage of physical machine A is leveled so as to be flatter. Here, leveling means that the increase / decrease in the resource usage amount in each unit time within a predetermined period becomes small, and the resource usage amount becomes uniform with respect to the time axis.

FIG. 9 is a diagram illustrating a process of selecting a virtual machine to be relocated from a plurality of virtual machines operating on a physical machine. The physical machine PM_A shown in FIG. 9 is the above physical machine A. Virtual machines VM11, VM12, VM13 are running on physical machine PM_A. On the other hand, the graph in FIG. 9 shows the resource usage amount R_USE1 of the physical machine PM_A in the predetermined period Tw. In the above process S2, the management device selects the virtual machine VM13 having the waveform R_USE3 having the highest similarity to the waveform at the predetermined period Tw of the resource usage R_USE1 of the physical machine PM_A as the virtual machine a.

As shown in FIG. 9, when the virtual machine VM13 is relocated from the physical machine PM_A to another physical machine, the high waveform of the physical machine PM_A is larger because the high resource usage R_USE3 of the virtual machine VM13 is reduced. descend. On the other hand, the low waveform of the physical machine PM_A drops slightly only by reducing the low resource usage R_USE3 of the virtual machine VM13. As a result, the waveform of the resource usage R_USE2 of the physical machine PM_A after the virtual machine VM3 is relocated is leveled so that the increase / decrease of the resource usage is suppressed from the waveform R_USE1 before the relocation and becomes a more constant value. NS.

Next, the management device has a relatively low similarity to the waveform of the normal resource usage of the virtual machine a among all (plural) physical machines except the physical machine A, preferably the source having the lowest similarity. Select physical machine B with usage waveform (S3). By relocating (migrating) virtual machine a to such physical machine B, the waveform of the resource usage of physical machine B can be leveled to a more constant value. In this case, select physical machine B so that the resource usage of physical machine B does not run out of resources after relocating virtual machine a.

FIG. 10 is a diagram illustrating a process of selecting a physical machine B to relocate a virtual machine a from a plurality of managed physical machines other than the physical machine A. The physical machine PM_B shown in FIG. 10 is the above physical machine B. Virtual machines VM21 and VM22 are running on physical machine PM_B. On the other hand, the graph of FIG. 10 shows the resource usage amount R_USE11 of the physical machine PM_B in a predetermined period Tw. In the above process S3, the management device sets the physical machine VMPM_B having the waveform R_USE11 having the lowest similarity to the waveform at the predetermined period Tw of the resource usage R_USE3 of the virtual machine VM13 which is the virtual machine a to be relocated. Select B.

As shown in FIG. 10, when the virtual machine VM3 is relocated to the physical machine PM_B, the high part of the waveform of the physical machine PM_B is slightly increased by adding the low resource usage R_USE3 of the virtual machine VM3. On the other hand, where the waveform of the physical machine PM_B is low, the high resource usage R_USE3 of the virtual machine VM3 is added and the value increases significantly. As a result, the waveform of the resource usage R_USE12 of the physical machine PM_B after the virtual machine VM3 is relocated is leveled so that the increase / decrease of the resource usage is suppressed from the waveform R_USE11 before the relocation and becomes a more constant value. NS.

The management device selects the physical machine B having the waveform of the free resource capacity having the highest similarity to the waveform R_USE3 of the resource usage of the virtual machine VM3 to be relocated (the waveform of the waveform R_USE11 turned upside down). Is also good. A high degree of similarity with the waveform of free resource capacity is almost equivalent to a low degree of similarity with the waveform of resource usage.

[Waveform similarity]
Next, the similarity of the above waveforms will be described. Since multiple virtual machines are running on a physical machine, the scale differs between the resource usage of one virtual machine and the resource usage of a physical machine running multiple virtual machines. Therefore, in the present embodiment, the resource usage of the virtual machine and the physical machine is normalized in advance, and the Euclidean distance between the waveforms of the normalized resource usage is measured to determine the similarity. It is judged that the closer the Euclidean distance is, the higher the similarity is.

Resource usage normalization can be calculated as follows:
First, the average value (x _av , y _{av ) of the resource usage (x t} , y _t ) at each unit time in each waveform cycle of the physical machine or virtual machine x, y to be compared is calculated by the following formula. Calculate with. In the following formula, t is time and T is the period of the waveform (predetermined period). The time t = 0 to t = T means from the beginning to the end of the waveform cycle.

Second, the standard deviation (σ _x , σ _y ) of the resource usage (x _t , y _t ) is calculated by the following formula.

The standard deviation σ is the square root of the cumulative value of the square of the difference between the resource usage and the average value, and indicates the thickness of the distribution on the left and right of the average value of the resource usage distribution. That is, it shows the magnitude of the waveform of the resource usage centered on the average value in the vertical direction.

_{Third, the resource usage (x t} , y _t ) at each unit time in the waveform cycle of each physical machine or virtual machine to be compared is normalized by the following formula, and the resource usage data (x _nt , y). Convert to _nt).

That is, the difference between the resource usage and the average value in each unit time is divided by the standard deviation to obtain the normalized resource usage. This normalized resource usage divides the waveform of the difference between the resource usage and the average value by the standard deviation, which is the thickness of the distribution, so the size of the waveform of the normalized resource usage is approximately -1. It will be between +1. As a result, the magnitudes (scales) of the waveforms of the normalized resource usage of the physical machine and the virtual machine are almost the same.

Fourth, the Euclidean distance d between the waveforms of the normalized resource usage to be compared is calculated by the following formula. The Euclidean distance d is the square root of the cumulative value of the square of the difference between the data at each unit time of both normalized resource usage.

FIG. 11 is a diagram illustrating the Euclidean distance. On the left side of FIG. 11, the waveform of the normalized resource usage to be compared is shown. The Euclidean distance d of the two waveforms can be obtained by summing the squares of the differences in the normalized resource usage at each unit time from time t = 0 to t = T to obtain the square root.

As shown on the right side of FIG. 11, the Euclidean distance d is obtained by plotting the values of the waveform x and the waveform y at the unit times t1 and t2 on the two-dimensional coordinates of the unit times t1 and t2, and the coordinate points of the waveform X (x _t1). , X _t2 ) and the coordinate point (y _t1 , y _t2 ) of the waveform Y are the same as the following formula calculated by the Pythagorean theorem.

Returning to FIG. 8, the management device executes the optimum placement program of the virtual machine, simulates a plan for relocating the virtual machine a from the physical machine A to the physical machine B, and relocates the physical machines A and B. Predict resource usage (S4). Then, the management device determines the end of the relocation, executes the relocation of the virtual machine if the condition of the end determination is true, and repeats the processes S2-S4 if the condition of the end determination is false (S5). To relocate the virtual machines in process S5, the relocation plans for all virtual machines selected in process S2-S4, which are repeated multiple times, are executed in order.

[Determining the necessity of optimal placement and determining the end of optimal placement]
Next, a method for evaluating the leveling of the waveform of the resource usage of the physical machine in the present embodiment will be described. Using the evaluation value ρ of the leveling described below, the start judgment and the end judgment of the optimum placement process of the virtual machine are performed.

The fact that the waveform is leveled simply means that the size of the waveform is small. Therefore, the waveform size (evaluation value of waveform leveling) W is calculated by the following formula.

FIG. 12 is a diagram illustrating the magnitude W of the above waveform. The waveform size W according to the above formula is the same as the length of the Euclidean distance between the resource usage waveform and the average value waveform. That is, the Euclidean distance between the waveform of _{the resource usage x t and} the waveform of the average value x _av shown in FIG. 12 is the square of the difference between the _{resource usage x t} and the average value x _{av at each unit time.} The square root of the cumulative value.

Considering that there is a certain amount of bias (waveform size) in the resource usage per unit time in the entire virtual environment, the evaluation value ρ for leveling of all physical machines is as shown in the following formula. be.

FIG. 13 is a diagram illustrating an evaluation value ρ for leveling of all physical machines. An example _{of the resource usage wall} of the entire virtual environment is shown on the left side of FIG. This is the sum of the resource usage of all physical machines in the virtual environment. On the other hand, on the right side of FIG. 13, _{an example of resource usage W p1 to} W _p4 of each of the four physical machines in the virtual environment is shown. The resource usage waveforms of each physical machine are different in magnitude and polarity. Therefore, the total size W _p1 to _W-pA waveforms of all physical machines is larger than the sum W _all resource usage of all physical machines in the virtual environment.

Then, the evaluation value ρ of the leveling of all physical machines by the above formula is the sum of the waveform sizes of the resource usage of each of all physical machines in the entire virtual environment (W _all = W _p1 + W _p2 + ・ ・ +). W _pA) is a ratio obtained by dividing the virtual sum of environmental overall resource usage (total resource usage of all physical machines) W _all the. The reason for this is as follows. Relocating the virtual machine to any physical machine does not change _{the resource usage waveform Wall of the entire virtual environment.} However, if the placement of the virtual machine on the physical machine is optimized, the size of the waveform of each physical machine can be made more constant, that is, leveled, and the evaluation value ρ of the leveling in that case becomes smaller. On the other hand, if the virtual machine is improperly arranged on the physical machine, the size of the waveform of each physical machine is not leveled, and the waveform of some physical machines becomes large. In that case, the evaluation value ρ for leveling becomes large. This is the evaluation value ρ for leveling of all physical machines.

In the present embodiment, the start condition and the end condition of the virtual machine optimum placement process of whether or not the virtual machine should be rearranged in the virtual environment are determined based on the evaluation value ρ of the above leveling. For example, the start condition is that the leveling evaluation value ρ is larger than the starting reference value ρx, or the current index is worse (increased) than the leveling evaluation value ρ in the previous optimum placement process. do. Further, the first termination condition is that the leveling evaluation value ρ becomes smaller than the reference value for termination, or that the reduction of the leveling evaluation value ρ has stopped.

As the second termination condition, it is assumed that the increase in the minimum free space of the physical machine has stopped in the process of repeatedly relocating the virtual machine. When the virtual machine relocation process is performed, as shown in FIG. 9, the minimum free space of the physical machine A increases. As described above, the minimum free space of a physical machine is the minimum value of the free resource space of each physical machine for each unit time, which is the smallest of all physical machines.

By increasing this minimum free space, it is possible to suppress the possibility of exceeding the available resource capacity (maximum resource capacity) of the physical machine A even if the business volume of the business application suddenly increases. Therefore, as the second termination condition, it is assumed that the increase in the minimum free capacity of the physical machine has stopped in the process of repeatedly relocating the virtual machine. This is because it is unlikely that the minimum free capacity will increase even if the rearrangement process is performed further.

The first end condition and the second end condition are, for example, determined by whether or not one of the end conditions is true, whether or not both end conditions are true, and the first end condition. Various combinations can be applied, such as giving priority to the condition and using the second end condition as an option.

[Execution timing of virtual machine relocation]
As shown in FIG. 8, in the virtual machine relocation process, a virtual machine relocation plan is detected, a simulation thereof is performed, and an end determination is made as to whether or not the virtual machine relocation is necessary. Then, when the end condition is satisfied in the end judgment, the virtual machine relocation plan is executed at once. The reason for this is that when the first end condition (the decrease in ρ stops) and the second end condition (the increase in the minimum free space stops) are combined as the end conditions for the optimal placement process of the virtual machine, the ρ When it is detected that the decrease has stopped and the increase in the minimum free capacity has stopped, ρ will start to increase or the minimum free capacity will start to decrease when the latest relocation plan is executed. Therefore, it may be preferable to execute the previous relocation plan rather than the latest relocation plan.

For example, the first end condition is set to end when ρ increases even once, and the second end condition is set to end when the minimum empty capacity decreases twice in a row, and either the first or second end condition is set. It is assumed that the end condition of the optimum placement process is that the value is true. In this case, if the first end condition (ends when ρ increases even once) and the second end condition (ends when the minimum empty capacity decreases twice in a row) become true at the same time, the immediately preceding relocation plan By executing the relocation plan two times before, the minimum free space of the physical machine can be maximized. Alternatively, by executing the relocation plan one time before the previous relocation plan, the leveling of the resource usage of the physical machine can be made the most advanced. Which one to select depends on the evaluation criteria of the optimal placement process of the virtual machine. The termination condition will be described again later.

[Execution timing of optimal placement processing of virtual machines]
For example, the following three execution timings can be considered as the execution timing for performing the optimum placement processing of virtual machines and leveling the resource usage of all physical machines. First, it is executed periodically at regular time intervals. This fixed time interval is, for example, weekly or monthly.

Secondly, it is executed when the virtual machines are rearranged by auto-scale-out or the like and the configuration of the virtual machines operating on each physical machine changes. Autoscale-out detects when a physical machine's resource usage is approaching its maximum and automatically relocates (migrates) some virtual machines running on that physical machine to another physical machine. This is a function of the hypervisor.

Third, the management device detects a resource shortage such as exhaustion of free resources of the physical machine.

In the present embodiment, the optimum placement process of the virtual machine is started at the above execution timing, and if the start condition of the above-mentioned optimum placement process is satisfied, the optimum placement process is executed. Then, if the end condition is satisfied in the process of the optimum placement process, the optimum placement process is terminated.

[Second Embodiment]
If there is a large bias, such as the resource usage of the entire virtual environment becoming very large in one time zone and very small in another time zone, try to flatten, that is, level the resource usage waveform of the physical machine. Then, the bias is concentrated on other physical machines. Therefore, in the second embodiment, the virtual machines are rearranged so that the waveform of the resource usage of each physical machine is close to the waveform of the resource usage of the entire virtual environment (target waveform). With such a rearrangement, the waveform of the resource usage of all physical machines will be balanced with a waveform close to the target waveform.

FIG. 14 is a diagram showing an example when there is a large bias in the resource usage of the entire virtual environment. The resource usage of the entire virtual environment is shown on the left side of FIG. According to this, the increase / decrease in the resource usage of the entire virtual environment is extremely biased depending on the time zone. In this case, when the processes S2 and S3 of the first embodiment are executed, the waveform of the resource usage of the physical machines PM1, PM2 and PM3 is similar to the waveform of the resource usage of each physical machine shown on the right side of FIG. Is flattened, but the waveform of the resource usage of the physical machine PM4 changes significantly by that amount.

In this way, the idea of flattening, that is, leveling the resource usage for each physical machine by the processes S2 and S3 of the first embodiment is not always the optimum method, and the entire virtualized environment. It is necessary to adopt a method that allows a certain amount of bias in resource usage.

FIG. 15 is a diagram showing an example in which the virtual machine is optimally arranged according to the second embodiment. In the second embodiment, the virtual machines are rearranged so that the waveforms of the resource usages of the physical machines PM1 to PM4 are close to the waveforms of the resource usages (target waveform) of the entire virtual environment. As a result, the bias of the resource usage waveform of the entire virtual environment is reduced to the extent that it is distributed to the resource usage waveforms of all physical machines, and the waveforms of all physical machines are flattened to a realistic level. Can be leveled. As shown, the waveforms of all physical machines are each leveled with a bias that reduces the degree of bias of the waveform in the entire virtual environment.

FIG. 16 is a flowchart of the optimum placement process of the virtual machine according to the second embodiment. The management device executes the optimal placement program for virtual machines, and the minimum value of the free resource capacity per unit time (cycle of resource usage waveform) is the highest among all the physical machines to be managed. Select the small physical machine A (S11). This process S11 is the same as the process S1 of FIG.

Next, the management device is _relative to the difference waveform (x _{nt --x all_nt} ) obtained by subtracting the normalized resource usage waveform x _{all_nt of the entire virtual environment from the normalized resource usage waveform x nt of physical machine A.} Select virtual machine a with a waveform of normalized resource usage that has a high degree of similarity, preferably the highest degree of similarity (S12). By relocating (migrating) such virtual machine a from physical machine A to another physical machine, the waveform x _nt of the resource usage of physical machine A can be made closer to the waveform of the target waveform x _{all_nt.}

FIG. 17 is a diagram illustrating a process S12 for selecting a virtual machine to be relocated from a plurality of virtual machines operating on a physical machine. _{The management device obtains} the waveform x all_nt of the normalized resource usage of the entire virtual environment _{and the waveform x nt} of the normalized resource usage of all virtual machines running on the physical machine, and the difference waveform (x _nt -x _{all_nt} ). And select the virtual machine a that has the waveform of the normalized resource usage with the highest similarity. Since the scale difference between the resource usage waveform of the entire virtual environment and the resource usage waveform of the virtual machine is large, the normalized resource usage is similar to the similarity of the waveforms in the first embodiment. In the waveform, it is determined that the smaller the Euclidean distance between the two waveforms, the higher the similarity.

Next, the management device calculates the difference waveform (x _{all_nt --x nt ) obtained by subtracting the normalized resource usage x nt} of the physical machines other than physical machine A from the normalized resource usage waveform x _{all_nt of the entire virtual environment.} do. Then, the management device selects the physical machine B having a difference waveform having a relatively high similarity, preferably the highest similarity, with the waveform xnt waveform of the normalized resource usage of the virtual machine a (S13). .. By relocating (migrating) the virtual machine a to such a physical machine B, the waveform of the physical machine B can be brought closer _{to the target waveform x all_nt.}

FIG. 18 is a diagram illustrating a process S13 for selecting a physical machine to be relocated to a virtual machine to be relocated. The management device calculates the difference waveform (x _{all_nt --x nt} ), and places the physical machine B, which has the most similar difference waveform to the waveform xnt of the virtual machine a to be relocated, to the destination of the relocation of the virtual machine a. Select for physical machine. Here, it should be noted that the direction of the difference calculation of the difference waveform is opposite to that in the case of FIG. In the case of FIG. 17, from the waveform xnt physical machine A differential waveform - Removal of the waveform of the virtual machine a most similar to the _{(x nt x all_nt) (x} nt - (x nt - x all_nt) = x all_nt), _Get closer to the target waveform x all_nt. On the other hand, in the case of FIG. 18, when the waveform of the virtual machine a most similar to the _{difference waveform (x all_nt --x nt} ) is added to the waveform x _{nt of the physical machine B, (x nt} + (x _{all_nt --x nt} ) = x _{all_nt ), again} approaching the target waveform x all_nt.

Returning to FIG. 16, the management device simulates the idea of relocating the virtual machine a from the physical machine A to the physical machine B, and predicts the resource usage of the physical machines A and B after the relocation (S14). Then, the management device determines the end of the relocation, executes the relocation of the virtual machine if the condition of the end determination is true, and repeats the processes S2-S4 if the condition of the end determination is false (S15). These processes S14 and S15 are the same as the processes S4 and S5 in FIG.

[Overall flowchart example]
FIG. 19 is a diagram showing an example of an overall flowchart of specific processing according to the first and second embodiments. First, an initial value, for example, ρx = 0, is set in _{the reference value ρ x} of the evaluation value ρ of the leveling of all physical machines. The start condition is that the evaluation value of the current standardization exceeds the reference value ρx, and the end condition is that the evaluation value ρ of the leveling of the first end condition has increased and the minimum empty of the second end condition. It is assumed that one of the cases where the capacity X is lowered twice in a row becomes true.

The management device executes the optimal placement program of the virtual machine to determine whether or not it matches the timing of executing the resource usage leveling process of the physical machine of the first embodiment (S21). For example, the necessity of leveling processing (the above-mentioned start determination) is determined (S22). Examples of execution timing are the above-mentioned three examples. Further, the necessity determination of the leveling process is whether or not the above-mentioned current leveling evaluation value ρ is larger than the _{reference value ρ x.}

The necessity of the leveling process, which is the start condition, is mainly determined by _{substituting the evaluation value of the leveling obtained by the previous leveling process into the reference value ρ x} , and then the evaluation value deteriorates and the current leveling is performed. It is a judgment as to whether or not the evaluation value ρ of is ρ> ρ _x. If it is worse (True of S22), the optimal placement process of virtual machines S23-S34 is executed to level the resource usage of all physical machines. If it does not deteriorate (False of S22), it ends without performing the leveling process. Further, the first time is a judgment as to whether or not the evaluation value ρ of the current leveling is _{larger than the initial value ρ x} = 0 of the reference value, and the judgment result is always True.

The necessity judgment (judgment of the start condition) of the leveling process is performed based on whether the current minimum free space is larger than the predetermined reference value or whether the current minimum free space is worse than the previous relocation. Is also good. Further, it may be performed depending on whether or not either the evaluation value ρ for leveling or the minimum empty capacity matches the above conditions. Alternatively, it may be performed depending on whether or not both the evaluation value ρ for leveling and the minimum empty capacity match the above conditions.

Then, the management device sets the number of rearrangements i to the initial value 0 (S23). In addition, the current value is substituted for the leveling evaluation value ρi at the number of rearrangements i, and the current minimum empty capacity is substituted for the minimum empty capacity Xi (S24). Up to this point, it is a series of initialization processes.

Next, the management device generates the relocation plan i + 1 (first time i + 1 = 1) th relocation plan (S25). This process S25 is either the process S1, S2, S3 of FIG. 8 of the first embodiment or the process S11, S12, S13 of FIG. 16 of the second embodiment. Regardless of which process is selected, the management device generates a relocation plan of "relocating (migrating) the virtual machine a of the physical machine A to the physical machine B" and adds it to the relocation data.

As the process S25, the waveform of the resource usage of the physical machine of the first embodiment is leveled (flattened), or the waveform of the resource usage of the physical machine of the second embodiment is used for the entire virtual environment. Which of the rearrangements to bring closer to the waveform of is selected, for example, based on the nature of the waveform of the entire virtual environment. That is, as shown in FIG. 14, when the deviation of the waveform of the entire virtual environment is smaller than the reference value, the first embodiment is selected, and when it is larger than the reference value, the second embodiment is selected.

Then, the management device simulates the relocation plan generated in S25, calculates the evaluation value ρ of the leveling when the relocation plan up to the number of relocations i + 1 is executed, and relocates the number of times i + 1. Substitute in the evaluation value ρ _{i + 1 at the second.} Further, the management device calculates the minimum free space when the execution is performed up to the number of relocations _{i + 1,} and substitutes it into the minimum free capacity X i + 1 at the number of relocations i + 1 (S26).

Next, the management device determines whether or not to end the leveling process (determination of the end condition) when the number of rearrangements is i + 1 (S27, S28). In the determination of the end condition, in the management device, the evaluation value ρ _{i + 1} of the leveling when the number of relocations i + 1 is executed is larger than _{the evaluation value ρ i} of the leveling when the number of relocations i is executed. Or, determine ρ _{i + 1} > ρ _i (S27). That is, when the leveling process is started, the waveform is leveled (flattened), and the evaluation value ρ of the leveling gradually decreases, but the evaluation value ρ increases as the leveling progresses to some extent. Therefore, the management device decides to continue the leveling process if the evaluation value ρ of the leveling is lowered (False of S27), and generates a relocation plan with the next relocation number i + 2. However, when it rises, it decides to stop the continuation of the leveling process (True of S27).

Further, in the case of False in S27, the management device further determines whether or not _{X i} -1> X _i and X _{i + 1} > X _{i are true for i ≧ 1 and the minimum free capacity (S28).} .. That is, when the leveling process is started, the minimum free space of the physical machine having the minimum free capacity increases as shown in FIG. 9, but when the leveling progresses to some extent, the minimum free capacity decreases. Therefore, the management device decides to stop the continuation of the leveling process when the minimum empty capacity Xi decreases twice in a row (True of S28), and unless the minimum empty capacity decreases twice in a row. Decide to continue the leveling process (S28 False). As for the judgment of whether or not it has decreased twice in a row, the minimum empty capacity increases when the leveling process is started, so if it decreases twice in a row, the increase in the minimum empty capacity will increase even if the leveling process is performed further. Because it cannot be expected. The two consecutive times may be three consecutive times or more consecutively, or may be decreased only once.

If both S27 and S28 are false, the management device increments the number of relocations to i = i + 1 (S29), and repeats the processes S25-S28. If either S27 or S28 is true, the management unit executes the relocation plan up to the number of relocations at which the minimum optimized capacity is maximized (S30).

That is, the management device detects the number of relocations i at which the minimum free capacity Xi becomes the maximum from the number of relocations 0 to i, and assigns it to the variable α (S30). Then, unless the variable α is 0 (False in S31), the management device executes the relocation plan up to the αth number of relocations (S32).

FIG. 20 is a diagram showing an example of a confirmation screen for a user when executing a virtual machine relocation plan. During the optimal placement process of the virtual machine, the management device sends a user confirmation screen as shown in FIG. 20 to the user terminal and obtains the user's consent before executing the relocation plan up to the αth number of relocations. It is preferable to do so. On this confirmation screen, the VM guest name (name of virtual machine a) to be relocated in the relocation plan, the current VM host name (name of physical machine A), and the VM host name after movement (name of physical machine B). Is shown. The user confirms the necessity of relocating the virtual machine on this confirmation screen, and returns either acceptance or refusal to the management device.

Then, the management device _{substitutes the leveling evaluation value ρ α-1} at the number of rearrangements α-1 _{into the reference value ρ x} of the leveling evaluation value (ρ _x = ρ _α-1 ), and this virtual machine The optimum placement process (leveling process) is completed. When the variable α is α = 0, it means that the minimum free space has never increased since the 0th relocation, so the management device displays a warning that optimization is not possible (S34).

After that, when the leveling execution timing comes (True of S21), the management device executes the processes S22-S33. In this way, each time the leveling execution timing comes, if the leveling necessity determination is true, the optimum placement process (leveling process) by the virtual machine relocation process is executed.

FIG. 21 is a diagram showing an example of the execution history of the relocation of the virtual machine. When the management device executes the relocation of the virtual machine (S32), as shown in FIG. 21, the management device executes the execution history, the execution date and time, the virtual machine name, the source physical machine name, the destination physical machine example, and the leveling evaluation value ρ. Changes in, changes in minimum empty capacity, etc. are recorded.

In FIG. 19, as the end determination, the repetition of the processes S25 and S26 is terminated when either S27 or S28 becomes true, but the repetition of the processes S25 and S26 is terminated when both S27 and S28 become true. good. In that case, since it is rare that S27 and S28 become true at the same time, the repetition of the processes S25 and S26 may be terminated when S27 and S28 become true at any of the rearrangement counts.

[Specific example of optimal placement of virtual machines]
Next, a specific example of the optimum placement of the virtual machine in the case of the first embodiment is shown. First, the management device keeps the correspondence table between the virtual machine and the physical machine and the operation history table of the virtual machine in the database.

FIG. 22 is a diagram showing an example of the correspondence table 30 between the virtual machine and the physical machine in the database of the management device and the operation history table 32 of the virtual machine. The correspondence table 30 between the virtual machine and the physical machine stores the correspondence between all the virtual machines and the physical machine in the virtual environment. Further, in the operation history table 32 of the virtual machines, a log of resource usage such as CPU usage and memory usage for all virtual machines is stored together with a time stamp.

FIG. 23 is a diagram showing a specific example of the resource usage of the virtual machine VMs of the physical machines PM1 to PM4. The resource usage is an average value or a maximum value in each unit time of hours 0 to 23. Further, the maximum usable resource amount of each physical machine PM is 100. In the chart of FIG. 23, the total resource usage “77” in the unit time 15 of the physical machine PM2 is the maximum, and the empty resource capacity in that case is 100-77 = 33, which is the minimum value among the four physical machines. ..

24 and 25 are diagrams showing values and states required in each process in the optimal placement process (relocation process) of the virtual machine by the management device.

The management device first equalizes the size of the resource usage waveform of each physical machine to the size of the waveform size of the resource usage of the entire virtual environment based on the resource usage history of FIG. 23. The evaluation value ρ ₀ (0 means the number of rearrangements i = 0) is obtained, and the necessity of the leveling process is determined (S22). Chart 40 of FIG. 24 shows the magnitude of the waveform of the resource usage of the physical machines PM1-PM4. Based on this, the management device _{calculates the evaluation value ρ 0} = 1.75. At the first time, the evaluation value ρ is _{larger than the initial value ρ x} = 0 (ρ ₀ > ρ _x ), so the management device decides to execute the leveling process (True in S22).

Next, the management device selects the physical machine having the smallest minimum resource free space per unit time (minimum free space physical machine) among all physical machines based on the resource usage history in FIG. 23. (S1 of S25). As a result, it is detected that it is a physical machine PM2. As described above, the physical machine PM2 has a resource usage amount of 77.0 in a unit time of 15, and a resource free space of 23.0, which is the smallest.

Therefore, the management device searches for a virtual machine having the highest similarity to the waveform of the normalized resource usage of the physical machine PM among the multiple virtual machines VM_b1 to VM_d6 running on the physical machine PM2 (S2 of S25). ). In Chart 42 of FIG. 24, the degree of similarity (actually, the Euclidean distance) with the waveform of each of the physical machines PM2 of the virtual machines VM_b1 to VM_b6 is shown. According to this, since the virtual machine VM_b2 has the shortest Euclidean distance from the waveform of the physical machine PM2 and the highest similarity of the waveforms, the management device selects the virtual machine VM_b2 as the virtual machine a to be relocated.

Furthermore, the management device is the most dissimilar to the waveform of the normalized resource usage of the virtual machine a to be relocated from among the physical machines PM1, PM3, PM4 other than the physical machine PM2 (the longest Euclidean distance). Select the physical machine of (S3 of S25). FIG. 44 of FIG. 24 shows the degree of similarity (actually, the Euclidean distance) of the virtual machine VM_b2 to be relocated with the physical machines PM1, PM3, and PM4, respectively. According to this, the physical machine PM3 has the longest Euclidean distance and is the most dissimilar, so the management device selects the physical machine PM3 as the relocation destination of the virtual machine VM_b2.

Next, the management device simulates a relocation (migration) plan for the virtual machine VM_b2 from the physical machine PM2 to PM3 (S26).

FIG. 25 is a diagram showing data of resource usage of the virtual machine and the physical machine after the virtual machine VM_b2 is relocated from the physical machine PM2 to the physical machine PM3. As shown in Chart 46 of FIG. 25, the resource usage of the physical machines PM2 and PM3 changes due to the rearrangement of the virtual machine VM_b2. That is, Chart 46 is the result of simulating the rearrangement plan.

Therefore, the management device executes the end determinations S27 and S28. First, when the first end determination S27 is executed, as shown in Chart 48 of FIG. 25, the size of the resource usage waveform of the physical machine PM2 decreases from 246.99 to 158.86, and the resource usage waveform of the physical machine PM3. The size of is reduced from 364.59 to 332.46. As a result, the leveling evaluation value ρ _{1 when the} number of rearrangements i = 1 decreases to ρ ₁ = 1.53 (ρ ₀ = 1.75). Therefore, the management device determines that the end determination S27 is False and that the relocation process is continued.

On the other hand, since the second end determination S28 determines the data of the minimum empty capacity Xi due to the relocation for two times, the management device does not make the determination in the first relocation process, and the end determination S28 also re-determines. Judged as continuation of placement processing.

Chart 50 of FIG. 25 shows the relocation simulation results when the number of relocations i = 0, 1 and the relocation simulation results after the number of relocations i = 2. According to this, the management device determined that the relocation process was completed in the first end determination S27 because the leveling evaluation value ρ increased from 1.16 to 1.17 when the number of relocations i = 4th, and the relocation candidate was determined. Is up to one of the number of rearrangements i = 1 to 3. The minimum free capacity is the largest among the number of relocations i = 1 to 3, i = 34.6 of the second physical machine PM1. Therefore, the management device executes the relocation plan (relocation plan with i = 1,2) up to the number of relocations i = 2 (S32), and ends the leveling process.

As described above, according to the present embodiment, it is possible to level (flatten) the waveform of the resource usage of all physical machines under the virtual environment as much as possible. Therefore, even if the resource usage of the virtual machine suddenly increases, the possibility of running out of resources can be suppressed, and the resource usage of the physical machine does not decrease excessively, and the resources of the physical machine can be efficiently transferred to the virtual machine. Can be used.

The above embodiments are summarized in the following appendix.

(Appendix 1)
The resource usage of each of the plurality of virtual machines deployed in each of the plurality of physical machines in the managed physical machine group and the plurality of physical machines per unit time is stored.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time within the predetermined period of the first physical machine is similar. Select the first virtual machine with the second waveform of the normalized resource usage, which has a relatively high degree,
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. Select the second physical machine you have and
Relocating the first virtual machine from the first physical machine to the second physical machine.
A virtual machine management program that lets a computer perform processing.

(Appendix 2)
The normalized resource usage is the difference between the resource usage per unit time within the predetermined period and the average value of the resource usage per unit time, and the resource usage per unit time within the predetermined period. The virtual machine management program according to Appendix 1, which is calculated by dividing by the standard deviation of.

(Appendix 3)
The closer the Euclidean distance, which is the square root of the square root of the difference between the first waveform and the second waveform, or the second waveform and the third waveform for each unit time, the more similar the similarity is. The virtual machine management program according to Appendix 1, which is determined not to be very similar.

(Appendix 4)
Stores the resource usage of multiple virtual machines deployed in each of the multiple physical machines in the managed physical machine group and the plurality of physical machines per unit time.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the plurality of physics are obtained from the normalized resource usage amount obtained by normalizing the resource usage amount per unit time within the predetermined period of the first physical machine. The first difference waveform obtained by subtracting the normalized total resource usage obtained by normalizing the total resource usage per unit time within the predetermined period of the machine, and the normalized resource usage having a relatively high similarity are the first. Select the first virtual machine with 2 waveforms,
Of the plurality of physical machines excluding the first physical machine, a second physical machine having a second difference waveform having a relatively high similarity to the second waveform of the normalized resource usage of the first virtual machine is used. The second differential waveform is selected and obtained by subtracting the normalized resource usage of the second physical machine from the normalized total resource usage.
Relocating the first virtual machine from the first physical machine to the second physical machine.
A virtual machine management program that lets a computer perform processing.

(Appendix 5)
The normalized resource usage is the difference between the resource usage per unit time within the predetermined period and the average value of the resource usage per unit time, and the resource usage per unit time within the predetermined period. Calculated by dividing by the standard deviation of
The normalized total resource usage is the difference between the total resource usage per unit time within the predetermined period and the average value of the total resource usage per unit time, and the difference is obtained for each unit time within the predetermined period. The virtual machine management program according to Appendix 4, which is calculated by dividing by the standard deviation of the total resource usage.

(Appendix 6)
The closer the Euclidean distance, which is the square root of the square root of the square of the difference between the first difference waveform and the second waveform, or between the second waveform and the second difference waveform, the more similar the similarity is. , The virtual machine management program according to Appendix 4, which determines that the distance is not similar.

(Appendix 7)
The process further
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
The sum of the first waveform sizes, which is the Euclidean distance between the resource usage for each unit time within the predetermined period of each of the plurality of physical machines and the average value of the resource usage, calculated based on the calculated resource usage. The ratio of the value to the second waveform size, which is the Euclidean distance between the total resource usage of the plurality of physical machines per unit time within the predetermined period and the average value of the total resource usage, is in a predetermined state. Make the first judgment as to whether or not it will decrease,
It has the process of repeating the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage until the first determination becomes true. The virtual machine management program according to Appendix 1 or 4.

(Appendix 8)
The process further
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
The first minimum value of the free resource amount of the physical machine for each unit time calculated based on the calculated resource usage amount is the first minimum value of the smallest physical machine among the plurality of physical machines. 2 Make a second judgment as to whether or not the minimum value changes from rising to falling.
It has the process of repeating the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage until the second determination becomes true. The virtual machine management program according to Appendix 1 or 4.

(Appendix 9)
The process further
The first minimum value of the free resource amount of the physical machine for each unit time calculated based on the calculated resource usage amount is the first minimum value of the smallest physical machine among the plurality of physical machines. 2 Make a second judgment as to whether or not the minimum value changes from rising to falling.
Until the second determination becomes true in addition to the first determination, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage amount. The virtual machine management program according to Appendix 7, wherein the above is repeated.

(Appendix 10)
The virtual machine management program according to Appendix 1 or 4, wherein the relocation process is repeatedly executed up to the number of repetitions that reaches the maximum value among the second minimum values for each of the plurality of repeated repetitions.

(Appendix 11)
The resource usage of each of the plurality of virtual machines deployed in each of the plurality of physical machines in the managed physical machine group and the plurality of physical machines per unit time is stored.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time within the predetermined period of the first physical machine is similar. Select the first virtual machine with the second waveform of the normalized resource usage, which has a relatively high degree,
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. Select the second physical machine you have and
Relocating the first virtual machine from the first physical machine to the second physical machine.
A virtual machine management method that has processing.

(Appendix 12)
Stores the resource usage of multiple virtual machines deployed in each of the multiple physical machines in the managed physical machine group and the plurality of physical machines per unit time.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the plurality of physics are obtained from the normalized resource usage amount obtained by normalizing the resource usage amount per unit time within the predetermined period of the first physical machine. The first difference waveform obtained by subtracting the normalized total resource usage obtained by normalizing the total resource usage per unit time within the predetermined period of the machine, and the normalized resource usage having a relatively high similarity are the first. Select the first virtual machine with 2 waveforms,
Of the plurality of physical machines excluding the first physical machine, a second difference waveform having a relatively high similarity to the second waveform of the normalized resource usage of the first virtual machine (using the normalized total resource). Select the second physical machine (the second difference waveform obtained by subtracting the normalized resource usage of the second physical machine from the amount), and the second difference waveform is obtained from the normalized total resource usage of the second physical machine. Obtained by subtracting the normalized resource usage
Relocating the first virtual machine from the first physical machine to the second physical machine.
A virtual machine management method that has processing.

(Appendix 13)
A resource usage storage means for storing resource usage per unit time of a plurality of virtual machines deployed in each of a plurality of physical machines in a group of managed physical machines and the plurality of physical machines.
A first physical machine selection means for selecting the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period among the plurality of physical machines.
Among the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time of the first physical machine within the predetermined period is similar. A first virtual machine selection means for selecting a first virtual machine having a second waveform of the normalized resource usage having a relatively high degree, and
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. A second physical machine selection means for selecting a second physical machine to have,
A virtual machine management device comprising a relocation means for relocating the first virtual machine from the first physical machine to the second physical machine.

(Appendix 14)
A resource usage storage means for storing resource usage per unit time of a plurality of virtual machines deployed in each of a plurality of physical machines in a group of managed physical machines and the plurality of physical machines.
A first physical machine selection means for selecting the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period among the plurality of physical machines.
Of the plurality of virtual machines operating on the first physical machine, the plurality of physics are obtained from the normalized resource usage amount obtained by normalizing the resource usage amount per unit time within the predetermined period of the first physical machine. The first difference waveform obtained by subtracting the normalized total resource usage obtained by normalizing the total resource usage per unit time within the predetermined period of the machine, and the normalized resource usage having a relatively high similarity are the first. The first virtual machine selection means for selecting the first virtual machine having two waveforms,
Of the plurality of physical machines excluding the first physical machine, a second difference waveform having a relatively high similarity to the second waveform of the normalized resource usage of the first virtual machine (using the normalized total resource). A second physical machine selection means for selecting a second physical machine (a second difference waveform obtained by subtracting the normalized resource usage of the second physical machine from the amount), and the second difference waveform uses the normalized total resource. Obtained by subtracting the normalized resource usage of the second physical machine from the quantity.
A virtual machine management device comprising a relocation means for relocating the first virtual machine from the first physical machine to the second physical machine.

PM: Physical machines, computers, information processing equipment
VM: Virtual machine
HV: Hypervisor, virtualization program
CPU: Processor
MEM: Main memory, memory
NW: Network 1: Management device 2: Management target physical machine group 13: Virtual machine management program 14: User portal site 15: VM configuration file, VM operation history database
R_MAX: Physical machine resource amount (maximum usage)
R_USED: Physical machine resource usage
xav: Average value of waveform
xnt: Normalized resource usage waveform
W: Waveform size ρ: Evaluation value of leveling

Claims (11)

The resource usage of each of the plurality of virtual machines deployed in each of the plurality of physical machines in the managed physical machine group and the plurality of physical machines per unit time is stored.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time within the predetermined period of the first physical machine is similar. Select the first virtual machine with the second waveform of the normalized resource usage, which has a relatively high degree,
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. Select the second physical machine you have and
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
Until the first determination becomes true, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage are repeated.
The first determination is the Euclidean distance between the resource usage amount for each unit time within the predetermined period and the average value of the resource usage amount of each of the plurality of physical machines calculated based on the calculated resource usage amount. The ratio of the total value of the first waveform size to the second waveform size, which is the Euclidean distance between the total resource usage of the plurality of physical machines for each unit time within the predetermined period and the average value of the total resource usage. Is a determination as to whether or not the state is reduced to a predetermined state during the repetition.
Further, when the first determination becomes true, the first virtual machine is rearranged from the first physical machine to the second physical machine.
A virtual machine management program that lets a computer perform processing. The resource usage of each of the plurality of virtual machines deployed in each of the plurality of physical machines in the managed physical machine group and the plurality of physical machines per unit time is stored.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time within the predetermined period of the first physical machine is similar. Select the first virtual machine with the second waveform of the normalized resource usage, which has a relatively high degree,
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. Select the second physical machine you have and
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
Until the second determination becomes true, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage are repeated.
In the second determination, among the minimum values of the free resource capacity for each unit time of each of the plurality of physical machines calculated based on the calculated resource usage amount, the smallest minimum value among the plurality of physical machines is used. , It is a judgment as to whether or not the rise changes to a fall during the repetition.
Further, when the second determination becomes true, the first virtual machine is rearranged from the first physical machine to the second physical machine.
A virtual machine management program that lets a computer perform processing. The normalized resource usage is the difference between the resource usage per unit time within the predetermined period and the average value of the resource usage per unit time, and the resource usage per unit time within the predetermined period. The virtual machine management program according to claim 1 or 2 , which is calculated by dividing by the standard deviation of. The closer the Euclidean distance, which is the square root of the square root of the difference between the first waveform and the second waveform, or the second waveform and the third waveform for each unit time, the more similar the similarity is. The virtual machine management program according to claim 1 or 2 , which is determined not to be very similar. Stores the resource usage of multiple virtual machines deployed in each of the multiple physical machines in the managed physical machine group and the plurality of physical machines per unit time.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the plurality of physics are obtained from the normalized resource usage amount obtained by normalizing the resource usage amount per unit time within the predetermined period of the first physical machine. The first difference waveform obtained by subtracting the normalized total resource usage obtained by normalizing the total resource usage per unit time within the predetermined period of the machine, and the normalized resource usage having a relatively high similarity are the first. Select the first virtual machine with 2 waveforms,
Of the plurality of physical machines excluding the first physical machine, a second physical machine having a second difference waveform having a relatively high similarity to the second waveform of the normalized resource usage of the first virtual machine is used. The second differential waveform is selected and obtained by subtracting the normalized resource usage of the second physical machine from the normalized total resource usage.
Relocating the first virtual machine from the first physical machine to the second physical machine.
A virtual machine management program that lets a computer perform processing. The normalized resource usage is the difference between the resource usage per unit time within the predetermined period and the average value of the resource usage per unit time, and the resource usage per unit time within the predetermined period. Calculated by dividing by the standard deviation of
The normalized total resource usage is the difference between the total resource usage per unit time within the predetermined period and the average value of the total resource usage per unit time, and the difference is obtained for each unit time within the predetermined period. The virtual machine management program according to claim 5 , which is calculated by dividing by the standard deviation of the total resource usage. The closer the Euclidean distance, which is the square root of the square root of the square of the difference between the first difference waveform and the second waveform, or between the second waveform and the second difference waveform, the more similar the similarity is. The virtual machine management program according to claim 5 , which determines that the distance is not similar. The process further
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
Until the first determination becomes true, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage are repeated.
The first determination is the Euclidean distance between the resource usage amount for each unit time within the predetermined period and the average value of the resource usage amount of each of the plurality of physical machines calculated based on the calculated resource usage amount. The ratio of the total value of the first waveform size to the second waveform size, which is the Euclidean distance between the total resource usage of the plurality of physical machines for each unit time within the predetermined period and the average value of the total resource usage. Is a determination as to whether or not the state is reduced to a predetermined state during the repetition .
When the first determination becomes true, the first virtual machine is relocated from the first physical machine to the second physical machine .
The virtual machine management program according to claim 5. The process further
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
Until the second determination becomes true, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage are repeated.
In the second determination , among the minimum values of the free resource capacity for each unit time of each of the plurality of physical machines calculated based on the calculated resource usage amount, the smallest minimum value among the plurality of physical machines is used. , It is a judgment as to whether or not the rise changes to a fall during the repetition.
When the second determination becomes true, the first virtual machine is relocated from the first physical machine to the second physical machine .
The virtual machine management program according to claim 5. The resource usage of each of the plurality of virtual machines deployed in each of the plurality of physical machines in the managed physical machine group and the plurality of physical machines per unit time is stored.
Among the plurality of physical machines, the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period is selected.
Of the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time within the predetermined period of the first physical machine is similar. Select the first virtual machine with the second waveform of the normalized resource usage, which has a relatively high degree,
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. Select the second physical machine you have and
When the first virtual machine is relocated from the first physical machine to the second physical machine, the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time is calculated.
Until the first determination becomes true, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage are repeated.
The first determination is the Euclidean distance between the resource usage amount for each unit time within the predetermined period and the average value of the resource usage amount of each of the plurality of physical machines calculated based on the calculated resource usage amount. The ratio of the total value of the first waveform size to the second waveform size, which is the Euclidean distance between the total resource usage of the plurality of physical machines for each unit time within the predetermined period and the average value of the total resource usage. Is a determination as to whether or not the state is reduced to a predetermined state during the repetition.
Further, when the first determination becomes true, the first virtual machine is rearranged from the first physical machine to the second physical machine.
A virtual machine management method that has processing. A resource usage storage means for storing resource usage per unit time of a plurality of virtual machines deployed in each of a plurality of physical machines in a group of managed physical machines and the plurality of physical machines.
A first physical machine selection means for selecting the first physical machine having the smallest minimum resource free space for each unit time within a predetermined period among the plurality of physical machines.
Among the plurality of virtual machines operating on the first physical machine, the first waveform of the normalized resource usage obtained by normalizing the resource usage per unit time of the first physical machine within the predetermined period is similar. A first virtual machine selection means for selecting a first virtual machine having a second waveform of the normalized resource usage having a relatively high degree, and
Of the plurality of physical machines excluding the first physical machine, the third waveform of the normalized resource usage having a relatively low similarity to the second waveform of the normalized resource usage of the first virtual machine is used. A second physical machine selection means for selecting a second physical machine to have,
A resource usage calculation means for calculating the resource usage of the plurality of virtual machines and the plurality of physical machines per unit time when the first virtual machine is relocated from the first physical machine to the second physical machine. Have and
Until the first determination becomes true, the selection of the first physical machine, the selection of the first virtual machine, the selection of the second physical machine, and the calculation of the resource usage are repeated.
The first determination is the Euclidean distance between the resource usage amount for each unit time within the predetermined period and the average value of the resource usage amount of each of the plurality of physical machines calculated based on the calculated resource usage amount. The ratio of the total value of the first waveform size to the second waveform size, which is the Euclidean distance between the total resource usage of the plurality of physical machines for each unit time within the predetermined period and the average value of the total resource usage. Is a determination as to whether or not the state is reduced to a predetermined state during the repetition.
Further, a virtual machine management device having a relocation means for relocating the first virtual machine from the first physical machine to the second physical machine when the first determination becomes true.

Images