Archive/XCP FAQ Dynamic Memory Control

Introduction

This document answers frequently-asked questions (FAQ) about Dynamic Memory Control (DMC), a feature included in Xen Cloud Platform (XCP).

We've written this document so that it makes most sense when read from start to finish, but feel free to dive in at any question.

If you'd like to see further questions answered in this document, contact us at xen-api@lists.xensource.com.

Authors

• Jonathan Knowles (2010)

FAQ

Now here are the questions and answers:

What is Dynamic Memory Control?

Dynamic Memory Control (DMC) is a technology provided by Xen Cloud Platform (XCP), starting from the 0.5 release.

DMC makes it possible to:

• change the amount of host physical memory assigned to any running virtual machine without rebooting it.
  (within limits specified by an administrator)
• start additional virtual machines on a host whose physical memory is currently full, by automatically reducing the memory allocations of existing virtual machines in order to make space.
  (within limits specified by an administrator)

In which environments can DMC help me?

DMC offers benefits in both server and desktop environments.

How can DMC help my virtual server environment?

DMC allows you to change the amount of host memory assigned to any running virtual server, without rebooting it.

In virtual server environments without DMC, it's necessary to decide in advance how much memory to assign each virtual server. Once a virtual server is running, its memory allocation is fixed, and it's not possible to change without rebooting. Rebooting production servers can be painful, so administrators are left with a choice between:

• Allocating a large, fixed amount of host memory to each virtual server:
  This decreases the risk that the virtual server will run out of memory, but leaves less host memory available for other virtual servers.
  or
• Allocating a small, fixed amount of host memory to each virtual server:
  This leaves more host memory available for other virtual servers, but increases the risk that the virtual server will run out of memory.

DMC frees you from having to make a fixed choice about how much memory to assign to each virtual server. Within certain limits, you can reduce or increase a virtual server's host memory allocation without rebooting the server.

This means you could start a virtual server with a small amount of host memory, but plan to add more later as necessary. Alternatively, you could start your virtual server with a large amount of host memory, but reduce this amount later on if it turns out that it needs less than you expected.

For example, consider a cloud computing provider that supplies virtual servers. The provider has different price plans, charging more for virtual servers with more memory. Customers can increase the amount of memory allocated to their virtual servers in return for paying an increased premium. When a customer decides to change their plan, the cloud provider can increase their virtual server's memory allocation without rebooting it. Conversely, when the customer no longer needs the extra memory, the provider can decrease the allocation, all without rebooting the customer's server.

How can DMC help my virtual desktop environment?

DMC allows you to change the amount of host memory assigned to any running virtual desktop, without rebooting it.

DMC also makes it possible to start additional virtual desktops on a host that's already full, automatically reducing the memory allocations of existing virtual desktops in order to make space.

In virtual desktop environments without DMC, it's necessary to decide in advance how much memory to assign each virtual desktop. Once a virtual desktop is running, its memory allocation is fixed, and its not possible to change without rebooting. Once you've exhausted the memory of a host, it's also not possible to start additional virtual desktops on that host.

When deciding how much memory to assign each virtual desktop, administrators are often left with a choice between:

• Allocating a large, fixed amount of host memory to each virtual desktop:
  This increases virtual desktop performance, but decreases the number of users that a single host can support.
  Allocating more memory makes sense when there are fewer users, but if the number of users increases, it becomes necessary to bring more hosts online to meet the demand.
  or
• Allocating a small, fixed amount of host memory to each virtual desktop:
  This increases the number of users that a single host can support, but decreases virtual desktop performance.
  Allocating less memory makes sense when there are more users, but if the number of users decreases, significant amounts of host memory may remain unused.

DMC frees you from having to make a fixed choice between host-memory-per-virtual-desktop and number-of-virtual-desktops-per-host. Once configured, DMC can automatically manage the trade-off between these two quantities, in response to changing numbers of users.

DMC can automatically decrease the host memory allocations of existing virtual desktops in order to start additional virtual desktops on hosts that are already full. After users shut down virtual desktops that are no longer required, DMC can automatically reuse the deallocated memory to increase the host memory allocations of the virtual desktops that remain on the host.

How does DMC work?

Using DMC, it's possible to operate a guest virtual machine in one of two modes:

• Target Mode
  The administrator specifies a memory target for the guest.
  XCP adjusts the guest's memory allocation to meet the target.
  or
• Dynamic Range Mode
  The administrator specifies a dynamic memory range for the guest.
  XCP chooses a target from within the range and adjusts the guest's memory allocation to meet the target.

When should I use target mode?

Specifying a target is particularly useful in virtual server environments, and in any situation where you know exactly how much memory you want a guest to use.

XCP will adjust the guest's memory allocation to meet the target you specify.

How do I use the command line to set a target?

From the XCP command line, issue the following command:

 xe vm-memory-target-set uuid=<uuid-of-vm> target=<new-target>

Anchor(dynamic-range-mode)

When should I use dynamic range mode?

Specifying a dynamic range is particularly useful in virtual desktop environments, and in any situation where you want XCP to repartition host memory dynamically in response to changing numbers of guests, or changing host memory pressure.

XCP chooses a target from within the range and adjusts the guest's memory allocation to meet the target.

Anchor(dynamic-range)

What is a dynamic memory range?

A dynamic memory range consists of a minimum and a maximum amount of memory.

Anchor(dynamic-minimum)

What is a dynamic memory minimum?

A dynamic memory minimum represents the lower limit of a dynamic memory range. It's the minimum amount of memory that the administrator is happy for a guest to use.

Anchor(dynamic-maximum)

What is a dynamic memory maximum?

A dynamic memory maximum represents the upper limit of a dynamic memory range. It's the maximum amount of memory that the administrator is happy for a guest to use.

How do I use the command line to set a dynamic range?

From the XCP command line, issue the following command:

 xe vm-memory-dynamic-range-set uuid=<uuid-of-vm> min=<new-min> max=<new-max>

Is it possible to change between target mode and dynamic range mode?

Yes.

It's possible to change between target mode and dynamic range mode at any time for any running guest. Simply specify a new target, or a new dynamic range, and XCP takes care of the rest.

Anchor(mode-equivalence)

Is there an equivalence between target mode and dynamic range mode?

Yes.

Setting a target of k is equivalent to setting a dynamic range where dynamic minimum = dynamic maximum = k.

How does XCP choose targets for guests in dynamic range mode?

XCP attempts to maximise the memory targets of guests in dynamic range mode:

• For an individual guest, XCP always chooses a target from within the guest's dynamic range. XCP never chooses a target greater than the guest's dynamic maximum, or lower than the guest's dynamic minimum.
• If there is enough host memory available, XCP chooses a target for each guest equal to its dynamic maximum.
• If there isn't enough host memory available to give every guest its dynamic maximum, XCP chooses the highest target possible for each guest, ensuring that every guest on the same host has a target that lies the same fractional distance between its dynamic minimum and maximum.
• When considering all guests running on a host, XCP never allows the sum of guests' dynamic memory minimum values to exceed the total host memory, after virtualization overheads have been taken into account.

For example, consider a host with 120 GiB of available memory, and guests with a dynamic range of [4 GiB ... 6 GiB]:

|| Guests per Host || Memory per Guest || Justification ||
|| 10 || more than enough host memory to assign each guest a target of 6 GiB (dynamic maximum) || 10 x 6 GiB = 60 GiB < 120 GiB ||
|| 20 || just enough host memory to assign each guest a target of 6 GiB (dynamic maximum) || 20 x 6 GiB = 120 GiB ||
|| 24 || only enough host memory to assign each guest a target of 5 GiB || 24 x 5 GiB = 120 GiB ||
|| 30 || only enough host memory to assign each guest a target of 4 GiB (dynamic minimum) || 30 x 4 GiB = 120 GiB ||

XCP will prevent any attempts to start more than 30 guests in the above scenario.

Anchor(current-target)

How can I determine the currently-chosen target for a guest in dynamic range mode?

From the XCP command line, issue the following command:

 xe vm-param-get uuid=<uuid-of-vm> param-name=memory-target

Additionally, XCP maintains a round-robin database (RRD) of memory targets for each guest. Inspecting the memory-target RRD will allow you to see how the memory target has changed over time.

How does XCP start additional guests on a host that's already full?

When a host's memory becomes full, XCP can start additional guests by automatically deallocating memory from existing guests, in order to reclaim space for the new guests.

When asked to start additional guests, XCP reduces the memory targets of running guests in proportion to their dynamic ranges, assigning all running guests new targets at the same fractional distance between their dynamic minimum and maximum values. Then:

• If all guests are able to respond to their reduced targets, then XCP has reclaimed enough memory and can start the additional guests.
• If some guests are unable to respond to their reduced targets due to memory pressure, then XCP attempts to make up the deficit by further reducing the memory targets of those guests not experiencing memory pressure.
• XCP continues this process until either it has reclaimed enough memory to start the new guests, or until it cannot reduce memory of existing guests any further (in which case it will refuse to start the additional guests).
• XCP will never reduce the memory targets of running guests below their dynamic minimum values.

XCP aims to allocate each additional guest enough host memory to boot at its dynamic maximum. Nevertheless, if the host has less than this amount available, then XCP will boot the guest with a lower memory target, provided that the target is no lower than the guest's dynamic minimum.

Once the additional guests have finished booting, XCP rebalances memory targets across all guests on the same host. XCP achieves this by choosing the highest target possible for each guest, ensuring that every guest on the same host has a target that lies the same fractional distance between its dynamic minimum and maximum.

How many additional guests can I start on a host that's already full?

The number of additional guests you can start depends on:

• the total amount of physical memory in the host; and
• the dynamic memory ranges of guests on the host.

For example, consider a host with 120 GiB of available physical memory, and guests with a dynamic range of [4 GiB ... 6 GiB]:

• If we start 10 guests:
  XCP has more than enough host memory to operate each guest at its dynamic maximum of 6 GiB (since 10 x 6 GiB = 60 GiB < 120 GiB).
• If we start an additional 10 guests (bringing the total to 20 guests):
  XCP still has enough host memory to operate each guest at its dynamic maximum (since 20 x 6 GiB = 120 GiB).
  However, once all 20 guests are running, the host's memory is full.
• If we attempt to start an additional 4 guests (bringing the total to 24 guests):
  XCP will no longer have enough memory to operate all guests at their dynamic maximum.
  It will attempt to reduce the host memory allocations of the existing 20 guests in order to make space.
  Once all 24 guests are running, XCP assigns all guests a target of 5 GiB - half-way between dynamic minimum and maximum (since 24 x 5 GiB = 120 GiB).
• If we attempt to start an additional 6 guests (bringing the total to 30 guests):
  XCP will attempt to reduce even further the host memory allocations of the existing 24 guests in order to reclaim enough space.
  Once all 30 guests are running, XCP assigns all guests a target of 4 GiB - the dynamic minimum and smallest allowed by the administrator (since 30 x 4 GiB = 120 GiB).
• XCP will prevent any attempts to start additional guests in this scenario, since that would mean reducing the targets of running guests below their dynamic minimum values.

In the above example, after filling up the host's memory by starting 20 guests, XCP is able to start an additional 10 guests.

When can I change a guest's memory target?

XCP allows you to change a guest's memory target when a guest is running or halted.

When can I change a guest's dynamic memory range?

XCP allows you to change a guest's dynamic memory range when a guest is running or halted.

Does XCP impose any limits on memory properties?

Yes.

XCP requires that every VM has dynamic and static memory properties that satisfy the following inequality:

static mininum ≤ dynamic minimum ≤ dynamic maximum ≤ static maximum

Does XCP impose any limits on memory targets?

Yes.

A guest's memory target may not exceed the static memory maximum for that guest, which is fixed at boot.

Does XCP impose any limits on dynamic memory ranges?

Yes.

A guest's dynamic memory maximum may not exceed the static memory maximum for that guest, which is fixed at boot.

When considering all guests running on a host, XCP never allows the sum of guests' dynamic memory minimum values to exceed the total host memory, after virtualization overheads have been taken into account.

Anchor(static-maximum)

What is a static memory maximum?

All guests have a static memory maximum.

The static memory maximum defines the maximum amount of physical memory that the guest operating system can address, from the time the guest boots up until the time the guest shuts down again.

How do I use the command line to set a static memory maximum?

Use the following command:

 xe vm-param-set uuid=<uuid-of-vm> memory-static-max=<new-value>

When is it possible to change the static memory maximum?

It's only possible to change a guest's static memory maximum when the guest is halted. Once the guest is running, the static memory maximum becomes fixed and cannot be changed again until the guest shuts down.

How large should I make static memory maximum for a guest?

The most appropriate value will depend on your circumstances.

Since it's not possible to change the static memory maximum of a running guest, it's important to choose a value that you're comfortable with before starting the guest. Once the guest has started, you'll need to reboot it if you ever want to increase its memory allocation beyond the static maximum of that guest.

So why not just set the static memory maximum to infinity?

There are two reasons.

1. Firstly, guest operating systems use various internal data structures for memory management, such as page tables.

These data structures occupy memory within the guest's address space. Many operating systems will size their memory management data structures in proportion to the total amount of memory believed to be present in the system. Increasing the static memory maximum can lead to an increase in the size of such data structures, reducing the amount of usable memory within the guest.

1. Secondly, all XCP guests incur host memory overhead due to virtualization.

A guest's memory overhead is directly proportional to its static memory maximum. Increasing the static memory maximum gives you greater flexibility to increase memory later on, but it also increases the virtualization overhead of that guest.

How can I determine the memory overhead of a guest?

All XCP guests incur host memory overhead due to virtualization.

The following CLI command allows you to check a guest's memory overhead:

 xe vm-param-get uuid=<uuid-of-vm> param-name=memory-overhead

How large is the memory overhead of a guest?

In general, a guest's memory overhead is directly proportional to its static memory maximum.

However, using the default settings, a guest incurs just under 1% of its static memory maximum in virtualization overhead.

Here are some examples:

|| Memory 
target || Static memory 
maximum || Memory overhead 
(% of target) || Memory footprint 
 (% of target) ||
|| 1024 MiB || 1024 MiB || 11 MiB (~ 1%) || 1035 MiB (~ 101%) ||
|| 1024 MiB || 2048 MiB || 19 MiB (~ 2%) || 1043 MiB (~ 102%) ||
|| 1024 MiB || 4096 MiB || 36 MiB (~ 4%) || 1060 MiB (~ 104%) ||
|| 2048 MiB || 4096 MiB || 36 MiB (~ 2%) || 2084 MiB (~ 102%) ||
|| 4096 MiB || 4096 MiB || 36 MiB (~ 1%) || 4132 MiB (~ 101%) ||

How do I calculate the memory footprint of a guest?

The memory footprint of a guest is equal to the sum of its overhead and the currently acquired target.

What technologies does DMC rely on?

DMC relies on two key technologies:

1. Memory balloon drivers running within each guest.
2. Memory populate-on-demand logic within the hypervisor.

What are balloon drivers for?

Balloon drivers make it possible to temporarily remove memory from running guests so that the memory can be used by other guests.

How do balloon drivers work and what is a memory balloon?

In order to add memory to or remove memory from a running guest, DMC relies completely on the action of a balloon driver running within the guest operating system. The balloon driver works by inflating or deflating a memory balloon - a special area of memory within the guest's physical address space.

During normal operation, every physical memory page within the guest is backed up by a physical memory page on the host.

When XCP wants to reduce a guest's memory allocation, it asks the balloon driver running in the guest to "inflate" its memory balloon. The balloon driver achieves this by using an OS-specific technique to allocate physical memory pages from the guest. Initially, these guest physical memory pages will be backed by host physical memory pages, just like any other pages in the guest. However, after acquiring guest physical memory pages, the balloon driver immediately informs the hypervisor that it may recycle the host physical memory pages that back them. The hypervisor immediately revokes the guest's access to these host physical memory pages, and makes them available for use by other guests.

Inflating a balloon measurably increases physical memory pressure within the guest. From the guest's point of view, it can no longer use the memory that's been taken away. The balloon driver appears to be rather like a long-lived process that's using some of the available guest physical memory.

When XCP wants to increase a guest's memory allocation, it asks the balloon driver running in the guest to "deflate" its memory balloon. However, in order to do this, the balloon driver must /first/ ask the hypervisor to re-back ballooned-out guest physical pages with host physical pages. The hypervisor may refuse, and will certainly do so if there are no remaining physical pages available on the host. (However, in normal circumstances, XCP will never ask the guest to increase its memory allocation if it thinks the hypervisor cannot meet the demand.)

What is memory populate-on-demand for?

Memory populate-on-demand is a hypervisor technology that makes it possible to reduce the host memory footprint of hardware-virtualized guests during boot.

Why is memory populate-on-demand necessary?

XCP relies on the action of a co-operating balloon driver in order to reduce the memory allocation of a running guest.

• With para-virtualized (PV) operating systems, we can make kernel modifications that allow guests start in a "pre-ballooned state", where the balloon is already inflated. Such guests use no more host memory during boot than they use during normal operation.
• With hardware-virtualized operating systems, we cannot in general make such kernel modifications. XCP must instead rely on a balloon driver that loads during the normal boot process. Normally, such guests use host memory equal to their static memory maximum during boot.

Populate-on-demand makes it possible to start hardware-virtualized guests in such a way that they require no more host memory during boot than they require during normal operation.

Does memory populate-on-demand apply to paravirtualized guests?

No.

Paravirtualized guests are able to boot in a "pre-ballooned" state, using no more memory during boot than they would use in normal operation.

When does memory populate-on-demand operate?

Memory populate-on-demand operates from the time a guest boots, until the balloon driver has fully-loaded.

Does DMC make it possible to overcommit physical host memory?

DMC does not provide a way for the total amount of guest-accessible physical memory to exceed the total amount of physical host memory. "Guest-accessible physical memory" means "physical memory that the guest is free to read from and write to".

DMC does makes it possible to dynamically repartition host physical memory among multiple guests. However on a host that's already full:

• it's only possible to increase the amount of physical memory accessible to one guest by reducing the amount of physical memory that's accessible to another guest.
• it's only possible to start additional guests by reducing the amount of physical memory that's accessible to existing guests.

During normal operation, XCP stores every guest physical memory page within a physical memory page on the host. When XCP wants to reduce a guest's memory allocation, it actually reduces its commitment to that guest by reducing the number of guest physical pages that are backed with physical memory pages on the host. It prevents the guest from attempting to access the unbacked pages by holding them within a memory balloon.

The concept of overcommitting physical memory is well-established within operating systems design.

Overcommitting, or committing beyond the capacity for realization (1), generally comes with some sort of risk that the system won't be able to meet its commitments, and may need to take remedial action in order to avoid breaking the commitment.

For example, several operating system kernels (including the Linux kernel) routinely overcommit physical host memory when allocating virtual memory to user-space processes. Initially after being handed out, virtual pages might not be backed up by physical pages, which enables the kernel to hand out many more virtual pages than there are physical pages on the host. The kernel lazily backs these virtual pages with physical pages on demand when processes access them for the first time.

In doing this, the kernel potentially obligates itself beyond the capacity for realization. The kernel takes a gamble that for most of the time, processes will not need to use all of the physical memory in the machine. However, if the memory usage ever does exceed the amount of physical memory, then the OS can choose to move some of the pages out of physical RAM and into swap. Of course this comes at a performance cost. Furthermore, if the swap space ever becomes full, then the kernel must deny further memory allocation requests, or kill off existing processes.

XCP never commits physical memory beyond its capacity to realize that commitment. At all times, XCP keeps its physical memory commitment at a level that is strictly less than the total amount of physical memory in the host.

Without overcommitting host memory, how does XCP start additional guests on a host that's already full?

When XCP starts additional guests on a host that's already full, it can only do so by reducing the amount of host physical memory that's allocated to guests already running. Consequently, the amount of host physical memory available to these guests decreases.

XCP does commit to giving each guest an amount of physical memory that's greater than or equal to its dynamic minimum. However, when considering all guests running on a host, XCP never allows the sum of guests' dynamic memory minimum values to exceed the total amount of physical memory in the host, after virtualization overheads have been taken into account. So XCP's total commitment can never exceed the amount of physical memory in the host.

See this earlier answer for more details.

Does DMC adjust memory in response to changes in guest memory pressure?

No.

It is possible to build a system that dynamically adds memory to virtual machines that have a greater need, and removes memory from virtual machines that don't currently need it.

Unfortunately, building a reliable system is not trivial.

One problem to solve is that modern operating systems attempt to maximise their performance by using a large proportion (if not all) of their spare memory for buffers. It's non-trivial to know through inspection of the VM just how much of its memory can be reclaimed at any moment in time.

Additionally, it may be fairly easy to tell when a guest is in trouble, (by inspecting the page fault rate) but it's rather more difficult to tell how much additional memory is required to lift a guest out of trouble and back into its comfort zone.

Another problem is that it's important to react quickly enough when a ballooned down guest has a sudden requirement for more memory. What if the memory is not available when the guest needs it?

Finally, it's important to come up with measures of memory pressure that work well across different operating system families running on the same host, without inadventantly producing a system with bias towards a particular OS family.

Does DMC enable XCP to perform transparent physical page sharing?

No.

Prototype and experimental versions of the Xen hypervisor provide the ability to perform transparent physical page sharing. Future versions of XCP may incorporate this technology as it matures.

Does DMC make use of hypervisor-level swap?

No.

Prototype and experimental versions of the Xen hypervisor provide the ability to perform hypervisor-level swap. Future versions of XCP may incorporate this technology as it matures.

Will XCP provide the ability to overcommit physical host memory in future releases?

Prototype versions of the Xen hypervisor provide the ability to perform hypervisor-level swap and transparent page sharing. Future versions of XCP may incorporate these technologies as they mature, making it possible to overcommit physical host memory.

References

1. http://www.thefreedictionary.com/overcommit