Xen NUMA Roadmap

From Xen
Revision as of 17:37, 19 February 2014 by Dariof (talk | contribs) (NUMA in Other Virtualization Platforms)

NUMA means that memory access times of a program running on a CPU depends on the relative distance between the specific CPU and memory area. In fact, in NUMA systems, there usually is more than one memory controller, each one local to a processor (or set of processors). Accessing memory on a local controller is very fast, while getting and storing data from and on a remote one is quite more complex and slow.

About This Page

Xen is already capable of running on NUMA machines (e.g., look at cpupool-numa-split), but both performances and user interface can be improved. This page acts as a collection point of all the features that are under development, wrt NUMA support in Xen. This is being done with the hope of facilitating as much as possible the collaboration between the various community members on this specific field, trying to avoid any effort duplication/wasting.

For details on the single features (e.g., who is in charge of them, what is the status, what release is being targeted, etc.), see the specific feature pages (TODO breakdown the below in single feature pages). Notice that the complete list of open, not currently owned, development tasks for the Xen Project is hosted in this page: Xen Development Projects. There are some items there that map to some of the features listed here (or part of them). If aiming at getting started with some Xen development, be sure you look there.

Other Pages About NUMA in Xen

For general information about NUMA on Xen, check this page Xen NUMA Introduction.

NUMA in Other Virtualization Platforms

Some information on how NUMA is handled in VMWare virtualization solutions can be found here:

And some on NUMA in Linux:

Updating this page

This is a Wiki, so, please, go ahead and update/fix. If, when doing that, you also could drop a line to Dario Faggioli, the maintainer of the page itself, or, even better, start a conversation in the xen-devel mailing list, that would be great (in the latter case, just make sure you've read this).

NUMA Features

Automatic VM placement


This is about picking up a NUMA node (or a set of NUMA nodes) where a newly created VM would best execute, in order to maximize its own and the system overall performances.


Basics are there. The old xm/XenD toolstack had a placement logic implemented (in XenD) which was not included in the new xl/libxl toolstack in the first place. That has been fixed recently, and now (starting from Xen 4.2) automatic placement is available for xl/libxl users. That being said, there is still a lot of room for improvements and making the placement algorithm more advanced and powerful.

Work items

NUMA aware scheduling


This is about not statically pinning VCPUs on nodes' PCPUs and just have them prefer running on the nodes where their memory resides. NUMA-awareness for the credit scheduler is one of the key feature planned for Xen 4.3.


Patches posted and reviewed. Repost required quite a while because of extensive benchmarking performed in between the two releases of the patch series, along with the discovery and the need to fix some scheduling anomalies in the Credit scheduler.

Work items

Virtual NUMA topology exposure to guests


If a guest ends up on more than one nodes, make sure it knows it's running on a NUMA platform (smaller than the actual host, but still NUMA). It is something very important for some specific kind of workloads, for instance, HPC ones. In fact, it the guest OS (and application) has any NUMA support, exporting a virtual topology to the guest is the only way to render that effective, and perhaps filling at least to some extent the gap introduced by the needs of distributing the guests on more than one node. Under the name of vNUMA, this is one of the key and most advertised feature of VMWare vSphere 5 ("vNUMA: what it is and why it matters").


First of all, it must be considered that this interacts with some of the above points:

  • automatic placement for resuming/migrating domains: if they have a virtual topology, better not to change it;
  • memory migration: it can change the actual topology (should we update it on-line or disable memory migration?)

Dynamic memory migration


Between different nodes of one host, either upon user request or automatically, as a form of load balancing (similar to what happens on the CPU with the NUMA-aware scheduler. Memory migration is one of the feature desirable for Xen 4.3.


Started, but not yet ready to leave some developer's private patch queue in their dev-boxes. The need to support both HVM and PV guests complicate things quite a bit. Xenbus, qemu, a lot of inherent characteristics of the Xen architecture get in the way of having it simply done within the hypervisor (as it happens for NUMA aware scheduling). The current idea being pursued is for it to happen at low toolstack level (perhaps with the hypervisor exporting statistics that will help toolstacks and users to undertake proper decisions), sort-of mimicking a suspend-resume cycle.

Work item

  • Dario Faggioli (<dario.faggioli@citrix.com>), WORKING: enable moving memory from one node to another (on the same host) upon user request and doing that in small chunks, so that (ideally) no downtime will be perceived by the VM.
  • Dario Faggioli (<dario.faggioli@citrix.com>), PLANNED: track how many and from whom non node-local memory is being accessed. Report from the hypervisor to the upper layers so user or toolstack can properly consume it.
  • Make sure everything, not only VMs' pages is allocated (see alloc_{dom,xen}_heap_page()) on the proper nodes and, if that is the case, properly moved to another one (e.g., the per-VCPU stack and data segments).

IONUMA support


If not only memory, but also I/O controller are attached to specific nodes, you'll end up with devices which are better used b VMs running on those nodes (or vice-versa, VMs that are better run on the proper node if/when they want to use a specific device). Yang Zhang did some previous investigation on this situation, which, BTW, goes under the name IONUMA, and the result is this XenSummit-2011 presentation: "I/O Scalability in Xen".


Looks like The Right Thing^TM can be made happen acting at both Dom0 and hypervisor level. Hypervisor level, that looks preferrable for a number of reasons, would mean instrumenting the XENMEM_exchange a little bit, but not before having verified where all the information we need to understand the IONUMA characteristics of the host (which device is where?) are and how to get there. It is also important to investigate on actual IONUMA enabled hardware how big of an issue is to neglect it.

Work item

  • Neo Jia (<cjia_AT_nvidia_DOT_com>), PLANNED: collecting IONUMA information. Where are the information about what device is attached to which controller on what node? When could they become available?
  • Neo Jia (<cjia_AT_nvidia_DOT_com>), PLANNED: export IONUMA information to the user. As we currently do with things like xl info -n, which tells the user what PCPUs are part of which node, allow for something similar wrt devices-nodes mappings.
  • Neo Jia (<cjia_AT_nvidia_DOT_com>), PLANNED: performance assessment. What happens, from an I/O throughput perspective, if we give memory and run a VM as far as possible from the node where the device is attached? How bad is that? What happens in the opposite (best) case?
  • Neo Jia (<cjia_AT_nvidia_DOT_com>), PLANNED: Dom0/Driver IONUMA. Devices should have their DMA buffers allocated on (or as close as possible to) the node to which their controllers are attached.
  • Neo Jia (<cjia_AT_nvidia_DOT_com>), PLANNED: guest IONUMA. When guest boots with some passed-through devices, we should try to allocate the memory from the node where the device resides and, especially for multi-node guests, let the guest itself know the IONUMA topology.

Discussion on whether or not this is something worthwhile to have, and (if yes) how to deal with it happened here.

Benchmarking and performances evaluation

Performance evaluation is extremely important here. It is both needed that we:

  • agree on a set of meaningful benchmarks, for answering questions like 'are we actually improving performances for the right workloads?', 'aren't we introducing any performance regressions?', etc.;
  • figure out how to automatically run them concurrently in varying number of VM and, possibly, on different hosts (i.e., NUMA hosts with different characteristics).

Useful Benchmarks

Benchmarks should be run inside a varying number of VMs running concurrently. Some useful number can be obtained from the following suites:

  • SpecJBB2005: really looks a good one, very stable and consistent resutls.
  • lmbench (some of the lat_*): seems good, but needs more investigation.
  • stream: should be considered, but seems to have issues running with more than one thread (needs more investigation).