Xen NUMA Roadmap

From Xen
Revision as of 19:00, 11 December 2012 by Dariof (talk | contribs) (Started deep page restructuring)
Jump to navigationJump to search

What's this page about

Of course, Xen already runs quite effectively on NUMA machines (for instance, look out cpupools), but there are spots where we can do better and introduce interface and performances improvements. The purpose of this page is, in fact, to act as collection point of all the new features and the needed improvement to make Xen better support complex virtualization workloads on NUMA servers and hosts.

It will also indicates what features are under development, and who in the Xen community is doing that, in contrast with all the ones that are just pending but have not been claimed by any developers yet. Finally, for the items that are under development, this page will (try to) track their progresses.

The ultimate purpose of this page is to provide detailed context and ease collaboration of people involved in NUMA related developments on Xen.

Related pages

The complete list of open development tasks and projects of any kind for Xen is hosted in this page: Xen Development Projects. The features/items from this page that have been formulated in such a way that they constitutes a well formed Development Project (and that, of course, are not being taken care of by anyone yet) are also part of that page. So, in summary, if wanting to start developing on Xen and looking for some random place from where to start, look there. If aiming at the same, but with particular interest in NUMA, or even if just interested in how well Xen supports NUMA machines and how things will likely evolve in the future, look here!

Another feature oriented service provided by the Xen community is our uservoice. Go there to see what are the feature that most Xen users (at least, the ones that use that service) would like to see implemented, and in which priority order. That is not strictly related to this page here, but if thinking there is some NUMA related quirk we're missing, feel free to add it to both.

Updating this page

Given the Wiki nature of this page, anything that looks wrong or incomplete can well be fixed directly by who notices that. If any kind of clarification is needed, feel free to contact the person in charge of the item in question and/or Dario Faggioli. However, since we are talking about development plans, it would be nice if any doubt and/or modification would result in a conversation on the xen-devel mailing list, where the real development of Xen takes place. For some guidance on how to interact with that list, have a look here.

Improved NUMA support TODO list

Just notice that each assigned Work item contains the name and e-mail of the person working on it. In that context, WORKING ON IT means work has already started and patches could have already been submitted or will shortly be. On the other hand, PLANNED, means the person is keen on doing the job, but any code have been written yet (if wanting to take over, dropping a note could be useful, as there could be interesting information that he can share). If there is no name at all, the item is something identified as useful, but still unclaimed. Finally, Barred items are to be considered achieved (with the name telling who did the job).

Automatic VM placement

Description

This is about picking up a NUMA node (or a set of NUMA nodes) where a newly created VM would best execute, in order to maximize its own and the system overall performances.

Status

Basics are there. The old xm/XenD toolstack had a placement logic implemented (in XenD) which was not included in the new xl/libxl toolstack in the first place. That has been fixed recently, and now (starting from Xen 4.2) automatic placement is available for xl/libxl users. That being said, there is still a lot of room for improvements and making the placement algorithm more advanced and powerful.

Work items


IONUMA

Properly support IONUMA (Neo Jia: working on it?):

  • Dom0 IONUMA, i.e., devices should get their DMA buffers from the node to which their controllers are attached (currently Dom0 allocates these buffers without taking NUMA into account at all)
  • Guest IONUMA, i.e., when guest boots with some passed-through devices, we need to allocate the memory from the node where the device resides, as well as let guest know the IONUMA topology (topic mentioned during Xen Summit 2011 in this presentation I/O Scalability in Xen)

Placement in Xen internals

  • Automatic placement of Dom0, if possible. See here (current code only affects DomUs);
  • placing internal Xen items, such as the per-cpu stacks and data area, on the local NUMA node, rather than unconditionally on node 0 as it is at the moment. That might mean changing to alloc_{dom,xen}heap_page() or perhaps only use them (properly) in more places (Andrew/XS team will work on it).

NUMA aware scheduling

Don't pin vcpus on nodes' pcpus, just have them _prefer_ running on the nodes where their memory is. (Dario: patches posted. Review gooing on [here http://lists.xen.org/archives/html/xen-devel/2012-10/msg00569.html]).

Dynamic memory migration

Between different nodes of the host, as the counter-part of the NUMA-aware scheduler. See here (Dario: working on it).

Virtual NUMA topology exposure to guests

A.k.a guest-numa. If a guest ends up on more than one nodes, make sure it knows it's running on a NUMA platform (smaller than the actual host, but still NUMA). This interacts with some of the above points:

  • consider this during automatic placement for resuming/migrating domains (if they have a virtual topology, better not to change it);
  • consider this during memory migration (it can change the actual topology, should we update it on-line or disable memory migration?)

Discussion on whether or not this is something worthwhile to have, and (if yes) how to deal with it going on here.

NUMA and memory shrinking and sharing

In some more details:

  • page sharing on NUMA boxes: it's probably sane to make it possible disabling sharing pages across nodes;
  • ballooning and its interaction with placement (races, amount of memory needed and reported being different at different time, etc.).
  • trascendent memory (Tmem) as a mechanism for discriminating between frequently and infrequently used data, and thus helping allocating them properly. In fact (Dan Magenheimer, can't work on it but can provide mentoring to anyone interested):
    • Tmem very nicely separates infrequently-used data from frequently-used data (API/ABI that now fully in upstream Linux);
    • add to Xen something like "alloc_page_on_any_node_but_the_current_one()" (or "any_node_except_this_guests_node_set" for multinode guests) and have Xen's tmem implementation use it (especially in combination with selfballooning). This could solve a significant part of the NUMA problem when running tmem-enabled guests.

Inter-VM dependencies and communication issues

If a workload is made up of more than just a VM and they all share the same (NUMA) host, it might be best to have them sharing the nodes as much as possible, or perhaps do right the opposite, depending on the specific characteristics of he workload itself, and this might be considered during placement, memory migration and perhaps scheduling.

Huge amount of work is being done here, at the Computer Lab of the University of Cambridge, by Anil Madhavapeddy and Malte Schwarzkopf. This is the page through which everyone can contribute to that project, submitting their own results.

Benchmarking and performances evaluation

In general, meaning both agreeing on a (set of) relevant workload(s) and on how to extract meaningful performances data from there (and maybe how to do that automatically?).

Retrieved from "https://wiki.xenproject.org/index.php?title=Xen_NUMA_Roadmap&oldid=5785"