Difference between revisions of "Credit2 Scheduler"

From Xen
(General reorganization)
(General reorganization)
Line 2: Line 2:
   
 
== Introduction ==
 
== Introduction ==
  +
  +
=== Design Goals ===
   
 
Credit2 is a general purpose scheduler for Xen, designed with the following goal in mind:
 
Credit2 is a general purpose scheduler for Xen, designed with the following goal in mind:
Line 12: Line 14:
 
* client virtualization
 
* client virtualization
   
== Contributing ==
+
=== Using Credit2 ===
  +
  +
For using Credit2 as the main scheduler, Xen boot command line should be edited, adding <tt>sched=credit2</tt>.
  +
  +
Another way of using and/or testing Credit2 is by means of [[Cpupools_Howto|cpupools]].
  +
  +
=== Contributing ===
   
 
Thoughts, suggestions, questions and patches via email to [mailto:xen-devel@lists.xen.org xen-devel]. See the [http://lists.xen.org/xen-devel list info page] for subscription information and the [http://lists.xen.org/archives/html/xen-devel/ archives].
 
Thoughts, suggestions, questions and patches via email to [mailto:xen-devel@lists.xen.org xen-devel]. See the [http://lists.xen.org/xen-devel list info page] for subscription information and the [http://lists.xen.org/archives/html/xen-devel/ archives].
   
 
If asking questions, please refer to [[Asking Developer Questions]]. If sending patches, please see [[Submitting Xen Patches]].
 
If asking questions, please refer to [[Asking Developer Questions]]. If sending patches, please see [[Submitting Xen Patches]].
  +
  +
For non-developers, even ''just'' testing your favorite workload, and observing and reporting any any regressions, improvements, and so on, would be much appreciated. Remember that a scheduler's job is only hard when there's more work to do than there is CPU to do it, so make sure you're focusing on multiple competing workloads.
  +
  +
=== Readiness Criteria ===
  +
  +
This is an open issue: when shall Credit2 (but also a scheduler in general) be considered ready for production use, when ready for becoming Xen default scheduler? What we need is a '''standardized set of regression tests'''.
  +
  +
Of course, that will greatly help in future development too. In fact, if anyone proposes a change, we can test the change for any regressions in other workloads.
   
 
== Status ==
 
== Status ==
   
Credit2 is still marked as '''experimental''' software. Development is ongoing in order to make it suitable for production usage. Broadly speaking this is the current status of Credit2:
+
Credit2 is still marked as '''experimental''' software. Development is ongoing in order to make it suitable for production usage. Broadly speaking this is the current situation:
  +
 
* stable and reliable; it is part of Xen upstream and of the OSSTest automated test suite since ages
 
* stable and reliable; it is part of Xen upstream and of the OSSTest automated test suite since ages
 
* benchmarking showed improved fairness, with respect to Credit1, especially in latency sensitive workloads
 
* benchmarking showed improved fairness, with respect to Credit1, especially in latency sensitive workloads
Line 29: Line 46:
   
 
* CPU Topology:
 
* CPU Topology:
** [[Hyperthreading]]: avoid sharing threads if possible, and to "discount" sharing of threads.
+
** [[Hyperthreading]]: avoid sharing threads if possible, and to "discount" sharing of threads. '''Status:''' Uma [mailto:uma.sharma523@gmail.com <uma_DOT_sharma523_AT_gmail_DOT_com>] is working on this, during her [http://umasharma17.wordpress.com/category/opw/ participation] to the [[OutreachProgram/Round9|OPW program]]. '''Time horizon:''' end of OPW9, in March 2015
  +
** '''Hard and Soft scheduling affinity:''' support for vcpu pinning and soft scheduling affinity (aka [[Xen_4.3_NUMA_Aware_Scheduling|NUMA aware scheduling]] is still missing. '''Status:''' Justin [mailto:jtweaver@hawaii.edu jtweaver_AT_hawaii_DOT_edu] is working on it. Patches sent: RFC, v1. '''Time horizon:''' April 2015
  +
* Credit Algorithm:
  +
** '''Cap and Reservation:''' both features are available in Credit1, and they should be implemented in Credit2 too. '''Status:''' no one working on this. Dario [mailto:dario.faggioli@citrix.com <dario_DOT_faggioli_AT_citrix_DOT_com>] will look into this at some point... In case someone is interested, speak up! '''Time horizon:''' undefined.
  +
** '''Credit math optimization:''' as of now, for the sake of correctness, every credit update involves a division, which may be slow. Some profiling and (most likely) optimization are required. It may be best to wait on this until the cap and reservation system is working, because the algorithm may change somewhat. '''Status:''' no one working on this. '''Time horizon:''' undefined
  +
  +
=== Open Issues ===
  +
  +
Here they are some open questions. Open problems that, although not critical, looks worth investigating:
  +
  +
* How to deal well with ''mixed workload'', e.g., situations where someone is listening to iTunes and there's a flash ad in background burning 100% of the CPU? Can we allow the audio to play well while still giving the VM only its fair share?
  +
* There exists a lot of CPU and/or IO throughput benchmarks, automatically outputting tons of numbers for one to look at. But as far as, e.g., audio and video are concerned, what are ways of ''automatically and programmatically'' measuring quality?
   
 
=== Milestones ===
 
=== Milestones ===
  +
 
* 2010 24 December: Simple load balancer checked into xen-unstable; credit2 can now handle multiple sockets.
 
* 2010 24 December: Simple load balancer checked into xen-unstable; credit2 can now handle multiple sockets.
 
* 2010 December: Credit2 integrated into the [https://blog.xenproject.org/2013/02/02/xen-automatic-test-system-osstest/ OSSTest automated test suite]
 
* 2010 December: Credit2 integrated into the [https://blog.xenproject.org/2013/02/02/xen-automatic-test-system-osstest/ OSSTest automated test suite]
 
* 2010 14 April: Credit2 checked into xen-unstable
 
* 2010 14 April: Credit2 checked into xen-unstable
 
* Prototype credit2 scheduler designed for latency-sensitive workloads and very large systems included in [[Xen_4.1_Release_Notes|Xen 4.1]]
 
* Prototype credit2 scheduler designed for latency-sensitive workloads and very large systems included in [[Xen_4.1_Release_Notes|Xen 4.1]]
 
 
== To-do ==
 
 
** '''Load balancing algorithm''': There's a fairly simple load-balancing algorithm in place, but it could use some work.
 
 
* Development: Credit algorithm
 
** '''Mixed work problem''': The current patch will work well if a VM is only doing audio. But what if someone is listening to iTunes and there's a flash ad in the background burning 100% of the CPU? Can we allow the audio to play well while still giving the VM only its fair share?
 
** '''Cap and Reservation''': With the current algorithm, how do we implement the "cap" and "reservation" features?
 
* Development: Honing
 
** '''Credit math optimization''': The current patch was made to be correct, not necessarily the most efficient. Every credit update involves a division, which may be slow. Some profiling and optimization would be useful here. Probably best to wait on this until the cap and reservation system is working, because the algorithm may change somewhat.
 
* Testing
 
** '''Automated audio / video measurement''': My current method of measuring audio quality is to play some music and count the number of "skips" I hear over five minutes. It would be great if this could be done programmatically somehow.
 
** '''More workloads, more configurations''': Test your favorite workload, and see if there are any regressions, improvements, and so on. Remember that a scheduler's job is only hard when there's more work to do than there is CPU to do it, so make sure you're focusing on multiple competing workloads.
 
** '''Test framework design''': I would like to have a standardized set of regression tests that can be run as we develop the scheduler, to see (a) when we've gotten where we want to be, and (b) so in the future, if anyone proposes a change, we can test the change for any regressions in other workloads.
 
   
 
== References ==
 
== References ==
* [http://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=2216&context=theses IN PERFECT XEN, A PERFORMANCE STUDY OF THE EMERGING XEN SCHEDULER (Dec 2013)]
+
* "In Perfect Xen, a Performance Study of the Emerging Xen Scheduler" (Dec 2013) [http://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=2216&context=theses text]
* [[XenSummit]] Asia 2009: a brief summary of the design target, goals, interface, credit design, and load balancing design. ([http://www.xen.org/files/xensummit_intel09/George_Dunlap.pdf text] [http://www.xen.org/files/xensummit_intel09/Xenschedulerstatus.pdf slides])
+
* [[XenSummit]] Asia 2009: a brief summary of the design target, goals, interface, credit design, and load balancing design. ([http://www.xen.org/files/xensummit_intel09/George_Dunlap.pdf text], [http://www.xen.org/files/xensummit_intel09/Xenschedulerstatus.pdf slides])
<!-- ##== Design inputs == -->
 
<!-- ## * [[[Credit2Dev|design target]]] -->
 
<!-- ## * [[[Credit2Dev|design goals]]] -->
 
<!-- ## * [[[Credit2Dev|target interface]]] -->
 
<!-- ##== Current design == -->
 
<!-- ## * [[[Credit2Dev|credit design]]] -->
 
<!-- ## * [[[Credit2Dev|load balance design]]] -->
 
<!-- ##== Misc == -->
 
   
 
[[Category:Xen]]
 
[[Category:Xen]]

Revision as of 07:57, 14 January 2015

This page's purpose is to ease collaboration and coordination when doing Credit2 development. It is also meant as a catch-all for everything relating to Credit2 itself.

Introduction

Design Goals

Credit2 is a general purpose scheduler for Xen, designed with the following goal in mind:

  • fairness: fairness is the ability of the scheduler to give each VM its fair share of physical CPU capacity. Such fair share should depend on the VM's scheduling parameters, such as weight, cap, and reservation.
  • responsiveness: responsiveness should be intended here as the capability of the scheduler to work well with latency-sensitive workloads.

These design goals have been derived from the more general objective to create a scheduler which can work well in several use cases, with particular attention to:

  • server consolidation
  • virtual desktop server solutions
  • client virtualization

Using Credit2

For using Credit2 as the main scheduler, Xen boot command line should be edited, adding sched=credit2.

Another way of using and/or testing Credit2 is by means of cpupools.

Contributing

Thoughts, suggestions, questions and patches via email to xen-devel. See the list info page for subscription information and the archives.

If asking questions, please refer to Asking Developer Questions. If sending patches, please see Submitting Xen Patches.

For non-developers, even just testing your favorite workload, and observing and reporting any any regressions, improvements, and so on, would be much appreciated. Remember that a scheduler's job is only hard when there's more work to do than there is CPU to do it, so make sure you're focusing on multiple competing workloads.

Readiness Criteria

This is an open issue: when shall Credit2 (but also a scheduler in general) be considered ready for production use, when ready for becoming Xen default scheduler? What we need is a standardized set of regression tests.

Of course, that will greatly help in future development too. In fact, if anyone proposes a change, we can test the change for any regressions in other workloads.

Status

Credit2 is still marked as experimental software. Development is ongoing in order to make it suitable for production usage. Broadly speaking this is the current situation:

  • stable and reliable; it is part of Xen upstream and of the OSSTest automated test suite since ages
  • benchmarking showed improved fairness, with respect to Credit1, especially in latency sensitive workloads
  • lacking extensive testing with many and diverse combinations of workloads
  • lacking some optimization; both the crediting algorithm and the load balancing logic are, computationally, rather expensive (more than Credit1)

Missing Features

  • CPU Topology:
  • Credit Algorithm:
    • Cap and Reservation: both features are available in Credit1, and they should be implemented in Credit2 too. Status: no one working on this. Dario <dario_DOT_faggioli_AT_citrix_DOT_com> will look into this at some point... In case someone is interested, speak up! Time horizon: undefined.
    • Credit math optimization: as of now, for the sake of correctness, every credit update involves a division, which may be slow. Some profiling and (most likely) optimization are required. It may be best to wait on this until the cap and reservation system is working, because the algorithm may change somewhat. Status: no one working on this. Time horizon: undefined

Open Issues

Here they are some open questions. Open problems that, although not critical, looks worth investigating:

  • How to deal well with mixed workload, e.g., situations where someone is listening to iTunes and there's a flash ad in background burning 100% of the CPU? Can we allow the audio to play well while still giving the VM only its fair share?
  • There exists a lot of CPU and/or IO throughput benchmarks, automatically outputting tons of numbers for one to look at. But as far as, e.g., audio and video are concerned, what are ways of automatically and programmatically measuring quality?

Milestones

  • 2010 24 December: Simple load balancer checked into xen-unstable; credit2 can now handle multiple sockets.
  • 2010 December: Credit2 integrated into the OSSTest automated test suite
  • 2010 14 April: Credit2 checked into xen-unstable
  • Prototype credit2 scheduler designed for latency-sensitive workloads and very large systems included in Xen 4.1

References

  • "In Perfect Xen, a Performance Study of the Emerging Xen Scheduler" (Dec 2013) text
  • XenSummit Asia 2009: a brief summary of the design target, goals, interface, credit design, and load balancing design. (text, slides)