Difference between revisions of "Xenopsd"

From Xen
m
Line 18: Line 18:
 
# xenopsd should be able to handle operations on managed VMs concurrently (rather than serially) and avoid operations on one VM having dependencies on other VMs (eg VM_suspend should not depend on a dom0 VBD_plug)
 
# xenopsd should be able to handle operations on managed VMs concurrently (rather than serially) and avoid operations on one VM having dependencies on other VMs (eg VM_suspend should not depend on a dom0 VBD_plug)
 
# xenopsd should be aware of driver-domains; there shall be no hardcoded "domid = 0" anywhere in the code
 
# xenopsd should be aware of driver-domains; there shall be no hardcoded "domid = 0" anywhere in the code
  +
# xenopsd should support cancellation for all blocking operations
   
 
== Design overview ==
 
== Design overview ==
  +
  +
At the high-level, we'll have a JSON-rpc interface whose concrete IDL is created with rpc-light. This will support operations to:
  +
# add/modify/remove VM metadata
  +
# start/stop/suspend/resume/migrate VMs
  +
# blocking poll for events
  +
Operations received through the interface will either be processed immediately (eg an event poll) or be queued, with one queue per VM. This ensures that VM operations are handled sequentially. The queue contents will be introspectable and cancellable through the API.
  +
  +
A thread pool with a configurable number of threads will take operations from the per-VM queues and execute them.
  +
  +
Machine-affecting operations (e.g. domain create, invocations of libxenguest and, in the future, libxl) will be through a backend plugin. There will be two plugins initially:
  +
# xen: this will use the existing xapi domain management code
  +
# simulator: this will pretend to manage domains without actually generating any side-effects.
  +
The backend interface will be as simple as possible. The backend will be selectable at load-time.
  +
  +
The "xen" backend will monitor the domains running on a host, and look for:
  +
# domain creations
  +
# domain shutdowns
  +
# xenstore device frontend/backend changes
  +
Any changes on a managed VM will be converted into special messages:
  +
# VM_check_state
  +
# VBD_check_state
  +
These messages will be put in the appropriate per-VM queue and will cause the higher-level logic to poll the current state of the device (which may have changed since the event was triggered) and
  +
# initiate reboots
  +
# cleanup device state (e.g. call the SMAPI vdi_detach)
  +
  +
The "simulator" backend will support a backdoor "trigger" operation which will allow a test harness to inject events.
   
 
[[File:xenopsd-internals.svg]]
 
[[File:xenopsd-internals.svg]]

Revision as of 21:27, 30 December 2011

The xenops daemon

This is an attempt to improve the architecture of the XCP toolstack by splitting *domain management* into a separate service, known as the *xenops daemon*. This should have the following advantages:

  1. it helps untangle the domain management code from the rest of xapi (which will focus on multi-host resource pool management)
  2. it forces us to be explicit about the interface xapi needs, making the future libxl port more predictable and likely to succeed
  3. it forces us to be explicit about the interface xenopsd needs, preventing (for example) the accidental insertion of a expensive RPC to a remote host through an internal xapi interface in a hot codepath. This is currently a very easy mistake to make and a better architecture would make it harder.

Design principles

  1. xenopsd should ignore domains that it is not managing. This will allow (eg) domains started with 'xl' to safely co-exist
    1. xenopsd could therefore be told to manage one VM, like 'xenvm' (part of XCI)
    2. xenopsd could therefore be told to manage lots of VMs, like 'xapi' currently does
  2. xenopsd should be capable of servicing all guest requests without making remote RPCs. This will allow (eg) VMs to reboot in an XCP resource pool even when the master is offline.
  3. xenopsd should not require xapi to be running (although it will require a storage service to be available in order to attach new disks)
  4. xenopsd should have a "simulator" backend as well as a real "xen" backend to allow us to replace the current "XIU" simulation system which doesn't support all operations (e.g. VM migrate)
  5. xenopsd should have a simulator-based test suite to allow us to test the component at build time.
  6. xenopsd should support a level-triggered (rather than edge-triggered) event system, to allow clients (ie 'xapi') to resynchronise as quickly as possible. It will be assumed that clients always want to 'diff' their current state against the 'tip' state and that they don't care about the exact sequence of historical changes (those are left for the audit log).
  7. xenopsd should be able to handle operations on managed VMs concurrently (rather than serially) and avoid operations on one VM having dependencies on other VMs (eg VM_suspend should not depend on a dom0 VBD_plug)
  8. xenopsd should be aware of driver-domains; there shall be no hardcoded "domid = 0" anywhere in the code
  9. xenopsd should support cancellation for all blocking operations

Design overview

At the high-level, we'll have a JSON-rpc interface whose concrete IDL is created with rpc-light. This will support operations to:

  1. add/modify/remove VM metadata
  2. start/stop/suspend/resume/migrate VMs
  3. blocking poll for events

Operations received through the interface will either be processed immediately (eg an event poll) or be queued, with one queue per VM. This ensures that VM operations are handled sequentially. The queue contents will be introspectable and cancellable through the API.

A thread pool with a configurable number of threads will take operations from the per-VM queues and execute them.

Machine-affecting operations (e.g. domain create, invocations of libxenguest and, in the future, libxl) will be through a backend plugin. There will be two plugins initially:

  1. xen: this will use the existing xapi domain management code
  2. simulator: this will pretend to manage domains without actually generating any side-effects.

The backend interface will be as simple as possible. The backend will be selectable at load-time.

The "xen" backend will monitor the domains running on a host, and look for:

  1. domain creations
  2. domain shutdowns
  3. xenstore device frontend/backend changes

Any changes on a managed VM will be converted into special messages:

  1. VM_check_state
  2. VBD_check_state

These messages will be put in the appropriate per-VM queue and will cause the higher-level logic to poll the current state of the device (which may have changed since the event was triggered) and

  1. initiate reboots
  2. cleanup device state (e.g. call the SMAPI vdi_detach)

The "simulator" backend will support a backdoor "trigger" operation which will allow a test harness to inject events.

Xenopsd-internals.svg

Code status