Live-Updating Xen

Live-Updating Xen

Current State

Merged upstream

• Multiboot2 support (i.e. relocation support) merged in kexec-tools v2.0.20

Posted upstream, in review

Early cleanups and fixes (v1): https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00000.html

TODO: For early `vmap()` we really want to make it officially OK to free boot-allocated pages with `free_xenheap_pages()` and even `free_domheap_pages()`. This involves fixing the esoteric corner cases in which it currently (rarely) doesn't quite work. Plan is to merge the `PGC_allocated` bit into the `PGC_state` bits, giving us 3 bits which can encode 8 states, of which 6 are currently valid: { inuse, offlining, broken_offlining, offline, broken, free }. We use the all-zeroes as 'never touched by the heap' moving inuse to 1, and then free_xenheap_pages() and free_domheap_pages() can check for that and call init_heap_pages() instead of free_heap_pages() if necessary. And we have one spare state for future use. (Varad)

Posted as RFC

• Physical memory management over kexec Handover protocol documentation Potentially out of date PDF version
• Management of live update data stream passing domains' state from Running Xen to Target Xen.
• Definition of state record format based on migration stream record format.
• Reservation of domain-owned pages in Target Xen as heap allocator starts up.

In development hacks

• PV domain save/restore over kexec with certain caveats.

<dwmw2_gone>    [root@localhost ~]# xl info | grep cc_compile_date
<dwmw2_gone>    cc_compile_date        : Wed Jan 22 21:10:38 GMT 2020
<dwmw2_gone>    [root@localhost ~]# KEXEC_LIVE_UPATE=1  ./kexec-tools/build/sbin/kexec xen2 --append="console=vga,com1 crashkernel=128M<4G no-real-mode insert_l1d_flush=0 dom0_max_vcpus=1 liveupdate=128M@2936M:0xb7800000"  --mem-min=0xaf800000 -t multiboot2-x86 -f
<dwmw2_gone>    can't get linerar framebuffer address
<dwmw2_gone>    kexec failed: Invalid argument
<dwmw2_gone>    [root@localhost ~]# xl info | grep cc_compile_date
<dwmw2_gone>    cc_compile_date        : Wed Jan 22 21:45:36 GMT 2020
<dwmw2_gone>    Wheee. Really must fix that -EINVAL :)
<andyhhp>       is that a kexec reload actually preserving dom0?
<dwmw2_gone>    yep
<andyhhp>       ship it :)
<dwmw2_gone>    a carefully configured dom0 with 2l event channels, one vcpu

Being worked on

• Pass M2P over (dwmw2)
• Refactor internal LU_DOMAIN_INFO record to post upstream (dwmw2)
• Refactor internal page list record into a single uint64_t per contiguous range of MFNs of the same time (dwmw2)
• Continue fixing PV Dom0 (Julien / Varad)
  • Support more than one vCPU (Varad)
  • FIFO event channels (Varad)
• Refactor internal PV save/restore (once it's all working perfectly) for posting upstream; especially all the hacks through domain creation (TBD)
• Save/restore HVM domains (Julien)
• Upstreaming Guest transparent HVM migration support (Paul)

• kexec-tools `--live-update` support including memory layout based on `KEXEC_RANGE_MA_LIVEUPDATE` and `liveupdate=` command line. (Varad)

Initial proof-of-concept with patches from Varad's tree (link below) - no kexec involved:

Boot xen with domkill_leakguest cmdline param.
Save a PV domain state, leave guest memory in the RAM:
# xl save -s domU domU.img 

Restore domain state reusing magic mfns. The shared_info page contents are preserved:
# xl restore -T domU.img <l3tab_mfn> <l2tab_mfn> <shared_info_mfn>

TODO: Restore console, reconstruct guest pagetables from shared_info.

Development trees

• http://git.infradead.org/users/dwmw2/xen.git/shortlog/refs/heads/bootcleanup
• https://xenbits.xen.org/gitweb/?p=people/dwmw2/xen.git;a=shortlog;h=refs/heads/lu-master
• https://github.com/varadgautam/xen/tree/liveupdate-devel

TODO

This list will move to the JIRA instance

• Devel milestone: PV domU persists across domain destroy/create
• Dom0 persists across kexec
• HVM guests persist across kexec
• PV guests persist across kexec
• One guest persists across kexec
• Multiple guests persist across kexec
• Guests exercise workloads
• Update to same Xen binary as the Target Xen
• Update to a Xen binary with a minor change, like a new printk
• Update to a Xen binary with a fix for an XSA
• Update to a new minor version
• Update to a new major version

More information

• Slides from Xen Summit 2019
• Video recording of the Xen Summit 2019 talk

Design Session Notes from Xen Summit 2019

• Brief project overview:
  • We want to build Xen Live-update
  • early prototyping phase
  • IDEA: change running hypervisor to new one without guest disruptions
  • Reasons:
    • Security - we might need an updated versions for vuln mitigation
    • Development cycle accelaration - fast switch to hypervisor during dev
    • Maintainability - reduce version diversity in the fleet
  • We are currently eyeing a combination of guest transparent live migration and kexec into a new xen buildb
  • For more details: Live-Update talk

• Terminology:
  • Running Xen -> The xen running on the host before update (Source)
  • Target Xen -> The xen we are updating *to*

• Design discussions:

• Live-update ties into multiple other projects currently done in the Xen-project:
  • Secret free Xen: reduce the footprint of guest relevant data in Xen
    • less state we might have to handle in the live update case
  • dom0less: bootstrap domains without the involvement of dom0
    • this might come in handy to at least setup and continue dom0 on target xen
    • If we have this this might also enable us to de-serialize the state for other guest-domains in xen and not have to wait for dom0 to do this

• We want to just keep domain and hardware state
  • Xen is supposedly completely to be exchanged
  • We have to keep around the IOMMU page tables and do not touch them
    • this might also come in handy for some newer UEFI boot related issues?
    • We might have to go and reinject certain interrupts
  • do we need to dis-aggregate xenheap and domheap here?
    • We are currently trying to avoid this

• A key stepstone for Live-update is guest transparent live migration
  • This means we are using a well defined ABI for saving/restoring domain state
    • We do only rely on domain state and no internal xen state
  • The idea is to migrate the guest not from one machine to another (in space) but on the same machine from one hypervisor to another (in time)
  • In addition we want to keep as much as possible in memory unchanged and feed this back to the target domain in order to save time
  • This means we will need additional info on those memory areas and have to be super careful not to stomp over them while starting the target xen
  • for live migration: domid is a problem in this case
    • randomize and pray does not work on smaller fleets
    • this is not a problem for live-update
    • BUT: as a community we shoudl make this restriction go away

• Exchanging the Hypervisor using kexec
  • We have patches on upstream kexec-tools merged that enable multiboot2 for Xen
  • We can now load the target xen binary to the crashdump region to not stomp over any valuable date we might need later
  • But using the crashdump region for this has drawbacks when it comes to debugging and we might want to think about this later
    • What happens when live-update goes wrong?
    • Option: Increase Crashdump region size and partition it or have a separate reserved live-update region to load the target xen into
    • Separate region or partitoned region is not a priority for V1 but should be on the road map for future versions

• Who serializes and deserializes domain state?
  • dom0: This should work fine, but who does this for dom0 itself?
  • Xen: This will need some more work, but might covered mostly by the dom0less effort on the arm side
    • this will need some work for x86, but Stefano does not consider this a lot of work
  • This would mean: serialize domain state into multiboot module and set domains up after kexecing xen in the dom0less manner
    • make multiboot module general enough so we can tag it as boot/resume/create/etc.
      • this will also enable us to do per-guest feature enablement
      • finer granular than specifying on cmdline
      • cmdline stuff is mostly broken, needs to be fixed for nested either way
      • domain create flags is a mess

• Live update instead of crashdump?
  • Can we use such capabilities to recover from a crash be "restarting" xen on a crash?
    • live updating into (the same) xen on crash
  • crashing is a good mechanism because it happens if something is really broken and most likely not recoverable
  • Live update should be a concious process and not something you do as reaction to a crash
    • something is really broken if we crash
    • we should not proactively restart xen on crash
      • we might run into crash loops
  • maybe this can be done in the future, but it is not changing anything for the design
    • if anybody wants to wire this up once live update is there, that should not be too hard
    • then you want to think about: scattering the domains to multiple other hosts to not keep them on broken machines

• We should use this opportunity to clean up certain parts of the code base:
  • interface for domain information is a mess
    • HVM and PV have some shared data but completely different ways of accessing it

• Volume of patches:
  • Live update: still developing, we do not know yet
  • guest transparent live migration:
    • We have roughly 100 patches over time
    • we believe most of this has just to be cleaned up/squashed and will land us at a reasonable much lower number
    • this also needs 2-3 dom0 kernel patches

• Summary of action items:
  • coordinate with dom0less effort on what we can use and contribute there
  • fix the domid clash problem
  • Decision on usage of crash kernel area
  • fix live migration patch set to include yet unsupported backends
    • clean up the patch set
    • upstream it

• Longer term vision:
  • Have a tiny hypervisor between Guest and Xen that handles the common cases
    • this enables (almost) zero downtime for the guest
    • the tiny hypervisor will maintain the guest while the underlying xen is kexecing into new build

• Somebody someday will want to get rid of the long tail of old xen versions in a fleet
  • live patch old running versions with live update capability?
  • crashdumping into a new hypervisor?
    • "crazy idea" but this will likely come up at some point