Difference between revisions of "LivePatch"

From Xen
Line 240: Line 240:
== xsplice-build--tools ==
== xsplice-build--tools ==
* Provide better error-code - saying what is unsupported if the user tries to do it.
* Able to generate payloads that NOP functions.
* Able to generate payloads that NOP functions.
* Able to generate payloads only against .data sections.
* Able to generate payloads only against .data sections.

Revision as of 13:37, 10 May 2016


Xen 4.7 implements v1 of the xSplice Design.

There are three parts to utilize this technology:

  • Hypervisor implementing the XSPLICE_SYSCTL hypercall
  • Tool to upload/apply/revert the payloads: xen-xsplice
  • Tool to generate the payloads.

The first two are part of the Xen Project source:


while the last one is in a seperate repository:


How to enable it

The code for hypervisor is NOT enabled by default. When compiling the hypervisor one must change the config to build it.

It can be done by editing xen/.config to have CONFIG_XSPLICE=y or using:

 $ git clone git://xenbits.xen.org/xen.git
 $ cd xen/xen
 $ make menuconfig

And selecting Common Features|xSplice live patching support.

It is marked as TECH PREVIEW.

Once you have it built and booted, you can verify that xSplice is built in via:

 #xl dmesg | grep xsplice
 (XEN) xsplice: : build-id: 578109e77f259ef01d50dc756006224cb3a36f92

or alternatively via:

 # xl info | grep build_id
 build_id               : 578109e77f259ef01d50dc756006224cb3a36f92

The value: 578109e77f259ef01d50dc756006224cb3a36f92 is different on every build. (This build-id is used to keep track of payloads dependencies and make sure they are built against the correct hypervisor built).

The tool to patch/load/etc is called xen-xsplice and is also part of the Xen source tree. It is used to load/apply/revert/unload the payloads.

ELF payload file

The design details how the structures look, see Design of payload format for details.

The external tool:


can generate the payloads against the hypervisor.

Build hot-patches

To build hot-patches aka payloads aka binary fixes against the hypervisor, you need two things:

  • Xen hypervisor which supports xSplice hypercalls.
  • Tool to build binary fixes against hypervisor.

The tool is available at at:


Or alternatively the three examples that are part of the Xen code base.


The workflow of the tool is to:

  • Build a Xen hypervisor without the patch (alternatively one can just point it to xen-syms which is preferred.
  • Build a Xen hypervisor with a patch.
  • Compare the two above.
  • Create an payload with the difference.

The tool is not capable of:

  • Generating payloads only with .data changes (as in only for variables),
  • Inline patching - it only patches the whole function,
  • Cannot NOP out functions.

This explanation was lifted from this link and modified a bit.

This transcript should demonstrate how to use the tool.

NOTEThe payloads need to embedded an build-id of the hypervisor it will be loaded against. That means one MUST NOT recompile the target Xen hypervisor. Hence the workload here is done on a copy of the source.

This assumes that the xen-syms built originally is in /boot/xen-syms We use /tmp to store a copy of the source. The version of the source code MUST match the version that /boot/xen-syms was built. One can easily verify that by (on target system):

 # xl info | grep xen_changeset
 $ Fri Apr 15 10:35:21 2016 -0400 git:7552904

 $ cd /tmp/
 $ git clone git://xenbits.xen.org/xen.git
 $ cd xen
 $ git checkout 7552904
 $ # Build a debug Xen. Make sure XSPLICE is enabled
 $ cat xen/.config | grep XSPLICE
 $ # Write a patch
 $ mkdir ~/work
 $ git diff > ~/work/test1.patch
 $ git reset --hard
 $ # Write another patch
 $ git diff > ~/work/test2.patch
 $ git reset --hard
 $ # Write another patch
 $ git diff > ~/work/test3.patch
 $ git reset --hard
 $ cd /tmp
 $ git clone git://xenbits.xen.org/people/konradwilk/xsplice-build-tools.git
 $ cd xsplice-build-tools
 $ make
 $ ./xsplice-build -s /tmp/xen -p ~/work/test1.patch -o out1 --xen-debug --debug --xen-syms /boot/xen-syms
 $ ./xsplice-build -s /tmp/xen -p ~/work/test2.patch -o out2 --xen-debug --debug --xen-syms /boot/xen-syms
 $ ./xsplice-build -s /tmp/xen -p ~/work/test3.patch -o out3 --xen-debug --debug --xen-syms /boot/xen-syms
 $ # copy out*/test*.xsplice onto the host (or if locally skip this step)
 On the host:
 # xen-xsplice upload test1 test1.xsplice
 # xen-xsplice upload test2 test2.xsplice
 # xen-xsplice upload test3 test3.xsplice
 # xen-xsplice apply test1
 # # Verify test1 is applied
 # xen-xsplice apply test2
 # # Verify test2 is also applied
 # xen-xsplice replace test3
 # # Verify test3 is applied and test1 and test2 are not
 # xen-xsplice revert test3
 # # Verify test3 is not applied
 # xen-xsplice unload test1
 # xen-xsplice unload test2
 # xen-xsplice unload test3

How to build built-in examples

The Xen Project also includes three regression test-cases that can be built against the hypervisor which modify the xen_extra_version. This provides an simple way to verify the changes as:

 # xl info | grep extra_version

will verify that the patch has taken place.

There are three regression/test-cases:

  • xen_hello_world: changes xen_extra_version() to return "Hello World".
  • xen_bye_world: over-writes the xen_hello_world patch so it returns "Bye World."
  • xen_replace_world: Used to replace the two other payloads and make xen_extra_version() print "Hello Again World".

To build them:

 $ cd xen
 $ make -C xen tests
 $ find . -name *.xsplice

T The idea is to do (on host with Xen hypervisor that has xSplice built-in):

 $ # copy them the host.
 # cd /usr/lib/debug
 # ls
 xen-syms-4.7-unstable  xen_bye_world.xsplice  xen_hello_world.xsplice  
 # xen-xsplice load xen_hello_world.xsplice 
 Uploading xen_hello_world.xsplice (12336 bytes)
 Performing apply:. completed
 # xl info | grep extra
 xen_extra              : Hello World
 # xen-xsplice load xen_bye_world.xsplice 
 Uploading xen_bye_world.xsplice (9244 bytes)
 Performing apply:. completed
 # xl info | grep extra
 xen_extra              : Bye World!
 # xen-xsplice upload replace_bye_and_hello xen_replace_world.xsplice 
 Uploading xen_replace_world.xsplice (9328 bytes)
 # xen-xsplice list
  ID                                     | status
 xen_hello_world                         | APPLIED
 xen_bye_world                           | APPLIED
 replace_bye_and_hello                   | CHECKED
 # xen-xsplice replace replace_bye_and_hello
 Performing replace:. completed
 # xl info | grep extra
 xen_extra              : Hello Again World!
 # xen-xsplice list
  ID                                     | status
 xen_hello_world                         | CHECKED
 xen_bye_world                           | CHECKED
 replace_bye_and_hello                   | APPLIED


We will have monthly IRC meetings to discuss code/issues/etc.

They are on the fourth Tuesday of every month. At 10:00AM EST on #xsplice channel on irc.freenode.net


v1 of patchset is in the Xen code base. We still have a list of TODOs and track it here.


Make OSSTest do regression tests.


Things that need to be done in there:

  • xen-xsplice better smarts. Right now you need to do 'upload','check' and 'apply'. Three operations which could be folded in one - since the system admin probably want to have all of those done at once. The 'all' option does that.
  • Instead of using 'xen-xsplice' we should use 'xl'. That above mentioned logic could be part of 'xl' while 'xen-xsplice' is an accessory function for expert users?
  • If hypervisor is compiled without xSplice the tool should print simply '"xsplice support not available in Xen'
  • If the build-id is wrong and one tries to apply the payload the message of 'hello failed with 1(Operation not permitted)' is lacking. We should have more details (and maybe a different error?). Or perhaps check the build-d as as being upload (however we may load them out of order). Perhaps have the toolstack check the build-id? How to do that against loaded patches (have them locally saved?)


  • Provide better error-code - saying what is unsupported if the user tries to do it.
  • Able to generate payloads that NOP functions.
  • Able to generate payloads only against .data sections.


Currently the implementation has many TODOs implemented.

However we need to expand the hypervisor to include support for:

  • build_id support Done!
  • The check for --build-id needs to deal with localized binutils.
  • Make .xsplice.funcs be an SHF_MERGE type with .sh_entsize having the size of the struct.
  • Make the usage of tools/symbols use common list model
  • Need 'dl_sym' functionality in hypervisor to parse ELF payload. Do dynamic linking. (Ross) Done!
  • Lookup and insertion in the symbol table. Compute the proper offset and virtual address based on symbol name (<do_domctl>) (Ross) Done!
  • When figuring out the new_addr and the name is <symbol>+<offset> - compute that.
  • Need to either remove duplicate names from the symbol table or have the hypervisor implement proper ELF parsing - integrate the elf sections and parse that during runtime. Done!
  • Insertion of exceptions in the exception table. Also support resorting it. The hot-patch may have new exceptions that need to be dealt with.Done!
  • Code to verify signature of the ELF payload.
  • Checking the build-id against the hypervisor.Done!
  • Code to patch and revert the hot-patch (for simplicity we could boot Xen with 'cpus=1' so we don't have to worry about proper code to trigger patching)Done!
    • Patching requires saving the initial state. That needs to be saved for revert.Done!
    • Patching needs allocation of memory, putting in the code at memory.Done!
    • Doing the verification - checking if any code that uses the to-be-patched-function is in use. Save what the virtual address for the functions is - and verify the stack when patching (this incidentally is what .xsplice_section structures can have).
    • Deal with proper page table types for the modification (ro for .data, x for .text, etc).Done!
    • Code to revert.Done!
  • Code to activate patching mechanism. On VMEXIT routine right before checking the do_softirq. Ditto for PV paths. Call the top patching mechanism - which can IPI other CPUs (or wait until all CPUs have come to the same point).Done!
  • The code for doing the patching - call from __do_softirq - can also be used for runtime microcode loading.
  • Add functionality to execute ELF code.Done!
  • Patch with NOPs if .xsplice.funcs->new_size has 0.
  • Seperate pages for .text; .rodata; .bss and .data
  • A better mechanism to "mask" NMIs during patching. The existing mechanism looses NMI if they have been sent and we don't have a mechanism to replay them. Note that this is also fixes alternative section patching.
  • Make it work under ARM32:
    • Support ARM32 ELF relocations
    • Support ARM32 virtual addresses
    • Support a similar to CPU flushing semantics as x86
    • Support NMI? (or something similar?)
  • Make it work under ARM64:
    • Support ARM64 ELF relocations
    • Support ARM64 virtual addresses
    • Support a similar to CPU flushing semantics as x86
    • Support NMI? (or something similar?)