Tuning Xen for Performance
Tuning your Xen installation: recommended settings
Storage options
LVM is the fastest storage backend available but it limits the options for live migration. See Storage_options for more details.
Memory
If the host has more memory than a typical laptop/desktop system, then do not rely on dom0 ballooning. Instead set the dom0 memory to be something between 1 and 4GB adding dom0_mem=1024M to the Xen command line.
1GB is enough for a pretty large host, more will be needed if you expect your users to use advanced storage types as ZFS or distributed filesystems.
The reason to not give all ram to dom0 is that it takes time to re-purpose it went domUs are started. On a host with 128GB Ram you're getting things stuck for minutes while dom0 is ballooning down to make space.
Dom0 VCPUs
If the host has more then 16 cores, limiting the number of Dom0 VCPUs to 8 and pinning them can improve performances. See: Can I dedicate a cpu core (or cores) only for dom0? .
HAP vs. shadow
HAP stands for hardware assisted paging and requires a CPU feature called EPT by Intel and RVI by AMD. It is used to manage the guest's MMU. The alternative is shadow paging, completely managed in software by Xen. On HAP TLB misses are expensive so if you have really random access, HAP will be expensive. On shadow page table updates are expensive. HAP is enabled by default (and it is the recommended setting) but can be disabled passing hap=0 in the VM config file.
PV vs PV on HVM
Linux, NetBSD, FreeBSD and Solaris can run as PV or PV on HVM guests (see this page for instructions on how to install a Debian PV guest and this page for an example PV on HVM config file). Memory intensive workloads that involve the continuous creation and destruction of page tables can perform better when run in a PV on HVM guest. Examples are kernbench and sql-bench. On the other hand memory workloads that run on a quasi-static set of page tables run better on a PV guests. An example of this kind of workloads is specjbb. See Xen_Linux_PV_on_HVM_drivers#Performance_Tradeoffs for more details.
