XenParavirtOps: Difference between revisions

From Xen
Jump to navigationJump to search
(Add missed square bracket for add into category Developers working)
No edit summary
Line 254: Line 254:
# [[#troubleshoot|Troubleshoot]] if necessary
# [[#troubleshoot|Troubleshoot]] if necessary


=== Getting the current stable version ===
=== Getting the mainline Linux kernel ===
<span id="stable_version"></span>
<span id="build_linux_kernel"></span>
With [http://blog.xen.org/index.php/2011/06/14/linux-3-0-how-did-we-get-initial-domain-dom0-support-there/ v3.0 - how did dom0 support get there] has technical details, but the summary is that the 3.0 is "With Linux 3.0 we now have the major components to be classified as a working initial domain – aka – dom0.". There are bugs, which are documented in [http://wiki.xensource.com/xenwiki/Linux_30_bugs Linux 3.0 and further bugs]


In order to get a proper Xen mainline kernel please check: [[Mainline_Linux_Kernel_Configs]]
To download it: [ftp://ftp.kernel.org/pub/linux/kernel/v3.0/linux-3.0.8.tar.bz2]

or the latest from [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=summary Linus Torvalds tree]: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

=== Getting the current development version ===
<span id="dev_version"></span>
The development is happening in Jeremy's [git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
] and Konrad's [git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git] git trees.

==== xen.git branches ====

The current day-to-day development is happening in a [http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=summary Konrad's git tree] and [http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=summary Jeremy's git tree]

Both repositories have numerous topic branches to track individual lines of development, and a couple of roll-up branches which contain everything merged together for easy compilation and running.

'''NOTE! All active git branches require at least Xen 4.0.1, using older version (4.0.0 or older) will cause problems, xend not starting, etc.'''

Current active branches are:

* linux-next (in Konrad's repository) upstream/xen (in Jeremy's tree) - patches that are queued up
to go to Linus Torvalad's tree. This includes bug-fixes, new features, etc.
* testing (in Konrad's repository) is off patches being run/developed and obviously going through testing - which means they might not compile or work properly.

=== Downloading the git tree ===

<pre><nowiki>
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux
$ cd linux
</nowiki></pre>

or:

<pre><nowiki>
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git linux
$ cd linux
</nowiki></pre>


that will automatically check out the 'origin/master' default branch. Note that you need at least 256 MB of free memory, otherwise the "git clone" will fail.

Changing the branch: You most probably want to use the "linux-next" (or "upstream/xen") branch, so do this:


<pre><nowiki>
$ cd /usr/src
$ cd linux
$ git reset --hard
$ git checkout -b upstream/xen origin/upstream/xen
Branch upstream/xen set up to track remote branch refs/remotes/origin/upstream/xen
Switched to a new branch "upstream/xen"
$ git pull
$ git log | less
</nowiki></pre>


(or replace "origin/upstream/xen" with "testing" for example)

Later when you want to update the tree use:


<pre><nowiki>
$ cd linux
$ make clean
$ git pull
</nowiki></pre>


=== Configuring the kernel ===
<span id="configure_xen"></span>

Configure as normal; you can start with your current .config file or use '''make defconfig'''
and then depending on what you want your kernel to do (it can do both):


<pre><nowiki>
make menuconfig
</nowiki></pre>

NOTE0: Make sure you have correct CPU type (Processor Family) set in the kernel configuration, Xen Dom0 options won't show up at all if you have too old CPU selected (too old means a CPU that doesn't support PAE; Pentium Pro was the first CPU to have PAE).

NOTE1: If you're building 32 bit version of the kernel, you first need to enable PAE support, since Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won't show up at all before you've enabled PAE for 32 bit builds (Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support). PAE is not needed for 64 bit kernels.

==== Configure kernel for domU support ====

# If building 32 bit kernel make sure you have CONFIG_X86_PAE enabled (which is set by selecting CONFIG_HIGHMEM64G)
#* non-PAE mode doesn't work in 2.6.25, and has been dropped altogether from 2.6.26 and newer kernel versions.
# Enable these core options (Processor type and features| Paravirtualized guest support]
#* CONFIG_PARAVIRT=y
#* CONFIG_XEN=y
#* CONFIG_PARAVIRT_GUEST=y
#* CONFIG_PARAVIRT_SPINLOCKS=y
# And Xen pv console device support (Device Drivers|Character devices
#* CONFIG_HVC_DRIVER=y
#* CONFIG_HVC_XEN=y
# And Xen disk and network support (Device Drivers|Block devices and Device Drivers|Network device support)
#* CONFIG_XEN_FBDEV_FRONTEND=y
#* CONFIG_XEN_BLKDEV_FRONTEND=y
#* CONFIG_XEN_NETDEV_FRONTEND=y
# And the rest (Device Drivers|Xen driver support)
#* CONFIG_XEN_PCIDEV_FRONTEND=y
#* CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
#* CONFIG_XEN_FBDEV_FRONTEND=y
#* CONFIG_XEN_XENBUS_FRONTEND=y
#* CONFIG_XEN_SAVE_RESTORE=y
#* CONFIG_XEN_GRANT_DEV_ALLOC=m
# And for tmem support:
#* CONFIG_XEN_TMEM=y
#* CONFIG_CLEANCACHE=y
#* CONFIG_FRONTSWAP=y
#* CONFIG_XEN_SELFBALLOONING=y

==== Configure kernel for dom0 support ====

NOTE: Xen dom0 support depends on ACPI support. Make sure you enable ACPI support or you won't see Dom0 options at all.

In addition to the config options above you also need to enable:

* CONFIG_X86_IO_APIC=y
* CONFIG_ACPI=y
* CONFIG_ACPI_PROCFS=y (optional)
* CONFIG_XEN_DOM0=y
* CONFIG_PCI_XEN=y
* CONFIG_XEN_DEV_EVTCHN=y
* CONFIG_XENFS=y
* CONFIG_XEN_COMPAT_XENFS=y
* CONFIG_XEN_SYS_HYPERVISOR=y
* CONFIG_XEN_GNTDEV=y
* CONFIG_XEN_BACKEND=y
* CONFIG_XEN_NETDEV_BACKEND=m
* CONFIG_XEN_BLKDEV_BACKEND=m
* CONFIG_XEN_PCIDEV_BACKEND=m
* CONFIG_XEN_PRIVILEGED_GUEST=y
* CONFIG_XEN_BALLOON=y
* CONFIG_XEN_SCRUB_PAGES=y
* CONFIG_XEN_DEV_EVTCHN=y
* CONFIG_XEN_GNTDEV=y

If you're using RHEL5 or CentOS5 as a dom0 (ie. you have old udev version), make sure you enable the following options as well:

* CONFIG_SYSFS_DEPRECATED=y
* CONFIG_SYSFS_DEPRECATED_V2=y

For more current Xen related config options check the example .config files from the troubleshooting section, and check the [[2.6.18-to-2.6.31-and-higher]] wiki page.

=== Building the kernel ===
<span id="build_kernel"></span>

There are two parts when building Xen: [[#build_xen|Xen]] and the Kernel. After Kernel configuration you have to do two things, build the kernel and build the ramdrive.

This is the Ubuntu way:

<pre><nowiki>
$ cd /usr/src/linux
$ sudo make
$ sudo apt-get install kernel-package fakeroot
$ export CONCURRENCY_LEVEL=<number_of_cores +1>
$ sudo make-kpkg clean
$ sudo fakeroot make-kpkg --initrd --append-to-version=-pv kernel-image kernel-headers
$ sudo dpkg -i ../<linux-image-3.0.XXX.deb>
</nowiki></pre>

The first line will make the kernel with the hidden configuration file (your file hopefully). You might have to do this several times because you will encounter errors that will requiere turning modules off, and start compiling again. It is a good plactice to use make clean when you change many modules at a time. The next line is used for dpkg but is not really that important. the sudo fakeroot line will create your ramdrive needed to boot. This is the debian way of making the image usable this will event copy it to boot but will not update grub.

The old-skool way:

<pre><nowiki>
cd linux
make clean
cp -a .config .config-old
make oldconfig
make menuconfig (if you need to change something)
make bzImage
make modules
make modules_install
# in the following lines replace "version" with the actual kernel version you're compiling.
make install
# And then generate initrd/initramfs image for your dom0 kernel, example for Fedora/RHEL/CentOS:
mkinitrd -f /boot/initrd-version.img version
# For Fedora Core the 'make install' does it automatically
</nowiki></pre>


==== Update modules ====
<span id="configure_modules"></span>

You need to update some new modules that your newly created kernel requires with XEN.


<pre><nowiki>
$ sudo vim /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
lp
rtc
# Added these lines
xen-evtchn
xen-gntdev
xen-netback
xen-blkback
xenfs
blktap
</nowiki></pre>

Again this is the Ubuntu way, find a way for your particular distribution to do the same.


=== Build Xen ===
=== Build Xen ===

Revision as of 18:46, 27 February 2012


What is paravirt_ops?

paravirt_ops (pv-ops for short) is a piece of Linux kernel infrastructure to allow it to run paravirtualized on a hypervisor. It currently supports VMWare's VMI, Rusty's lguest, and most interestingly, Xen.

The infrastructure allows you to compile a single kernel binary which will either boot native on bare hardware (or in hvm mode under Xen), or boot fully paravirtualized in any of the environments you've enabled in the kernel configuration, and lately also as Xen dom0.

It uses various techniques, such as binary patching, to make sure that the performance impact when running on bare hardware is effectively unmeasurable when compared to a non-paravirt_ops kernel.

At present paravirt_ops is available for x86_32, x86_64 and ia64 architectures.

Dom0 means the first guest that Xen boots. It usually (mostly always) is the one that has the driver support. The other guests that are booted (HVM or PV) are called DomU.

Current state in Linux kernel and in distributions

[Updated Oct 26th 2011]

Linux 3.0 (and later) can run as guest (domU) and as host (dom0). All necessary backends (and frontends) are in the upstream kernel. Look in Roadmap in the features section to get an idea what is happening in next kernel releases.

You can also visit the history section on the story of pvops.

Starting with Linux v2.6.37 dom0 support was added and with 3.0 having the necessary backend drivers. Fedora Core 16 and Ubuntu 11.10 ship with the 3.0 kernel (or later) and the user just needs to install the hypervisor: 

$yum install xen



$apt-get install xen

TODO: Verify the Ubuntu

Then reboot the machine and select Xen in the GRUB(2) menu. Also consult [[[XenBestPractices]]]

Are there other distributions that have a Xen dom0 kernels available?

Yes. See this wiki page for more information: [[[XenDom0Kernels]]] and [[[XenKernelFeatures]]]

Help! I am having troubles

First of, did you look in [[[XenCommonProblems]]] wiki page?

Did not find anything relevant? Was it any of these:

I have graphics card (DRM/TTM/KMS/Xorg) related problems with the pv_ops dom0 kernel..

Screen looks like a checker-board? See errors from the Radeon|Nouveau driver? If so please see the XenPVOPSDRM wiki page.

Dom0 console gets all weird and corrupted in the end of the boot process

Is the last line on the console something like Setting console screen modes and fonts ? Then you might want to disable "console-screen.sh" service from starting automatically and it should workaround the problem.

Dom0 console doesn't show any output and remains blank

Unfortunately, a patch that enables the xen dom0 kernel to use the VGA text console didn't made it into initial release of Linux 3.0.0. This patch was merged into Linux 3.1.0 and also to 3.0.2 stable update.

Maybe it is something obvious? How do I make the bootup more verbose?

If you do encounter problems, then getting as much information as possible is very helpful. If the domain crashes very early, before any output appears on the console, then booting with: earlyprintk=xen should provide some useful information. Note that earlyprintk=xen only works for domU/dom0 if you have Xen hypervisor built in debug mode! If you are running a debug build of Xen hypervisor (set "debug = y" in Config.mk in the Xen source tree), then you should get crash dumps on the Xen console. You can view those with "xm dmesg". Also, CTRL+O can be used to send SysRq (not really specific to pv_ops, but can be handy for kernel debugging).

OK, how do I add earlyprintk=xen and all that?

Edit /boot/grub/grub.conf (or /boot/grub2/grub.cfg) and make sure you have a correct grub entry to boot Xen hypervisor with dom0 kernel.

In grub.conf it's a good idea to enable all the logging options for Xen: loglvl=all guest_loglvl=all sync_console console_to_ring and for Linux pv_ops dom0 kernel: earlyprintk=xen debug loglevel=8, and set up a serial console to be able to see and capture the full boot messages from Xen and from dom0 kernel, in the case system doesn't start up properly or crashes.

For the full list of Xen bootup options, look in [[[XenHypervisorBootOptions]]] Here is an example: 


title        pv_ops dom0 debug (2.6.32.27) with serial console
root         (hd0,0)
kernel       /xen-4.0.gz dom0_mem=1024M loglvl=all guest_loglvl=all sync_console console_to_ring com1=115200,8n1 console=com1 lapic=debug apic_verbosity=debug apic=debug iommu=off
module       /vmlinuz-2.6.32.27 ro root=/dev/vg00/lv01 console=hvc0 earlyprintk=xen nomodeset initcall_debug debug loglevel=10
module       /initrd-2.6.32.27.img


I got a funky serial console (IPMI/AMT, etc) - Is there more information available how to debug and troubleshoot using a serial console?

For debugging and testing you should be using a computer with a built-in serial port on the motherboard (com1), or add a PCI serial card if your motherboard lacks a built-in serial port. You can also use SOL (Serial Over Lan) for logging the Xen hypervisor and dom0 kernel messages. Most server-class machines have SOL available through their management processor or IPMI. SOL device looks like a normal serial port for the OS/Xen, but enables you to connect to the serial console over a network, through the management processor.

For more details (and examples) please see [[[XenSerialConsole]]] wiki page.

Nothing is helping! Help!!

Before submitting bugreports to xen-devel mailinglist please read:

Please mail questions/answers/patches/etc to the Xen-devel mailing list.

Roadmap

Status updates:

Work in progress for 3.3

And naturally fixing bugs: Linux 3.0 and further has the list of the ones we are aware and working on.

Work in the future

Future work queued up (in no particular order):

  • xenwatch is single threaded with a mutex.In large installations (Amazon for example) this kernel thread causes all other guests to have to wait. Needs to be made parallel.
  • blkback support DIF/DIX [1] We should carry the DIF/DIX extra data along with the data requests through the PV layer. Not sure what this entails. Might need to expand the protocol similar to 'feature-discard'
  • blkback support serial numbers for PV guests.
  • pcifront: turn SWIOTLB when PCI devices are hot-plugged. There is code in lib/swiotlb.c that can allocate the SWIOTLB pool after early_bootmem.It is currently only used by IA64, but there should be no trouble extending it to be used under X86. It would allow us to have xen-pcifront automatically turn on SWIOTLB if it detects anything on the PV ring.
  • xen pciback. Do FLR on devices when they are taken out of the guest. Do DisINTx when guest has deallocated the PCI device.
  • fix balloon target driver to have a logarithmic scale of hitting the 'target'. Dan and Jan talked about it and Jan posted a patch for this some time ago. http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00143.html and http://lists.xensource.com/archives/html/xen-devel/2010-06/msg01076.html
  • netfront support skb coalescing. Borrow the ideas from Netchannel2 to coalesce the SKB's so that the backend does not have to jump all over the memory and instead can easily fetch the pages in a physical contingous manner.
  • IRQ bitmap ack. When a guest ACKs an interrupt (EOI) for IO-APICs interrupt, it needs to EOI it. There are two mechanisms to do it. paravirt has the slowest one where it makes a hypercall to EOI it. We need to implement the IRQ bitmap (PHYSDEVOP_pirq_eoi_gmfn) which does the EOI much faster.
  • Add EFI support in Linux pv-ops. Jan Beulich wrote patches to make Xen hypervisor be able to compile as an EFI application. "Build xen.efi, write up a config file for it to read (most importantly so it knows what Dom0 kernel and initrd to use), and you should be good to go (provided the EFI implementation isn't too flawed). This being an EFI application you can simply run it from the shell prompt. Parameter for Xen.efi is -cfg <file>, and <file> has: kernel=, ramdisk=, options=, video=gfx-x. The Linux pv-ops needs at least to parse XEN_VGATYPE_EFI_LFB data, E820 parsed (should be working) and give the ACPI subsystem a pointer to the ACPI DSDT without consulting EBDA.
  • microcode update. Jeremy wrote one http://oss.oracle.com/git/kwilk/xen.git/p=kwilk/xen.git;a=commit;h=0a49ceea0d032864a72a8744c82c3786a01f34f4 but sadly the upstream maintainer (Borislav, https://lkml.org/lkml/2011/1/30/108) was not too keen. He was thinking of making grub do the microcode update, but that sounds to have run afoul of what Ingo [x86 maintainer] thinks (http://mid.gmane.org/20110525193601.GE17864@elte.hu) in a different thread.
  • MTRR support. Usually this is used for graphics cards, but the 10G Myantic drivers do enable it on their PCIe MMIO space to speed up performance (sets it to Write Combine). We need to implement (or perhaps backport from 2.6.18) the Xen MTRR code.
  • oprofile/ftrace support. Linux upstream is using ftrace in favour of oprofile. We should implement ftrace to work under Xen.
  • remove VM_IO vesitages. Linux 3.0 has dropped most of the _PAGE_IOMAP code. There is one usage left in xen_make_pte and it should be redo to do proper M2P and P2M without consulting the _PAGE_IOMAP flag. Potentially remove the _PAGE_IOMAP usage in the 1-1 phys code. _PAGE_IOMAP used by mmap batches code to map foreign patches. Can be done with M2P override perhaps? The PAGE_IOMAP was used to say MFN==PFN. The outstanding issue is with xenfs doing hypercall on the behest of the tools - specifically mapping memory and providing the MFN value.
  • Device hotplug (MODULE_ALIAS) - missing for Xen PCI backend
  • netback zero-page copy: http://www.spinics.net/lists/linux-nfs/msg22575.html
  • hugepage support in PV.
  • PVSCSI drivers (scsiback backend driver for dom0 kernel and scsifront frontend driver for domU kernel). More info: http://wiki.xen.org/xenwiki/XenPVSCSI .
  • PVUSB drivers (usbback backend driver for dom0 kernel and usbfront frontend driver for domU kernel). More info: http://wiki.xen.org/xenwiki/XenUSBPassthrough . Or perhaps utilize usb-redir and QEMU support for passing through a USB device.
  • Suspend event channel support for faster checkpointing in Remus FT.

Changelog

  • Features in 2.6.26:
    • x86-32 support
    • SMP
    • Console (hvc0)
    • Blockfront (xvdX)
    • Netfront
    • Balloon (reversible contraction only)
    • paravirtual framebuffer + mouse (pvfb)
    • 2.6.26 onwards pv domU is PAE-only (on x86-32)
  • Features added in 2.6.27:
    • x86-64 support
    • Save/restore/migration
    • Further pvfb enhancements
  • Features added in 2.6.28:
    • ia64 (itanium) pv_ops xen domU support
    • Various bug fixes and cleanups
    • Expand Xen blkfront for > 16 xvd devices
    • Implement CPU hotplugging
    • Add debugfs support
  • Features added in 2.6.29:
    • bugfixes
    • performance improvements
    • swiotlb (required for dom0 support)
  • Features added in 2.6.30:
    • bugfixes.
  • Features added in 2.6.31:
    • bugfixes.
  • Features added in 2.6.32:
    • bugfixes.
  • Features added in 2.6.33:
    • bugfixes.
    • save/restore/migration bugfixes. These bugfixes can also be found from the 2.6.32.10 update.
  • Features added in 2.6.34:
    • bugfixes.
  • Features added in 2.6.35:
    • bugfixes.
  • Features added in 2.6.36:
    • Xen-SWIOTLB (required for Xen PCI frontend driver and Xen dom0 support).
    • Xen PV-on-HVM optimized paravirtualized drivers for fully virtualized (HVM) guest VMs.
    • Xen VBD (Virtual Block Device) online dynamic resize support for resizing guest disks (xvd*) on-the-fly.
  • Features added in 2.6.37:
    • Core Xen dom0 support (no backend drivers yet).
    • Xen PCI frontend driver required for Xen PCI Passthru to PV guest/domU.
    • Enhanced PV-on-HVM drivers: pirq remappings. Deliver IRQs as Xen event channels for better performance. Requires Xen 4.1 (or newer) hypervisor.
  • Features added in 2.6.38:
    • Generic Xen dom0 backend bits, required by all xen backend drivers.
    • xen-gntdev driver (grant device).
    • Bugfixes.
  • Features added in 2.6.39:
    • xen-netback backend driver to be used in dom0 to serve virtual networks to VMs. Currently unoptimized, optimizations will be added in later kernel versions.
    • Many dom0 related bugfixes and improvements.
    • PV-on-HVM driver fixes and improvements (xen balloon driver and PV spinlocks support for HVM guests).
    • xen-gntalloc driver for userspace grant allocation between Xen domains.
    • xen-gntdev support for HVM guests.
    • Xen watchdog driver.
    • 1-1 identity mapping in P2M. Allows us to automatically figure out if a page is for an I/O hole or not based on the E820. Fixes some device drivers.
    • IRQ code rework to support dynamic IRQs so that we're not limited to running 155 VMs.
    • Balloon driver has been prepared for memory hotplug and gntalloc.
    • save/restore bugfixes.
    • Dom0 startup crash fix when certain CONFIG_ options were set.
    • Bug fixes in xen-kbdfront, xen-netfront, xen-pcifront, and xen-blkfront.
    • Handling of guest events is now round-robin, fixes starvation issue of later guests not having their services served.
    • Many cleanups and bugfixes.
  • Features added in 3.0:
    • xen-blkback backend driver to be used in dom0 to serve virtual block devices (disks) to VMs.
    • Bugfixes and cleanups.
    • In 3.0.2: Enable use of vga text console in Xen dom0.
  • Features added in 3.1:
    • xen-pciback backend driver to be used in dom0 to support PCI passthru to VMs.
    • support for VGA text console in dom0.
    • memory hotplug support for xen balloon driver (allows adding more memory to the VM online / on-the-fly).
    • self-balloon driver to decrease memory in the guest and make the swap pages be shuffled by tmem to be compressed/shared/etc.
    • tmem driver to shuffle file-system and swap pages between guests as appropiate.
    • Xen PCI glue code cleanup.
    • Xen MMU debugfs tracing API support.
    • blkback providing completion latency that follows the hardware's completion latency.
  • Features added in 3.2:
    • blkback emulating the 'feature-barrier' option.
    • blkback providing TRIM/DISCARD operations ('feature-discard')
    • syncing wall clock time from Dom0 to hypervisor

I want to build the Xen/paravirt_ops components myself

Novice

If you are using a 3.0 kernel or later, you already have it working and there is no need to compile a new kernel - unless you feel adventurous.

Expert

The top level view is that you need to:

  1. Get a current kernel. The latest kernel.org kernel is generally a good choice. Or you can use the development version.
  2. Configure as normal; you can start with your current .config file or use make defconfig.
  3. Then configure .config Xen options
  4. Build the kernel
  5. Update #configure_modules modules (optional)
  6. Build the Xen hypervisor
  7. Update services (optional)
  8. Update GRUB1 or GRUB2
  9. Reboot and run it
  10. Troubleshoot if necessary

Getting the mainline Linux kernel

In order to get a proper Xen mainline kernel please check: Mainline_Linux_Kernel_Configs

Build Xen

Xen requirements for using pv_ops dom0 kernel

Xen hypervisor and tools need to have support for pv_ops dom0 kernels. In general it means:

  • The ability for the Xen hypervisor to load and boot bzImage pv_ops dom0 kernel.
  • The ability for the Xen tools to use the sysfs memory ballooning support provided by pv_ops dom0 kernel.
  • Current recommended 2.6.32.x version of pvops dom0 kernel requires new IOAPIC setup hypercall from Xen hypervisor.

This means you need to have at least Xen version 4.0.1.

Using older Xen versions is known to be problematic, for example Xen 4.0.0 libraries have problems with recent 2.6.32.x kernels, making xend fail to start due to evtchn/gntdev device node creation issues. Using Xen 3.4.2 or older won't work at all, since old hypervisor versions lack the new required IOAPIC setup hypercall and boot will fail with IRQ related issues.

It's recommended to run the latest Xen 4.0.x version, at least Xen 4.0.1.

Which kernel image to boot as dom0 kernel from your custom built kernel source tree?

If you have Xen hypervisor with bzImage dom0 kernel support, ie. xen 3.4 or later version, use "linux-2.6-xen/arch/x86/boot/bzImage" as your dom0 kernel (exactly the same kernel image you use for baremetal Linux).

If you have Xen hypervisor without bzImage dom0 kernel support, ie. any official Xen release up to at least Xen 3.3.1, or most of the Xen versions shipped with Linux distributions (before 2009-03), use "linux-2.6-xen/vmlinux" as your dom0 kernel. (Note that "vmlinux" is huge, so you can also gzip it, if you want to make it a bit smaller).

Also read the previous paragraphs for other requirements.

Get your sources from the repository, something like this can be used but the exact path will change: 


 $ cd /usr/src
 $ hg clone http://xenbits.xensource.com/xen-unstable.hg 
 $ cd xen-unstable.hg
 $ sudo make xen
 $ sudo make tools
 $ sudo make install-xen
 $ sudo make install-tools


If you encounter any error NOT related to deps report them to xen-devel, if you need help compiling you will find help in xen-users. At the end in dist folder you will find xen.gz follow that symlink to your real image.

Update services

The Ubuntu way to register a service is this: 


 $ sudo update-rc.d xend defaults 20 21
 $ sudo update-rc.d xendomains defaults 21 20

For Red Hat use chkconfig

For RC16 use systemctl: 

 $ sudo systemctl start xenstored.service


Other System Updates

The XENFS and XEN_COMPAT_XENFS config options are needed for /proc/xen support. If CONFIG_XEN_DEV_EVTCHN is compiled as a module, make sure to load the xen-evtchn.ko module or xend will not start.

You might also need to add a line to /etc/fstab. Xen 3.4.2 and newer automatically mount /proc/xen when /etc/init.d/xend is started, so no need to add xenfs mount entry to /etc/fstab on those systems: 


none /proc/xen xenfs defaults 0 0


GRUB1 and GRUB2 (Booting)

Working grub.conf example with VGA text console: 


title        Xen 4.0, dom0 Linux kernel 2.6.32.24
root         (hd0,0)
kernel       /boot/xen-4.0.gz dom0_mem=512M
module       /boot/vmlinuz-2.6.32.24 root=/dev/sda1 ro nomodeset
module       /boot/initrd.img-2.6.32.24


NOTE! You need to give correct root= parameter, ie. replace /dev/sda1 with your actual root device. Check your earlier grub kernel entries for the correct option. Also you need to have the "nomodeset" option for the time being. Check your earlier grub kernel entries for the correct option. Its good to look at your initial kernel boot configuration and use it as closely as possible, then modify the 40_custom file to suite your xen needs. Keep in mind that your initial kernel might not use the /dev/sdX it can use the UUID notation, well use it also.

Here is the content extracted from grub.cfg for a Ubuntu/Fedora Core kernel using grub2: 


menuentry 'GNU/Linux, with Linux 2.6.38-8-generic' --class gnu-linux --class gnu --class os {
    recordfail
    set gfxpayload=$linux_gfx_mode
    insmod part_msdos
    insmod ext2
    set root='(/dev/sda,msdos2)'
    search --no-floppy --fs-uuid --set=root 016e7c8a-4bdd-4873-92dd-d71171a49d6d
    linux    /boot/vmlinuz-2.6.38-8-generic root=UUID=016e7c8a-4bdd-4873-92dd-d71171a49d6d ro   quiet splash vt.handoff=7
    initrd    /boot/initrd.img-2.6.38-8-generic
}

Compare them and you will see the similarties, on this particular system the boot device is located in /dev/sda2 yet if used in grub the system will not boot. Insted GRUB uses the UUID schema to detect the devices, that way the root argument can be passed to the linux kernel image.

Here is the contents of 40_customCompare them and you will see the similarities, on this particular system the boot device is located in /dev/sda2 yet if used in grub the system will not boot instead GRUB searches for the correct UUID and its is used in the kernel image, this can be easily detected when looking at the original ubuntu boot entry..

Here is the contents of 40_custom: 


#! /sbin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'GNU/Linux, with Linux 2.6.32.40-pv' --class gnu-linux --class gnu --class os {
        recordfail
        insmod part_msdos
        insmod ext2
        search --no-floppy --fs-uuid --set=root 016e7c8a-4bdd-4873-92dd-d71171a49d6d
        set root='(/dev/sda,msdos2)'
        search --no-floppy --fs-uuid --set=root 016e7c8a-4bdd-4873-92dd-d71171a49d6d
        multiboot /boot/xen-4.2-unstable.gz
        module /boot/vmlinuz-2.6.32.40-pv placeholder root=UUID=016e7c8a-4bdd-4873-92dd-d71171a49d6d dom0_mem=1024 console=tty  quiet splash vt.handoff=7
        module /boot/initrd.img-2.6.32.40-pv
}

Xen MUST be started using multiboot this will point to the image created when xen was compiles, the next module line is the linux kernel built from the kernel repository, this can be either jeremy or kernel.org, after the image you append placeholder or dummy=dummy because grub will strip the first argument from the module line. The last line is the ram drive created after the kernel.

Here is another working example grub.conf with serial console output (good for debugging since you can easily log the full kernel boot messages even if it crashes): 


title        pv_ops dom0 (2.6.32.24) with serial console
root         (hd0,0)
kernel       /xen-4.0.gz dom0_mem=1024M loglvl=all guest_loglvl=all sync_console console_to_ring com1=19200,8n1 console=com1
module       /vmlinuz-2.6.32.24 ro root=/dev/vg00/lv01 console=hvc0 earlyprintk=xen nomodeset
module       /initrd-2.6.32.24.img

For more information about using a serial console with Xen please check the XenSerialConsole wiki page.

Booting DomU guests

If you've built a modular kernel, then all the modules will be the same either way. Some aspects of the kernel configuration have changed:

  • The console is now /dev/hvc0, so put "console=hvc0" on the kernel command line
  • Disk devices are always /dev/xvdX. If you want to dual-boot a system on both Xen and native, then it's best that use use lvm, LABEL or UUID to refer to your filesystems in your /etc/fstab.

Running

Troubleshooting, what to do if the custom built pv_ops dom0 kernel doesn't work/boot?

Make sure you read the ##troubleshooting section. It has tons of good suggestions. Also consult the [[[XenCommonProblems]]] page.

Historical contents

Timeline of pvops

Xen pv_ops (domU) support has been in mainline Linux since 2.6.23 (though it is probably first usable in 2.6.24), and is the basis of all on-going Linux/Xen development (the old Xenlinux patches officially ended with 2.6.18.x-xen, though various distros have their own forward-ports of them). Latest Linux kernels (2.6.27 and newer) are good for domU use. Starting from Fedora 9 all the new Fedora distribution versions include pv_ops based Xen domU kernel. Ubuntu 10.04 ("Lucid Lynx") and Debian 6.0 ("Squeeze") also includes Xen PV domU kernel. Red Hat Enterprise Linux 6.0 also includes pvops based Xen domU kernel.

Contributing

[This is obsolete, but it might be worth reading] Lots of work has been done to close the feature gaps compared to 2.6.18-xen. Many of these have been done - but there are still some left.

Devices

[Done] The Xen device model is more or less unchanged in the pv-ops kernel. Converting a driver from the xen-unstable or 2.6.18-xen tree should mostly be a matter of getting it to compile. There have been changes in the Linux device model between 2.6.18 and 2.6.26, so converting a driver will mostly be a matter of forward-porting to the new kernel, rather than any Xen specific issues.

CPU hotplug

[Done] All the mechanism should already be in place to support CPU hotplug; it should just be a matter of making it work. [2011-07-30]: It works with 3.0.

Device hotplug

[Done] In principle this is already implemented and should work. I'm not sure, however, that it's all plumbed through properly, so that hot-adding a device generates the appropriate udev events to cause devices to appear.

Device unplug/module unload

[Done] The 2.6.18-xen patches don't really support device unplug (and driver module unload), mainly because of the difficulties in dealing with granted pages. This should be fixed in the pvops kernel. The main thing to implement is to make sure that on driver termination, rather than freeing granted pages back into the kernel heap, they should be added to a list; that list is polled by a kernel thread which periodically tries to ungrant the pages and return them to the kernel heap if successful.

Development of PVOOPS in Ingo's tree

[This is obsolete, but it might be worth reading]

All x86 Xen/pv-ops changes queued for upstream Linus are in Ingo Molnar's tip.git tree. You can get general information about fetching and using this tree in his README. The x86/xen topic branch contains most of the Xen-specific work, though changes in other branches may be necessary too. Using the auto-latest branch is the merged product of all the other topic branches.

Testing

[This is obsolete, but it might be worth reading]

Xen/paravirt_ops has not had wide use or testing, so any testing you do is extremely valuable. If you have an existing Xen configuration, then updating the kernel to a current pv-ops and trying to use it as you usually would, then any feedback on how well that works (success or failure) would be very interesting. In particular, information about:

  • performance: better/worse/same?
  • bugs: outright crash, or something just not right?
  • missing features: what can't you live without?

Linux distribution support for pv_ops dom0 kernels

[This is obsolete, but it might be worth reading]

Fedora: Fedora 14 includes Xen 4.0.1 hypervisor and is able to run pvops dom0 kernel out-of-the-box. Fedora 14 does not ship with a dom0 capable kernel in the default distribution, but xendom0 kernel rpms are available from developer repositories. Fedora 13 and earlier versions ship with Xen 3.4.x and are not recommended for pvops dom0 usage, unless you update the Xen hypervisor to 4.0.x version. See this tutorial for more help: Fedora13Xen4Tutorial , and also check the Fedora dom0 wiki page: http://fedoraproject.org/wiki/Features/XenPvopsDom0 .

Debian: Debian 6.0 ("Squeeze") includes Xen 4.0.x hypervisor, and also dom0 capable kernel based on the pvops tree.

Other distributions: When using pvops dom0 kernel 2.6.32 or newer you need to have Xen hypervisor 4.0.1 or newer version.

2.6.32 and earlier kernels

[This is obsolete, but it might be worth reading] The kernel build process will build two kernel images: arch/x86/boot/bzImage and vmlinux. They are two forms of the same kernel, and are functionally identical. However, older versions of the Xen tools stack lack support loading bzImage files (pre-Xen 3.2), so you must use the vmlinux form of the kernel (gzipped, if you prefer).

Config files for 2.6.32

Example .config files:

64bit x86_64 (branch: xen/stable-2.6.32.x):

32bit PAE:

Those kernel configs are based on Fedora 11/12 kernel configuration, with some modifications. They've been tested to work on multiple systems. Note that these .config files have various debugging options enabled which will decrease performance so don't use these .config files for performance testing!

Retrieved from "https://wiki.xenproject.org/index.php?title=XenParavirtOps&oldid=2692"