Xen via libvirt for OpenStack juno

This document describes steps I take to setup a compute node with Xen via libvirt based on Ubuntu14.04 for Openstack juno. Openstack does not support this approach well as Xen via libvirt is in the group C of the hypervisor matrix support for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. :-) Let's get started!

Prerequisite: Follow "OpenStack Installation Guide for Ubuntu 14.04" to setup the control node and network node. This involves lots of configurations and could take a day or two. Check that the control node and network node is working.

Setup a Xen via libvirt compute node for Ubuntu14.04.

1) Add OpenStack juno to the repository.

1. apt-get update
2. apt-get install software-properties-common
3. add-apt-repository cloud-archive:juno
4. apt-get update

2). Install nova-compute-xen and sysfsutils

1. apt-get install nova-compute-xen sysfsutils python-novaclient

3). Edit the /etc/nova/nova.conf and add all the following lines. You can also follow steps in the installation guide to configure nova in the compute node. 3.1) In the [default] section, add the following. [default]...rpc_backend = rabbit rabbit_host = controller rabbit_password = RABBIT_PASS auth_strategy = keystone my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS vnc_enabled = True vncserver_listen = 0.0.0.0 vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS novncproxy_base_url = http://controller:6080/vnc_auto.html verbose = True

        MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface, typically 10.0.0.31 for the first compute node.

3.2) Add keystone_authtoken section [keystone_authtoken] auth_uri = http://controller:5000/v2.0 identity_uri = http://controller:35357 admin_tenant_name = service admin_user = nova admin_password = NOVA_PASS 3.3) Add glance section [glance] host=controller 4. Verify /etc/nova/nova-compute.conf content is as following. [DEFAULT] compute_driver=libvirt.LibvirtDriver [libvirt] virt_type=xen 5. Installing nova-compute-xen doesn't install xencommons script for us. However, we need xencommons to start the qemu process to provide block device backend service for qemu disks. I compile and install xen-4.4 from source, to get the xencommons script. Note this is likely not the preferred approach, and please update it if you know how.

5.1 Download xen-4.4 source code, compile and install.

   # apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev gettext libpixman-1-dev
   # apt-get build-dep xen
   # cd xen; 
   # ./configure
   # make dist -j16
   # make install 

5.2 Enable startup services

   # update-rc.d xencommons defaults 
   # update-rc.d xen-watchdog defaults

5.3 Install virt-manager

  # apt-get install virt-manager

If we do not install virt-manager, when you run "xl", it may say "xl: error while loading shared libraries: libxlutil.so.4.3: cannot open shared object file: No such file or directory"

6. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in dom0 kernel crashes when creating a Xen domU from OpenStack GUI (horizon).

   # wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
   # unzip master.zip
   # cd qemu-2.0.2-master/
   #  ./configure
   #  make -j16
   #  make install

6.1 Update /etc/init.c/xencommons to use the new version of qemu, rather than the default one. --- /etc/init.d/xencommons.bk 2014-11-14 10:57:32.183529318 -0700 +++ /etc/init.d/xencommons 2014-11-14 10:56:42.474535348 -0700 @@ -118,6 +118,7 @@ do_start () {

	${SBINDIR}/xenconsoled --pid-file=$XENCONSOLED_PIDFILE $XENCONSOLED_ARGS
	echo Starting QEMU as disk backend for dom0
	test -z "$QEMU_XEN" && QEMU_XEN="${LIBEXEC}/qemu-system-i386"

+ QEMU_XEN="/usr/local/bin/qemu-system-i386"

	$QEMU_XEN -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
		-monitor /dev/null -serial /dev/null -parallel /dev/null \
		-pidfile $QEMU_PIDFILE

7. Create a link at /usr/bin for pygrub

    # ln -s /usr/local/bin/pygrub /usr/bin/pygrub

8. Install and configure network component in compute node. Follow steps in this guide. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if Neutron and the nova security groups are used together. http://docs.openstack.org/juno/install-guide/install/apt/content/neutron-compute-node.html

8. Reboot the machine.