Xen via libvirt for OpenStack juno

This document describes steps I take to setup a compute node based on Ubuntu14.04 for OpenStack juno, using the Xen via libvirt approach. Openstack does not support this approach well as it is in the group C of the hypervisor matrix support for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let's get started!

---

Prerequisite: Follow "OpenStack Installation Guide for Ubuntu 14.04" to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configurations and could take a day or two. Check that the control node and network node is working.

1. Add OpenStack juno to the repository.

 apt-get update
 apt-get install software-properties-common
 add-apt-repository cloud-archive:juno
 apt-get update

2. Install nova-compute-xen, sysfsutils and python nova client

 apt-get install nova-compute-xen sysfsutils python-novaclient

3. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in the installation guide to configure nova in a compute node.

• In the [default] section, add the following.
  

 [default]
 ...
 rpc_backend = rabbit
 rabbit_host = controller
 rabbit_password = RABBIT_PASS
 auth_strategy = keystone
 my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
 vnc_enabled = True
 vncserver_listen = 0.0.0.0
 vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
 novncproxy_base_url = http://controller:6080/vnc_auto.html
 verbose = True
 

MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface, typically 10.0.0.31 for the first compute node.

• Add keystone_authtoken section
  

 [keystone_authtoken]
 auth_uri = http://controller:5000/v2.0
 identity_uri = http://controller:35357
 admin_tenant_name = service
 admin_user = nova
 admin_password = NOVA_PASS

• Add glance section

 [glance]
 host=controller

4. Verify /etc/nova/nova-compute.conf content is as following.

 [DEFAULT]
 compute_driver=libvirt.LibvirtDriver
 [libvirt]
 virt_type=xen

5. Installing nova-compute-xen doesn't install xencommons script for us. However, we need xencommons to start the qemu process to provide block device backend service for qemu disks. I compile and install xen-4.4 from source, to get the xencommons script. Note this is likely not the preferred approach, and please update it if you know how.

• • Download xen-4.4 source code, compile and install.

    apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 \
                           module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended \
                           texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial make gcc libc6-dev \
                           zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev \
                           iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils \
                           libyajl-dev gettext libpixman-1-dev
    apt-get build-dep xen
    cd xen; 
    ./configure
    make dist -j16
    make install 

• • Enable startup services

   update-rc.d xencommons defaults 
   update-rc.d xen-watchdog defaults

• • Install virt-manager

  apt-get install virt-manager

If we do not install virt-manager, when you run "xl", it may say "xl: error while loading shared libraries: libxlutil.so.4.3: cannot open shared object file: No such file or directory"

6. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in dom0 kernel crashes when creating a Xen domU from OpenStack GUI (horizon).

    wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
    unzip master.zip
    cd qemu-2.0.2-master/
     ./configure
     make -j16
     make install

• • Update /etc/init.c/xencommons to use the new version of qemu, rather than the default one.

       --- /etc/init.d/xencommons.bk	2014-11-14 10:57:32.183529318 -0700
       +++ /etc/init.d/xencommons	2014-11-14 10:56:42.474535348 -0700
       @@ -118,6 +118,7 @@ do_start () {
	${SBINDIR}/xenconsoled --pid-file=$XENCONSOLED_PIDFILE $XENCONSOLED_ARGS
	echo Starting QEMU as disk backend for dom0
	test -z "$QEMU_XEN" && QEMU_XEN="${LIBEXEC}/qemu-system-i386"
   +	QEMU_XEN="/usr/local/bin/qemu-system-i386"
	$QEMU_XEN -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
		-monitor /dev/null -serial /dev/null -parallel /dev/null \
		-pidfile $QEMU_PIDFILE

7. Create a link at /usr/bin for pygrub

    # ln -s /usr/local/bin/pygrub /usr/bin/pygrub

8. Install and configure network component in compute node. Follow steps in this guide. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if Neutron and the nova security groups are used together. http://docs.openstack.org/juno/install-guide/install/apt/content/neutron-compute-node.html

9. Reboot the machine.