Difference between revisions of "Xen via libvirt for OpenStack juno"

From Xen
 
(47 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This document describes steps I take to setup a compute node based on Ubuntu14.04 for OpenStack juno, using the Xen via libvirt approach. Openstack does not support this approach well as it is in the group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let's get started!
+
This document describes steps Xing Lin (University of Utah) took to setup a compute node based on Ubuntu 14.04 for OpenStack "juno", using the Xen Project via libvirt approach. As of the fall of 2014, Openstack does not support this approach well as Xen Project is in the Group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let's get started!
   
 
=Prerequisite=
 
=Prerequisite=
 
Follow [http://docs.openstack.org/juno/install-guide/install/apt/content/ "OpenStack Installation Guide for Ubuntu 14.04"] to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configurations and could take a day or two. Check that the control node and network node is working.
 
Follow [http://docs.openstack.org/juno/install-guide/install/apt/content/ "OpenStack Installation Guide for Ubuntu 14.04"] to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configurations and could take a day or two. Check that the control node and network node is working.
  +
 
=Steps=
 
=Steps=
  +
'''''NOTE:''' Steps 3, 4, and 5 are workarounds for bugs present in Ubuntu 14.04 (and probably in other Debian derivatives of that era). Future releases of Ubuntu may not require these workarounds.''
1. Add OpenStack juno to the repository.
 
apt-get update
 
apt-get install software-properties-common
 
add-apt-repository cloud-archive:juno
 
apt-get update
 
2. Install nova-compute-xen, sysfsutils and python nova client
 
apt-get install nova-compute-xen sysfsutils python-novaclient
 
3. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in [http://docs.openstack.org/juno/install-guide/install/apt/content/ch_nova.html#nova-compute-install the installation guide] to configure nova in a compute node.
 
* In the [default] section, add the following. <br/>
 
[default]
 
...
 
rpc_backend = rabbit
 
rabbit_host = controller
 
rabbit_password = RABBIT_PASS
 
auth_strategy = keystone
 
my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
 
vnc_enabled = True
 
vncserver_listen = 0.0.0.0
 
vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
 
novncproxy_base_url = http://controller:6080/vnc_auto.html
 
verbose = True
 
MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface for this compute node, typically 10.0.0.31 for the first compute node.
 
   
  +
1. Add OpenStack "juno" to the repository:
* Add keystone_authtoken section <br />
 
  +
apt-get update
[keystone_authtoken]
 
  +
apt-get install software-properties-common
auth_uri = http://controller:5000/v2.0
 
  +
add-apt-repository cloud-archive:juno
identity_uri = http://controller:35357
 
  +
apt-get update
admin_tenant_name = service
 
  +
2. Install nova-compute-xen, sysfsutils and python nova client:
admin_user = nova
 
  +
apt-get install nova-compute-xen sysfsutils python-novaclient
admin_password = NOVA_PASS
 
  +
3. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in Dom0 kernel crashes when creating a Xen Project DomU from OpenStack GUI (horizon). I have applied [http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg01112.html this patch] and make the modified qemu available in github:
* Add glance section
 
  +
wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
[glance]
 
  +
unzip master.zip
host=controller
 
  +
cd qemu-2.0.2-master/
4. Verify the conent of /etc/nova/nova-compute.conf is as following.
 
  +
apt-get build-dep qemu
[DEFAULT]
 
  +
./configure
compute_driver=libvirt.LibvirtDriver
 
  +
make -j16
[libvirt]
 
  +
make install
virt_type=xen
 
5. Installing nova-compute-xen doesn't install xencommons script for us. However, we need xencommons to start the qemu process to provide block device backend service for qemu disks. I compile and install xen-4.4 from source, to get the xencommons script. Note this is likely not the preferred approach, and please update it if you know a better approach.
 
   
  +
4. Add a patch to /etc/init.d/xen to start qemu process during startup:
* Download xen-4.4 source code, compile and install.
 
  +
--- /etc/init.d/xen 2014-11-18 20:54:10.788457049 -0700
apt-get install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 \
 
  +
+++ /etc/init.d/xen.bak 2014-11-18 20:53:14.804107463 -0700
module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended \
 
  +
@@ -228,6 +228,9 @@ case "$1" in
texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial make gcc libc6-dev \
 
  +
*) log_end_msg 1; exit ;;
zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev \
 
  +
esac
iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils \
 
  +
log_end_msg 0
libyajl-dev gettext libpixman-1-dev
 
  +
+ /usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
apt-get build-dep xen
 
  +
+ -monitor /dev/null -serial /dev/null -parallel /dev/null \
cd xen
 
  +
+ -pidfile /var/run/qemu-xen-dom0.pid
./configure
 
  +
;;
make dist -j16
 
  +
stop)
make install
 
  +
capability_check
  +
5. Create a link at /usr/bin for pygrub:
  +
ln -s /usr/lib/xen-4.4/bin/pygrub /usr/bin/pygrub
  +
6. Reboot the machine and boot into Xen Project Dom0.
   
  +
7. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in [http://docs.openstack.org/juno/install-guide/install/apt/content/ch_nova.html#nova-compute-install the OpenStack installation guide] to configure nova as a compute node.
* Enable startup services
 
  +
* In the default section, add the following: <br/>
update-rc.d xencommons defaults
 
  +
<pre>
update-rc.d xen-watchdog defaults
 
  +
[default]
  +
...
  +
rpc_backend = rabbit
  +
rabbit_host = controller
  +
rabbit_password = RABBIT_PASS
  +
auth_strategy = keystone
  +
my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
  +
vnc_enabled = True
  +
vncserver_listen = 0.0.0.0
  +
vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
  +
novncproxy_base_url = http://controller:6080/vnc_auto.html
  +
verbose = True
  +
</pre>
  +
''MANAGEMENT_INTERFACE_IP_ADDRESS'' is the IP address of the management network interface for this compute node, typically 10.0.0.31 for the first compute node.
   
  +
* Add keystone_authtoken section:<br />
* Install virt-manager
 
  +
<pre>
apt-get install virt-manager
 
  +
[keystone_authtoken]
If we do not install virt-manager, when you run "xl", it may say "xl: error while loading shared libraries: libxlutil.so.4.3: cannot open shared object file: No such file or directory"
 
  +
auth_uri = http://controller:5000/v2.0
  +
identity_uri = http://controller:35357
  +
admin_tenant_name = service
  +
admin_user = nova
  +
admin_password = NOVA_PASS
  +
</pre>
  +
* Add glance section:
  +
[glance]
  +
host=controller
  +
8. Verify the conent of /etc/nova/nova-compute.conf is as following:
  +
[DEFAULT]
  +
compute_driver=libvirt.LibvirtDriver
  +
[libvirt]
  +
virt_type=xen
  +
9. Install and configure network component in compute node:
  +
Follow the steps outlined in this [http://docs.openstack.org/juno/install-guide/install/apt/content/neutron-compute-node.html OpenStack install guide for neutron compute nodes]. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if both Neutron and the nova security groups are used together.
   
  +
10. Finally step: now you should be able to launch an instance from horizon. In my case, I launched an instance running cirros-0.3.3-x86_64. When I login the compute node, I can see this instance running with virsh:
6. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in dom0 kernel crashes when creating a Xen domU from OpenStack GUI (horizon). I have applied [http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg01112.html this patch] and make the modified qemu available in github.
 
  +
# virsh --connect=xen:///
wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
 
  +
Welcome to virsh, the virtualization interactive terminal.
unzip master.zip
 
  +
cd qemu-2.0.2-master/
 
  +
Type: 'help' for help with commands
./configure
 
  +
'quit' to quit
make -j16
 
  +
make install
 
  +
virsh # list
  +
Id Name State
  +
----------------------------------------------------
  +
1 instance-0000003b running
   
  +
=Reference=
* Update /etc/init.c/xencommons to use the new version of qemu, rather than the default one.
 
  +
* [http://en.wikibooks.org/wiki/QEMU/Linux Compile and Install QEMU]<br />
--- /etc/init.d/xencommons.bk 2014-11-14 10:57:32.183529318 -0700
 
  +
* [http://wiki.sebeka.k12.mn.us/virt:xen_4.4_14.04 Compile and Install Xen Project 4.4 in Ubuntu 14.04]
+++ /etc/init.d/xencommons 2014-11-14 10:56:42.474535348 -0700
 
  +
* Bugs which needed to be worked around in steps 3-5 have been filed with Debian. See the [http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg02088.html email describing this] with pointers to the bug reports. The bug in number 5 has already had a patch submitted in libvirt.org, but it will take time before it is generally available. Check the bugs to see if they are still needed in future releases.
@@ -118,6 +118,7 @@ do_start () {
 
  +
* The source blog post for this page: [https://blog.xenproject.org/2014/12/01/using-xen-project-on-openstack-juno-via-libvirt/ Using Xen Project on OpenStack “Juno” via Libvirt]
${SBINDIR}/xenconsoled --pid-file=$XENCONSOLED_PIDFILE $XENCONSOLED_ARGS
 
echo Starting QEMU as disk backend for dom0
 
test -z "$QEMU_XEN" && QEMU_XEN="${LIBEXEC}/qemu-system-i386"
 
+QEMU_XEN="/usr/local/bin/qemu-system-i386"
 
$QEMU_XEN -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
 
-monitor /dev/null -serial /dev/null -parallel /dev/null \
 
-pidfile $QEMU_PIDFILE
 
   
  +
[[Category:Users]]
7. Create a link at /usr/bin for pygrub
 
  +
[[Category:OpenStack]]
ln -s /usr/local/bin/pygrub /usr/bin/pygrub
 
  +
[[Category:Libvirt]]
 
  +
[[Category:HowTo]]
8. Install and configure network component in compute node.
 
  +
[[Category:Integration]]
Follow steps in this [http://docs.openstack.org/juno/install-guide/install/apt/content/neutron-compute-node.html guide]. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if Neutron and the nova security groups are used together.
 
 
9. Reboot the machine.
 
 
=Reference=
 
[http://en.wikibooks.org/wiki/QEMU/Linux Compile and Install QEMU]<br />
 
[http://wiki.sebeka.k12.mn.us/virt:xen_4.4_14.04 Compile and Install Xen 4.4 in Ubuntu14.04]
 

Latest revision as of 01:18, 4 February 2015

This document describes steps Xing Lin (University of Utah) took to setup a compute node based on Ubuntu 14.04 for OpenStack "juno", using the Xen Project via libvirt approach. As of the fall of 2014, Openstack does not support this approach well as Xen Project is in the Group C of the hypervisor support matrix for Openstack. You can hardly find any tutorial online describing this approach and this might be the first. Let's get started!

Prerequisite

Follow "OpenStack Installation Guide for Ubuntu 14.04" to setup the control node and network node, following the three-node architecture with OpenStack Networking (neutron). This involves lots of configurations and could take a day or two. Check that the control node and network node is working.

Steps

NOTE: Steps 3, 4, and 5 are workarounds for bugs present in Ubuntu 14.04 (and probably in other Debian derivatives of that era). Future releases of Ubuntu may not require these workarounds.

1. Add OpenStack "juno" to the repository:

apt-get update
apt-get install software-properties-common
add-apt-repository cloud-archive:juno
apt-get update

2. Install nova-compute-xen, sysfsutils and python nova client:

apt-get install nova-compute-xen sysfsutils python-novaclient

3. Install qemu-2.0.2 with a patch fixing unmapping of persistent grants. Current qemu releases (including 2.0.2, 2.1.2 and 2.2.0-rc1) do not have this patch included and this will result in Dom0 kernel crashes when creating a Xen Project DomU from OpenStack GUI (horizon). I have applied this patch and make the modified qemu available in github:

wget https://github.com/xinglin/qemu-2.0.2/archive/master.zip
unzip master.zip
cd qemu-2.0.2-master/
apt-get build-dep qemu
./configure
make -j16
make install

4. Add a patch to /etc/init.d/xen to start qemu process during startup:

--- /etc/init.d/xen	2014-11-18 20:54:10.788457049 -0700
+++ /etc/init.d/xen.bak	2014-11-18 20:53:14.804107463 -0700
@@ -228,6 +228,9 @@ case "$1" in
		*) log_end_msg 1; exit ;;
	esac
	log_end_msg 0
+	/usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
+	    -monitor /dev/null -serial /dev/null -parallel /dev/null \
+	    -pidfile /var/run/qemu-xen-dom0.pid
	;;
  stop)
	capability_check

5. Create a link at /usr/bin for pygrub:

 ln -s /usr/lib/xen-4.4/bin/pygrub /usr/bin/pygrub

6. Reboot the machine and boot into Xen Project Dom0.

7. Edit the /etc/nova/nova.conf, to configure nova service. You can also follow steps in the OpenStack installation guide to configure nova as a compute node.

  • In the default section, add the following:
 [default]
 ...
 rpc_backend = rabbit
 rabbit_host = controller
 rabbit_password = RABBIT_PASS
 auth_strategy = keystone
 my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
 vnc_enabled = True
 vncserver_listen = 0.0.0.0
 vncserver_proxyclient_address = MANAGEMENT_INTERFACE_IP_ADDRESS
 novncproxy_base_url = http://controller:6080/vnc_auto.html
 verbose = True

MANAGEMENT_INTERFACE_IP_ADDRESS is the IP address of the management network interface for this compute node, typically 10.0.0.31 for the first compute node.

  • Add keystone_authtoken section:
 [keystone_authtoken]
 auth_uri = http://controller:5000/v2.0
 identity_uri = http://controller:35357
 admin_tenant_name = service
 admin_user = nova
 admin_password = NOVA_PASS
  • Add glance section:
[glance]
host=controller

8. Verify the conent of /etc/nova/nova-compute.conf is as following:

[DEFAULT]
compute_driver=libvirt.LibvirtDriver
[libvirt]
virt_type=xen

9. Install and configure network component in compute node: Follow the steps outlined in this OpenStack install guide for neutron compute nodes. Note, in /etc/neutron/neutron.conf, I did not set "allow_overlapping_ips = True" in the default section, because it is said to set this property to False if both Neutron and the nova security groups are used together.

10. Finally step: now you should be able to launch an instance from horizon. In my case, I launched an instance running cirros-0.3.3-x86_64. When I login the compute node, I can see this instance running with virsh:

# virsh --connect=xen:///
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
Id    Name                           State
----------------------------------------------------
1     instance-0000003b              running

Reference