Difference between revisions of "RHEL6 Xen4 Tutorial"

From Xen
(Migrated from wiki)
 
m (Updating the dom0 kernel)
 
(13 intermediate revisions by 10 users not shown)
Line 5: Line 5:
 
<!-- Original date: Tue Aug 30 06:31:20 2011 (1314685880000000) -->
 
<!-- Original date: Tue Aug 30 06:31:20 2011 (1314685880000000) -->
   
  +
__NOTOC__
 
  +
== Binary Xen rpm repositories for EL6 ==
= Redhat Enterprise Linux 6 Xen 4.0 tutorial =
 
  +
Current repositories:
This tutorial explains how to install and configure RHEL6 to run as Xen dom0 (host), running Xen hypervisor 4.0. Note that this is unsupported by Redhat! See the end of this wiki page for more information about RHEL6 Xen binary rpm yum repository. This tutorial explains how to compile Xen and dom0 kernel from sources/srpms.
 
  +
* Xen Made Easy - http://xen.crc.id.au (Install Guide - http://xen.crc.id.au/support/guides/install/) ''2015-Jan-24: Xen Made Easy worked under RHEL 6.6. The steps below are more complex, but give a more complete view of what to do. I did not complete the steps below, I switched over to Xen Made Easy. However, I did update some URLs below.''
  +
  +
Older repositories (mostly dead):
  +
* Michael Young's EL6 Xen rpm repository (also includes xen dom0 capable kernel based on rhel6 2.6.32 kernel):
  +
* http://xenbits.xenproject.org/people/mayoung/EL6.xen/
  +
* http://xenbits.xenproject.org/people/mayoung/testing/
  +
* Gitco.de repository is based on this tutorial, but contains xen dom0 capable kernel rpm based on the xen/kernel.org kernel (not rhel6 kernel):
  +
* http://www.gitco.de/linux/x86_64/centos/6/
  +
* http://www.gitco.de/linux/x86_64/centos/6/gitco-centos6-x86_64.repo
  +
  +
= Red Hat Enterprise Linux 6 Xen 4.0 tutorial =
  +
This tutorial explains how to install and configure RHEL6 to run as Xen dom0 (host), running Xen hypervisor 4.0. Note that this is unsupported by Red Hat! See the end of this wiki page for more information about RHEL6 Xen binary rpm yum repository. This tutorial explains how to compile Xen and dom0 kernel from sources/srpms.
   
 
As a default RHEL6 does NOT:
 
As a default RHEL6 does NOT:
Line 37: Line 49:
 
== RHEL6 installation and configuration ==
 
== RHEL6 installation and configuration ==
 
=== Installation: ===
 
=== Installation: ===
* Download "rhel-server-6.0-x86_64-dvd.iso" from Redhat Network (RHN).
+
* Download "rhel-server-6.0-x86_64-dvd.iso" from Red Hat Network (RHN).
 
* Burn it to DVDR, or if you have a server with remote management, use virtual media ISO mapping.
 
* Burn it to DVDR, or if you have a server with remote management, use virtual media ISO mapping.
 
* Boot from the DVD.
 
* Boot from the DVD.
Line 51: Line 63:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
 
[root@el6 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
 
DEVICE="eth0"
 
DEVICE="eth0"
Line 58: Line 70:
 
ONBOOT="yes"
 
ONBOOT="yes"
 
BOOTPROTO="dhcp"
 
BOOTPROTO="dhcp"
</nowiki></pre>
+
</pre>
   
 
ie. make sure NM_CONTROLLED="no" and ONBOOT="yes". Then run "ifup eth0" to activate networking.
 
ie. make sure NM_CONTROLLED="no" and ONBOOT="yes". Then run "ifup eth0" to activate networking.
Line 65: Line 77:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# chkconfig --list network
 
[root@el6 ~]# chkconfig --list network
</nowiki></pre>
+
</pre>
   
 
Make sure your Internet connection works OK at this point! Next make sure "/etc/hosts" file has an entry for your hostname:
 
Make sure your Internet connection works OK at this point! Next make sure "/etc/hosts" file has an entry for your hostname:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# cat /etc/hosts
 
[root@el6 ~]# cat /etc/hosts
 
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
 
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
 
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
 
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
 
127.0.0.1 el6 el6.localdomain
 
127.0.0.1 el6 el6.localdomain
</nowiki></pre>
+
</pre>
   
 
Full FQDN hostname in this example is "el6.localdomain".
 
Full FQDN hostname in this example is "el6.localdomain".
Line 84: Line 96:
   
   
<pre><nowiki>
+
<pre>
 
SELINUX=disabled
 
SELINUX=disabled
</nowiki></pre>
+
</pre>
   
 
Then it's time to register your system to RHN, so you can fetch updates and install additional packages using yum.
 
Then it's time to register your system to RHN, so you can fetch updates and install additional packages using yum.
Line 104: Line 116:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# yum update
 
[root@el6 ~]# yum update
</nowiki></pre>
+
</pre>
   
 
Then install some commonly used tools:
 
Then install some commonly used tools:
   
   
<pre><nowiki>
+
<pre>
[root@el6 ~]# yum install screen vim wget tcpdump ntp ntpdate man smartmontools links lynx ethtool xorg-x11-xauth</nowiki></pre>
+
[root@el6 ~]# yum install screen vim wget tcpdump ntp ntpdate man smartmontools links lynx ethtool xorg-x11-xauth
  +
</pre>
   
 
Then increase grub timeout from grub.conf, and comment out "hiddenmenu" option:
 
Then increase grub timeout from grub.conf, and comment out "hiddenmenu" option:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# cat /boot/grub/grub.conf
 
[root@el6 ~]# cat /boot/grub/grub.conf
 
# grub.conf generated by anaconda
 
# grub.conf generated by anaconda
Line 140: Line 153:
 
kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot rd_LVM_LV=vg00/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi crashkernel=auto rhgb quiet
 
kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot rd_LVM_LV=vg00/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi crashkernel=auto rhgb quiet
 
initrd /initramfs-2.6.32-71.el6.x86_64.img
 
initrd /initramfs-2.6.32-71.el6.x86_64.img
</nowiki></pre>
+
</pre>
   
 
At this point you should reboot the system to the newest kernel.
 
At this point you should reboot the system to the newest kernel.
Line 148: Line 161:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# yum groupinstall "Development tools" "Additional Development" "Debugging Tools" "System administration tools" "Compatibility libraries" "Console internet tools" "Desktop Platform Development"
 
[root@el6 ~]# yum groupinstall "Development tools" "Additional Development" "Debugging Tools" "System administration tools" "Compatibility libraries" "Console internet tools" "Desktop Platform Development"
</nowiki></pre>
+
</pre>
   
 
This is around 212 MB of downloads.
 
This is around 212 MB of downloads.
Line 157: Line 170:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# yum install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo libuuid-devel
 
[root@el6 ~]# yum install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo libuuid-devel
</nowiki></pre>
+
</pre>
   
 
This is around 82 MB of downloads.
 
This is around 82 MB of downloads.
Line 166: Line 179:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# yum install glibc-devel.i686
 
[root@el6 ~]# yum install glibc-devel.i686
</nowiki></pre>
+
</pre>
   
 
Now you have all the basic tools, libraries and headers installed.
 
Now you have all the basic tools, libraries and headers installed.
Line 175: Line 188:
 
Fedora xen-4.0.1-6 rpm package includes some additional patches and fixes backported from upstream Xen 4.0.2-rc*.
 
Fedora xen-4.0.1-6 rpm package includes some additional patches and fixes backported from upstream Xen 4.0.2-rc*.
   
  +
('''FIXME:''' These instructions should be updated to a more recent version of Xen.)
   
  +
('''FIXME:''' These instructions should be updated to support RHEL Server 7 and discuss how to select the appropriate version of Fedora. One possibility is to look at the https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux for the version of Fedora)
<pre><nowiki>
 
  +
  +
<pre>
 
[root@el6 ~]# mkdir /root/src && cd /root/src
 
[root@el6 ~]# mkdir /root/src && cd /root/src
 
[root@el6 src]# wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/releases/14/Everything/source/SRPMS/xen-4.0.1-6.fc14.src.rpm
 
[root@el6 src]# wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/releases/14/Everything/source/SRPMS/xen-4.0.1-6.fc14.src.rpm
Line 182: Line 198:
 
[root@el6 src]# cd /root/rpmbuild/SPECS
 
[root@el6 src]# cd /root/rpmbuild/SPECS
 
[root@el6 SPECS]# rpmbuild -bb xen.spec
 
[root@el6 SPECS]# rpmbuild -bb xen.spec
</nowiki></pre>
+
</pre>
   
 
After a while you should see output like:
 
After a while you should see output like:
   
   
<pre><nowiki>
+
<pre>
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-4.0.1-6.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-4.0.1-6.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-libs-4.0.1-6.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-libs-4.0.1-6.el6.x86_64.rpm
Line 196: Line 212:
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-licenses-4.0.1-6.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-licenses-4.0.1-6.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-debuginfo-4.0.1-6.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/xen-debuginfo-4.0.1-6.el6.x86_64.rpm
</nowiki></pre>
+
</pre>
   
 
Try installing the rpms:
 
Try installing the rpms:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 SPECS]# cd /root/rpmbuild/RPMS/x86_64/
 
[root@el6 SPECS]# cd /root/rpmbuild/RPMS/x86_64/
 
[root@el6 x86_64]# rpm -Uvh xen*4.0.1-6*.rpm
 
[root@el6 x86_64]# rpm -Uvh xen*4.0.1-6*.rpm
 
error: Failed dependencies:
 
error: Failed dependencies:
 
/usr/bin/qemu-nbd is needed by xen-runtime-4.0.1-6.el6.x86_64
 
/usr/bin/qemu-nbd is needed by xen-runtime-4.0.1-6.el6.x86_64
</nowiki></pre>
+
</pre>
   
 
So what's happening here? It seems RHEL6 does not have "qemu-common" rpm available, which includes "qemu-nbd" binary. So we need to grab qemu src.rpm from Fedora and rebuild it for RHEL6 to get the required "qemu-common" rpm.
 
So what's happening here? It seems RHEL6 does not have "qemu-common" rpm available, which includes "qemu-nbd" binary. So we need to grab qemu src.rpm from Fedora and rebuild it for RHEL6 to get the required "qemu-common" rpm.
Line 213: Line 229:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 x86_64]# rpm -qa | grep qemu
 
[root@el6 x86_64]# rpm -qa | grep qemu
 
qemu-img-0.12.1.2-2.113.el6.x86_64
 
qemu-img-0.12.1.2-2.113.el6.x86_64
</nowiki></pre>
+
</pre>
   
 
Ok, so it's version 0.12.x. Fedora 13 has qemu 0.12.5, so it's close enough:
 
Ok, so it's version 0.12.x. Fedora 13 has qemu 0.12.5, so it's close enough:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 x86_64]# cd /root/src
 
[root@el6 x86_64]# cd /root/src
 
[root@el6 src]# wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/updates/13/SRPMS/qemu-0.12.5-1.fc13.src.rpm
 
[root@el6 src]# wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/updates/13/SRPMS/qemu-0.12.5-1.fc13.src.rpm
Line 227: Line 243:
 
[root@el6 src]# cd /root/rpmbuild/SPECS
 
[root@el6 src]# cd /root/rpmbuild/SPECS
 
[root@el6 SPECS]# rpmbuild -bb qemu.spec
 
[root@el6 SPECS]# rpmbuild -bb qemu.spec
</nowiki></pre>
+
</pre>
  +
  +
('''FIXME:''' 2015-01-24: The above wget did not, I tried wget http://archives.fedoraproject.org/pub/archive/fedora/linux/updates/13/SRPMS/qemu-0.13.0-1.fc13.src.rpm )
   
 
After a while you'll see:
 
After a while you'll see:
   
   
<pre><nowiki>
+
<pre>
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-0.12.5-1.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-0.12.5-1.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-kvm-0.12.5-1.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-kvm-0.12.5-1.el6.x86_64.rpm
Line 248: Line 266:
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-kvm-tools-0.12.5-1.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-kvm-tools-0.12.5-1.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-debuginfo-0.12.5-1.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-debuginfo-0.12.5-1.el6.x86_64.rpm
</nowiki></pre>
+
</pre>
   
 
So now we have the required "qemu-common" rpm. Let's install it to satisfy the package dependencies:
 
So now we have the required "qemu-common" rpm. Let's install it to satisfy the package dependencies:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 SPECS]# cd /root/rpmbuild/RPMS/x86_64
 
[root@el6 SPECS]# cd /root/rpmbuild/RPMS/x86_64
 
[root@el6 x86_64]# rpm -Uvh qemu-common-0.12.5-1.el6.x86_64.rpm
 
[root@el6 x86_64]# rpm -Uvh qemu-common-0.12.5-1.el6.x86_64.rpm
 
Preparing... ########################################### [100%]
 
Preparing... ########################################### [100%]
 
1:qemu-common ########################################### [100%]
 
1:qemu-common ########################################### [100%]
</nowiki></pre>
+
</pre>
   
 
Now we can also install Xen rpms:
 
Now we can also install Xen rpms:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 x86_64]# rpm -Uvh xen*4.0.1-6*.rpm
 
[root@el6 x86_64]# rpm -Uvh xen*4.0.1-6*.rpm
 
Preparing... ########################################### [100%]
 
Preparing... ########################################### [100%]
Line 274: Line 292:
 
7:xen-doc ########################################### [ 88%]
 
7:xen-doc ########################################### [ 88%]
 
8:xen-debuginfo ########################################### [100%]
 
8:xen-debuginfo ########################################### [100%]
</nowiki></pre>
+
</pre>
   
 
Now you're done installing Xen hypervisor, tools and libraries.
 
Now you're done installing Xen hypervisor, tools and libraries.
   
 
== Installing upstream Xen dom0 capable kernel ==
 
== Installing upstream Xen dom0 capable kernel ==
Note that we're installing upstream *Xen* kernel here, which is based on the long-term maintained kernel.org Linux 2.6.32.x kernel. This kernel is different from Redhat RHEL6 kernel. Many drivers are newer versions in Redhat's kernel, and Redhat's kernel might have more features than the kernel.org kernel. Due to the driver version differences you might experience problems on some hardware, for example on servers using LSI/DELL RAID adapters. You might need to manually update driver versions to fix these problems on affected systems. See the end of this wiki page for more information about kernel versions.
+
Note that we're installing upstream *Xen* kernel here, which is based on the long-term maintained kernel.org Linux 2.6.32.x kernel. This kernel is different from the RHEL6 kernel. Many drivers are newer versions in Red Hat's kernel, and Red Hat's kernel might have more features than the kernel.org kernel. Due to the driver version differences you might experience problems on some hardware, for example on servers using LSI/DELL RAID adapters. You might need to manually update driver versions to fix these problems on affected systems. See the end of this wiki page for more information about kernel versions.
   
 
Clone the kernel git repository from kernel.org and check out the "xen/stable-2.6.32.x" long-term maintained branch.
 
Clone the kernel git repository from kernel.org and check out the "xen/stable-2.6.32.x" long-term maintained branch.
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 x86_64]# mkdir /root/kernel && cd /root/kernel
 
[root@el6 x86_64]# mkdir /root/kernel && cd /root/kernel
 
[root@el6 kernel]# git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
 
[root@el6 kernel]# git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
 
[root@el6 kernel]# cd linux-2.6-xen
 
[root@el6 kernel]# cd linux-2.6-xen
 
[root@el6 linux-2.6-xen]# git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
 
[root@el6 linux-2.6-xen]# git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
</nowiki></pre>
+
</pre>
  +
  +
('''FIXME''': The git command failed, the message was "fatal: git checkout: updating paths is incompatible with switching branches.". http://lists.xen.org/archives/html/xen-devel/2012-01/msg02597.htm suggests "git checkout -b xen/next-2.6.32 origin/xen/next-2.6.32", which worked. However, I did not complete the tutorial, I switched over to Xen Made Easy, see above.)
  +
   
 
Now you have the kernel ready to be configured and compiled. Reference config-file used is for version "2.6.32.25". It should work with newer version aswell, "make oldconfig" takes care of that.
 
Now you have the kernel ready to be configured and compiled. Reference config-file used is for version "2.6.32.25". It should work with newer version aswell, "make oldconfig" takes care of that.
Line 295: Line 316:
 
NOTE! The reference .config file has various debugging options enabled, so don't use this kernel config for performance benchmarks!
 
NOTE! The reference .config file has various debugging options enabled, so don't use this kernel config for performance benchmarks!
   
  +
<pre>
 
<pre><nowiki>
 
 
[root@el6 linux-2.6-xen]# wget -O .config http://pasik.reaktio.net/xen/kernel-config/config-2.6.32.25-pvops-dom0-xen-stable-x86_64
 
[root@el6 linux-2.6-xen]# wget -O .config http://pasik.reaktio.net/xen/kernel-config/config-2.6.32.25-pvops-dom0-xen-stable-x86_64
 
[root@el6 linux-2.6-xen]# make oldconfig
 
[root@el6 linux-2.6-xen]# make oldconfig
 
[root@el6 linux-2.6-xen]# make -j4 bzImage && make -j4 modules && make modules_install
 
[root@el6 linux-2.6-xen]# make -j4 bzImage && make -j4 modules && make modules_install
</nowiki></pre>
+
</pre>
   
 
Then let's install the kernel and generate initrd (initramfs) image for the new kernel. Kernel version in this example is "2.6.32.25":
 
Then let's install the kernel and generate initrd (initramfs) image for the new kernel. Kernel version in this example is "2.6.32.25":
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 linux-2.6-xen]# depmod -a 2.6.32.25
 
[root@el6 linux-2.6-xen]# depmod -a 2.6.32.25
 
[root@el6 linux-2.6-xen]# cp -a arch/x86/boot/bzImage /boot/vmlinuz-2.6.32.25
 
[root@el6 linux-2.6-xen]# cp -a arch/x86/boot/bzImage /boot/vmlinuz-2.6.32.25
Line 312: Line 332:
 
[root@el6 linux-2.6-xen]# cd /boot
 
[root@el6 linux-2.6-xen]# cd /boot
 
[root@el6 boot]# dracut initramfs-2.6.32.25.img 2.6.32.25
 
[root@el6 boot]# dracut initramfs-2.6.32.25.img 2.6.32.25
</nowiki></pre>
+
</pre>
   
 
NOTE! If you're using a system with LSI and/or Dell RAID adapters (using megaraid_sas driver) you need to manually update the driver version at this point! Linux 2.6.32.25 ships with too old driver version, and it wont' work (disks won't get enabled on system startup). See this email for information about updating the driver version: http://lists.xensource.com/archives/html/xen-devel/2010-11/msg00250.html .
 
NOTE! If you're using a system with LSI and/or Dell RAID adapters (using megaraid_sas driver) you need to manually update the driver version at this point! Linux 2.6.32.25 ships with too old driver version, and it wont' work (disks won't get enabled on system startup). See this email for information about updating the driver version: http://lists.xensource.com/archives/html/xen-devel/2010-11/msg00250.html .
Line 319: Line 339:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# cat /boot/grub/grub.conf
 
[root@el6 ~]# cat /boot/grub/grub.conf
 
# grub.conf generated by anaconda
 
# grub.conf generated by anaconda
Line 347: Line 367:
 
module /vmlinuz-2.6.32.25 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot nomodeset
 
module /vmlinuz-2.6.32.25 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot nomodeset
 
module /initramfs-2.6.32.25.img
 
module /initramfs-2.6.32.25.img
</nowiki></pre>
+
</pre>
   
 
Then let's disable KSM services, those don't work with Xen currently:
 
Then let's disable KSM services, those don't work with Xen currently:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# chkconfig ksm off
 
[root@el6 ~]# chkconfig ksm off
 
[root@el6 ~]# chkconfig ksmtuned off
 
[root@el6 ~]# chkconfig ksmtuned off
</nowiki></pre>
+
</pre>
   
 
And then it's time to reboot into Xen ! Remember to choose the Xen entry from grub:
 
And then it's time to reboot into Xen ! Remember to choose the Xen entry from grub:
   
   
<pre><nowiki>
+
<pre>
[root@el6 ~]# reboot</nowiki></pre>
+
[root@el6 ~]# reboot
  +
</pre>
   
 
== Verifying Xen after reboot ==
 
== Verifying Xen after reboot ==
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# xm list
 
[root@el6 ~]# xm list
 
Name ID Mem VCPUs State Time(s)
 
Name ID Mem VCPUs State Time(s)
Line 406: Line 427:
 
Linux el6.localdomain 2.6.32.25 #1 SMP Sun Nov 14 21:50:39 EET 2010 x86_64 x86_64 x86_64 GNU/Linux
 
Linux el6.localdomain 2.6.32.25 #1 SMP Sun Nov 14 21:50:39 EET 2010 x86_64 x86_64 x86_64 GNU/Linux
   
</nowiki></pre>
+
</pre>
   
 
From the output above we can see we're running Xen hypervisor version 4.0.1 (xen_major, xen_minor, xen_extra) with Linux 2.6.32.25 dom0 kernel.
 
From the output above we can see we're running Xen hypervisor version 4.0.1 (xen_major, xen_minor, xen_extra) with Linux 2.6.32.25 dom0 kernel.
   
 
== Updating the dom0 kernel ==
 
== Updating the dom0 kernel ==
Linux 2.6.32.x is the long-term supported and maintained kernel from both kernel.org and xen.org, so it gets pretty frequent updates. Here are the steps to update your kernel tree so you don't have to clone the whole tree again.
+
Linux 2.6.32.x is the long-term supported and maintained kernel from both kernel.org and XenProject.org, so it gets pretty frequent updates. Here are the steps to update your kernel tree so you don't have to clone the whole tree again.
   
 
Clean up and update the git tree, then check the changelog for the recent changes:
 
Clean up and update the git tree, then check the changelog for the recent changes:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# mkdir /root/kernel/linux-2.6-xen
 
[root@el6 ~]# mkdir /root/kernel/linux-2.6-xen
 
[root@el6 linux-2.6-xen]# make clean
 
[root@el6 linux-2.6-xen]# make clean
 
[root@el6 linux-2.6-xen]# git pull
 
[root@el6 linux-2.6-xen]# git pull
 
[root@el6 linux-2.6-xen]# git log | less
 
[root@el6 linux-2.6-xen]# git log | less
</nowiki></pre>
+
</pre>
   
 
And then follow the steps in the previous chapters to build and install the new kernel version.
 
And then follow the steps in the previous chapters to build and install the new kernel version.
Line 431: Line 452:
   
   
<pre><nowiki>
+
<pre>
[root@el6 ~]# yum install libvirt virt-manager xorg-x11-xauth</nowiki></pre>
+
[root@el6 ~]# yum install libvirt virt-manager xorg-x11-xauth
  +
</pre>
   
 
The default version of libvirt in RHEL6 is not compiled with Xen support. You can verify this by using ssh with X11 forwarding from your laptop/desktop to your dom0, and trying to run "virt-manager":
 
The default version of libvirt in RHEL6 is not compiled with Xen support. You can verify this by using ssh with X11 forwarding from your laptop/desktop to your dom0, and trying to run "virt-manager":
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# virt-manager &
 
[root@el6 ~]# virt-manager &
 
[1] 2867
 
[1] 2867
Line 449: Line 471:
 
if ret is None:raise libvirtError('virConnectOpenAuth() failed')
 
if ret is None:raise libvirtError('virConnectOpenAuth() failed')
 
libvirtError: no connection driver available for xen:///
 
libvirtError: no connection driver available for xen:///
</nowiki></pre>
+
</pre>
   
 
This can be fixed by rebuilding libvirt rpms with Xen support enabled.
 
This can be fixed by rebuilding libvirt rpms with Xen support enabled.
Line 458: Line 480:
   
   
<pre><nowiki>
+
<pre>
[root@el6 ~]# rpm -qa | grep libvirt</nowiki></pre>
+
[root@el6 ~]# rpm -qa | grep libvirt
  +
</pre>
   
 
RHEL 6.0 seems to have libvirt version 0.8.1.
 
RHEL 6.0 seems to have libvirt version 0.8.1.
   
Let's download the src.rpm from Redhat ftp server, patch the spec file to enable Xen support, and try rebuilding it.
+
Let's download the src.rpm from Red Hat ftp server, patch the spec file to enable Xen support, and try rebuilding it.
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# cd /root/src
 
[root@el6 ~]# cd /root/src
 
[root@el6 src]# wget ftp://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/libvirt-0.8.1-27.el6.src.rpm
 
[root@el6 src]# wget ftp://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/libvirt-0.8.1-27.el6.src.rpm
Line 489: Line 512:
 
numactl-devel is needed by libvirt-0.8.1-27.el6.x86_64
 
numactl-devel is needed by libvirt-0.8.1-27.el6.x86_64
 
netcf-devel >= 0.1.4 is needed by libvirt-0.8.1-27.el6.x86_64
 
netcf-devel >= 0.1.4 is needed by libvirt-0.8.1-27.el6.x86_64
</nowiki></pre>
+
</pre>
   
 
Ok, so we need more packages installed:
 
Ok, so we need more packages installed:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 SPECS]# yum install libnl-devel xhtml1-dtds libudev-devel libpciaccess-devel yajl-devel libpcap-devel avahi-devel parted-devel device-mapper-devel numactl-devel netcf-devel
 
[root@el6 SPECS]# yum install libnl-devel xhtml1-dtds libudev-devel libpciaccess-devel yajl-devel libpcap-devel avahi-devel parted-devel device-mapper-devel numactl-devel netcf-devel
</nowiki></pre>
+
</pre>
   
 
After that let's try again:
 
After that let's try again:
   
   
<pre><nowiki>
+
<pre>
[root@el6 SPECS]# rpmbuild -bb libvirt.spec</nowiki></pre>
+
[root@el6 SPECS]# rpmbuild -bb libvirt.spec
  +
</pre>
   
 
After a while you'll see:
 
After a while you'll see:
   
   
<pre><nowiki>
+
<pre>
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-0.8.1-27.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-0.8.1-27.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-client-0.8.1-27.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-client-0.8.1-27.el6.x86_64.rpm
Line 513: Line 537:
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-python-0.8.1-27.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-python-0.8.1-27.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-debuginfo-0.8.1-27.el6.x86_64.rpm
 
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-debuginfo-0.8.1-27.el6.x86_64.rpm
</nowiki></pre>
+
</pre>
   
 
Then let's install the xen-enabled rpms (note you need to use --force because the same version of libvirt is already installed):
 
Then let's install the xen-enabled rpms (note you need to use --force because the same version of libvirt is already installed):
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# cd /root/rpmbuild/RPMS/x86_64/
 
[root@el6 ~]# cd /root/rpmbuild/RPMS/x86_64/
 
[root@el6 x86_64]# rpm -Uvh --force libvirt-0.8.1-27.el6.x86_64.rpm libvirt-client-0.8.1-27.el6.x86_64.rpm libvirt-python-0.8.1-27.el6.x86_64.rpm
 
[root@el6 x86_64]# rpm -Uvh --force libvirt-0.8.1-27.el6.x86_64.rpm libvirt-client-0.8.1-27.el6.x86_64.rpm libvirt-python-0.8.1-27.el6.x86_64.rpm
Line 525: Line 549:
 
2:libvirt ########################################### [ 67%]
 
2:libvirt ########################################### [ 67%]
 
3:libvirt-python ########################################### [100%]
 
3:libvirt-python ########################################### [100%]
</nowiki></pre>
+
</pre>
   
 
Now make sure "libvirtd" is set to automatically start on boot:
 
Now make sure "libvirtd" is set to automatically start on boot:
   
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# chkconfig --list libvirtd
 
[root@el6 ~]# chkconfig --list libvirtd
 
libvirtd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
 
libvirtd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
</nowiki></pre>
+
</pre>
   
 
Then it's time to reboot back to Xen !
 
Then it's time to reboot back to Xen !
Line 539: Line 563:
 
== Verifying libvirt with Xen ==
 
== Verifying libvirt with Xen ==
   
<pre><nowiki>
+
<pre>
 
[root@el6 ~]# virsh list
 
[root@el6 ~]# virsh list
 
Id Name State
 
Id Name State
Line 548: Line 572:
 
Name ID Mem VCPUs State Time(s)
 
Name ID Mem VCPUs State Time(s)
 
Domain-0 0 1024 4 r----- 52.8
 
Domain-0 0 1024 4 r----- 52.8
  +
</pre>
 
</nowiki></pre>
 
   
 
So it seems to work. Now you're also able to use graphical "virt-manager" to manage Xen on RHEL6.
 
So it seems to work. Now you're also able to use graphical "virt-manager" to manage Xen on RHEL6.
Line 555: Line 578:
 
== Bridge problems and errors with custom dom0 kernel ==
 
== Bridge problems and errors with custom dom0 kernel ==
   
<pre><nowiki>
+
<pre>
 
# brctl show
 
# brctl show
 
bridge name    bridge id              STP enabled    interfaces
 
bridge name    bridge id              STP enabled    interfaces
Line 571: Line 594:
 
/sys/class/net/virbr0/bridge: No such file or directory
 
/sys/class/net/virbr0/bridge: No such file or directory
 
8000.000000000000Â Â Â Â Â Â yes
 
8000.000000000000Â Â Â Â Â Â yes
</nowiki></pre>
+
</pre>
   
   
This problem happens because Redhat added bridge IGMP snooping support to rhel6 2.6.32 kernel, and also to rhel6 bridge-utils. If you use mainline or custom Linux 2.6.32 kernel with el6 distro then you won't have that bridge IGMP snooping support in the kernel, and bridge-utils (brctl) will give errors like above.
+
This problem happens because Red Hat added bridge IGMP snooping support to rhel6 2.6.32 kernel, and also to rhel6 bridge-utils. If you use mainline or custom Linux 2.6.32 kernel with el6 distro then you won't have that bridge IGMP snooping support in the kernel, and bridge-utils (brctl) will give errors like above.
   
 
There are two workarounds:
 
There are two workarounds:
Line 609: Line 632:
 
== Network setup details and troubleshooting ==
 
== Network setup details and troubleshooting ==
   
See [[Fedora13Xen4Tutorial:]] for more information about virbr0 setup/configuration for NAT/DHCP, routing, firewall rules, etc.
+
See [[Fedora13Xen4Tutorial]] for more information about virbr0 setup/configuration for NAT/DHCP, routing, firewall rules, etc.
 
== Binary Xen rpm repositories for EL6 ==
 
 
Take a look at these repositories:
 
* Michael Young's EL6 Xen rpm repository (also includes xen dom0 capable kernel based on rhel6 2.6.32 kernel):
 
* http://xenbits.xen.org/people/mayoung/EL6.xen/
 
* http://xenbits.xen.org/people/mayoung/testing/
 
* Gitco.de repository is based on this tutorial, but contains xen dom0 capable kernel rpm based on the xen/kernel.org kernel (not rhel6 kernel):
 
* http://www.gitco.de/linux/x86_64/centos/6/
 
* http://www.gitco.de/linux/x86_64/centos/6/gitco-centos6-x86_64.repo
 
 
Additional repositories:
 
* Fedora xendom0 kernel rpms by Michael Young: http://repos.fedorapeople.org/repos/myoung/dom0-kernel/
 
* http://wiki.alteeve.com/files/an-cluster/rhel6/
 
   
 
== Links for more information ==
 
== Links for more information ==
Line 635: Line 644:
 
More information about Xen:
 
More information about Xen:
   
  +
* [[Xen4.0]]
* http://wiki.xensource.com/xenwiki/Xen4.0
 
* http://wiki.xensource.com/xenwiki/XenParavirtOps
+
* [[XenParavirtOps]]
* http://wiki.xensource.com/xenwiki/XenBestPractices
+
* [[XenBestPractices]]
* http://wiki.xensource.com/xenwiki/XenCommonProblems
+
* [[XenCommonProblems]]
* http://wiki.xensource.com/xenwiki/XenDom0Kernels
+
* [[XenDom0Kernels]]
* http://wiki.xensource.com/xenwiki/XenKernelFeatures
+
* [[XenKernelFeatures]]
   
 
Additional links:
 
Additional links:
Line 648: Line 657:
   
 
End of tutorial.
 
End of tutorial.
  +
  +
[[Category:Users]]
  +
[[Category:Beginners]]
  +
[[Category:Tutorial]]
  +
[[Category:Xen]]
  +
[[Category:Host Install]]
  +
[[Category:RHEL]]

Latest revision as of 03:45, 26 February 2015


Binary Xen rpm repositories for EL6

Current repositories:

  • Xen Made Easy - http://xen.crc.id.au (Install Guide - http://xen.crc.id.au/support/guides/install/) 2015-Jan-24: Xen Made Easy worked under RHEL 6.6. The steps below are more complex, but give a more complete view of what to do. I did not complete the steps below, I switched over to Xen Made Easy. However, I did update some URLs below.

Older repositories (mostly dead):

Red Hat Enterprise Linux 6 Xen 4.0 tutorial

This tutorial explains how to install and configure RHEL6 to run as Xen dom0 (host), running Xen hypervisor 4.0. Note that this is unsupported by Red Hat! See the end of this wiki page for more information about RHEL6 Xen binary rpm yum repository. This tutorial explains how to compile Xen and dom0 kernel from sources/srpms.

As a default RHEL6 does NOT:

  • RHEL6 does not ship Xen hypervisor or tools.
  • RHEL6 does not ship Xen dom0 capable kernel.

As a default RHEL6 DOES:

  • RHEL6 runs as Xen PV (paravirtual) domU using the pvops framework in the default kernel.
  • RHEL6 runs as Xen HVM (fully virtualized) guest.
  • RHEL6 ships with optimized Xen PV-on-HVM drivers for RHEL6 Xen HVM guests.

Topics in this tutorial:

  • How to install and configure RHEL6 system for running as Xen dom0 (host).
  • How to get Xen src.rpm from Fedora and rebuild and install it on RHEL6.
  • How to install upstream Xen dom0 capable kernel on RHEL6.
  • How to update the dom0 kernel to a newer version from the upstream git tree.
  • How to use libvirt and virt-manager to manage Xen on RHEL6.
  • Binary rpm repositories for EL6.
  • More information and links.

TODO list for this tutorial:

  • Fix the src.rpm compilation examples, shouldn't build as root.
  • Make sure everything works with SElinux enabled, and leave it enabled.

RHEL6 installation and configuration

Installation:

  • Download "rhel-server-6.0-x86_64-dvd.iso" from Red Hat Network (RHN).
  • Burn it to DVDR, or if you have a server with remote management, use virtual media ISO mapping.
  • Boot from the DVD.
  • Install as usual.
  • Choose "Basic Server" install.
  • Disk Partitioning: Create /boot partition as primary, make it ext3 and at least 2 GB in size. Create LVM PV (Physical Volume) and VG (Volume Group) to fill the rest of the disk space. Create root as LVM volume, make it ext4, and at least 36 GB in size. Create swap as LVM volume, make it at least as big as the amount of RAM in your server.
  • IMPORTANT! Make sure you leave FREE space in the LVM volume group. You can use that free space later to create LVM volumes for Xen VM virtual disks.
  • In this tutorial the hostname is set to "el6.localdomain".

Screenshot of disk partitioning: rhel6-xen4-tutorial-partitioning.jpg

Configuration after installation:

First fix networking by editing "/etc/sysconfig/network-scripts/ifcfg-ethX", usually "ifcfg-eth0", and make it look like:


[root@el6 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
HWADDR="00:11:22:33:44:55"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="dhcp"

ie. make sure NM_CONTROLLED="no" and ONBOOT="yes". Then run "ifup eth0" to activate networking.

Make sure network service is set to start automatically:


[root@el6 ~]# chkconfig --list network

Make sure your Internet connection works OK at this point! Next make sure "/etc/hosts" file has an entry for your hostname:


[root@el6 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1   el6 el6.localdomain

Full FQDN hostname in this example is "el6.localdomain".

Disable SElinux by editing "/etc/selinux/config" and make it have:


SELINUX=disabled

Then it's time to register your system to RHN, so you can fetch updates and install additional packages using yum.

Run "rhn_register" and fill in your RHN username and password. When registration is OK, there's an important extra step do!

You need to login to RHN website and:

  • Click "Manage Entitlements & Subscriptions".
  • Click the correct (newly added) system.
  • Click "(Alter Channel Subscriptions)".
  • Select ""RHEL Server Optional (v. 6 64-bit x86_64)" channel.
  • Click "Change Subscriptions".

Some required packages are only available in the "RHEL Server Optional (v. 6 64-bit x86_64)" channel. Failing to add that channel means you won't be able to install some required packages later on.

When you're done with RHN update the system and install latest (security) fixes:


[root@el6 ~]# yum update

Then install some commonly used tools:


[root@el6 ~]# yum install screen vim wget tcpdump ntp ntpdate man smartmontools links lynx ethtool xorg-x11-xauth

Then increase grub timeout from grub.conf, and comment out "hiddenmenu" option:


[root@el6 ~]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg00-lvroot
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-71.7.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.7.1.el6.x86_64 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot rd_LVM_LV=vg00/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi crashkernel=auto rhgb quiet
        initrd /initramfs-2.6.32-71.7.1.el6.x86_64.img
title Red Hat Enterprise Linux (2.6.32-71.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot rd_LVM_LV=vg00/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi crashkernel=auto rhgb quiet
        initrd /initramfs-2.6.32-71.el6.x86_64.img

At this point you should reboot the system to the newest kernel.

Installing required packages to compile Xen rpms

Install packages that are required to rebuild Xen src.rpm (all on one line):


[root@el6 ~]# yum groupinstall "Development tools" "Additional Development" "Debugging Tools" "System administration tools" "Compatibility libraries" "Console internet tools" "Desktop Platform Development"

This is around 212 MB of downloads.

Then install some additional packages that are required to build Xen with all features enabled:


[root@el6 ~]# yum install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo libuuid-devel

This is around 82 MB of downloads.

And finally 32bit version of glibc-devel is required aswell:


[root@el6 ~]# yum install glibc-devel.i686

Now you have all the basic tools, libraries and headers installed.

Rebuilding and installing Xen src.rpm

Fedora xen-4.0.1-6 rpm package includes some additional patches and fixes backported from upstream Xen 4.0.2-rc*.

(FIXME: These instructions should be updated to a more recent version of Xen.)

(FIXME: These instructions should be updated to support RHEL Server 7 and discuss how to select the appropriate version of Fedora. One possibility is to look at the https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux for the version of Fedora)

[root@el6 ~]# mkdir /root/src && cd /root/src
[root@el6 src]# wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/releases/14/Everything/source/SRPMS/xen-4.0.1-6.fc14.src.rpm 
[root@el6 src]# rpm -i xen-4.0.1-6.fc14.src.rpm
[root@el6 src]# cd /root/rpmbuild/SPECS
[root@el6 SPECS]# rpmbuild -bb xen.spec

After a while you should see output like:


Wrote: /root/rpmbuild/RPMS/x86_64/xen-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-libs-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-runtime-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-hypervisor-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-doc-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-devel-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-licenses-4.0.1-6.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/xen-debuginfo-4.0.1-6.el6.x86_64.rpm

Try installing the rpms:


[root@el6 SPECS]# cd /root/rpmbuild/RPMS/x86_64/
[root@el6 x86_64]# rpm -Uvh xen*4.0.1-6*.rpm
error: Failed dependencies:
        /usr/bin/qemu-nbd is needed by xen-runtime-4.0.1-6.el6.x86_64

So what's happening here? It seems RHEL6 does not have "qemu-common" rpm available, which includes "qemu-nbd" binary. So we need to grab qemu src.rpm from Fedora and rebuild it for RHEL6 to get the required "qemu-common" rpm.

Verify the version of "qemu" in RHEL6:


[root@el6 x86_64]# rpm -qa | grep qemu
qemu-img-0.12.1.2-2.113.el6.x86_64

Ok, so it's version 0.12.x. Fedora 13 has qemu 0.12.5, so it's close enough:


[root@el6 x86_64]# cd /root/src
[root@el6 src]# wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/updates/13/SRPMS/qemu-0.12.5-1.fc13.src.rpm
[root@el6 src]# rpm -i qemu-0.12.5-1.fc13.src.rpm
[root@el6 src]# cd /root/rpmbuild/SPECS
[root@el6 SPECS]# rpmbuild -bb qemu.spec

(FIXME: 2015-01-24: The above wget did not, I tried wget http://archives.fedoraproject.org/pub/archive/fedora/linux/updates/13/SRPMS/qemu-0.13.0-1.fc13.src.rpm )

After a while you'll see:


Wrote: /root/rpmbuild/RPMS/x86_64/qemu-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-kvm-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-img-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-common-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-user-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-x86-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-ppc-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-sparc-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-arm-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-mips-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-cris-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-m68k-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-system-sh4-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-kvm-tools-0.12.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/qemu-debuginfo-0.12.5-1.el6.x86_64.rpm

So now we have the required "qemu-common" rpm. Let's install it to satisfy the package dependencies:


[root@el6 SPECS]# cd /root/rpmbuild/RPMS/x86_64
[root@el6 x86_64]# rpm -Uvh qemu-common-0.12.5-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:qemu-common            ########################################### [100%]

Now we can also install Xen rpms:


[root@el6 x86_64]# rpm -Uvh xen*4.0.1-6*.rpm
Preparing...                ########################################### [100%]
   1:xen-licenses           ########################################### [ 13%]
   2:xen-libs               ########################################### [ 25%]
   3:xen-hypervisor         ########################################### [ 38%]
   4:xen-runtime            ########################################### [ 50%]
   5:xen                    ########################################### [ 63%]
   6:xen-devel              ########################################### [ 75%]
   7:xen-doc                ########################################### [ 88%]
   8:xen-debuginfo          ########################################### [100%]

Now you're done installing Xen hypervisor, tools and libraries.

Installing upstream Xen dom0 capable kernel

Note that we're installing upstream *Xen* kernel here, which is based on the long-term maintained kernel.org Linux 2.6.32.x kernel. This kernel is different from the RHEL6 kernel. Many drivers are newer versions in Red Hat's kernel, and Red Hat's kernel might have more features than the kernel.org kernel. Due to the driver version differences you might experience problems on some hardware, for example on servers using LSI/DELL RAID adapters. You might need to manually update driver versions to fix these problems on affected systems. See the end of this wiki page for more information about kernel versions.

Clone the kernel git repository from kernel.org and check out the "xen/stable-2.6.32.x" long-term maintained branch.


[root@el6 x86_64]# mkdir /root/kernel && cd /root/kernel
[root@el6 kernel]# git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
[root@el6 kernel]# cd linux-2.6-xen
[root@el6 linux-2.6-xen]# git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x

(FIXME: The git command failed, the message was "fatal: git checkout: updating paths is incompatible with switching branches.". http://lists.xen.org/archives/html/xen-devel/2012-01/msg02597.htm suggests "git checkout -b xen/next-2.6.32 origin/xen/next-2.6.32", which worked. However, I did not complete the tutorial, I switched over to Xen Made Easy, see above.)


Now you have the kernel ready to be configured and compiled. Reference config-file used is for version "2.6.32.25". It should work with newer version aswell, "make oldconfig" takes care of that.

NOTE! The reference .config file has various debugging options enabled, so don't use this kernel config for performance benchmarks!

[root@el6 linux-2.6-xen]# wget -O .config http://pasik.reaktio.net/xen/kernel-config/config-2.6.32.25-pvops-dom0-xen-stable-x86_64
[root@el6 linux-2.6-xen]# make oldconfig
[root@el6 linux-2.6-xen]# make -j4 bzImage && make -j4 modules && make modules_install

Then let's install the kernel and generate initrd (initramfs) image for the new kernel. Kernel version in this example is "2.6.32.25":


[root@el6 linux-2.6-xen]# depmod -a 2.6.32.25
[root@el6 linux-2.6-xen]# cp -a arch/x86/boot/bzImage /boot/vmlinuz-2.6.32.25
[root@el6 linux-2.6-xen]# cp -a System.map /boot/System.map-2.6.32.25
[root@el6 linux-2.6-xen]# cp -a .config /boot/config-2.6.32.25
[root@el6 linux-2.6-xen]# cd /boot
[root@el6 boot]# dracut initramfs-2.6.32.25.img 2.6.32.25

NOTE! If you're using a system with LSI and/or Dell RAID adapters (using megaraid_sas driver) you need to manually update the driver version at this point! Linux 2.6.32.25 ships with too old driver version, and it wont' work (disks won't get enabled on system startup). See this email for information about updating the driver version: http://lists.xensource.com/archives/html/xen-devel/2010-11/msg00250.html .

Then it's time to modify "grub.conf" and add an entry for the Xen+dom0 combination:


[root@el6 ~]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg00-lvroot
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-71.7.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.7.1.el6.x86_64 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot rd_LVM_LV=vg00/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi crashkernel=auto rhgb quiet
        initrd /initramfs-2.6.32-71.7.1.el6.x86_64.img
title Red Hat Enterprise Linux (2.6.32-71.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot rd_LVM_LV=vg00/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi crashkernel=auto rhgb quiet
        initrd /initramfs-2.6.32-71.el6.x86_64.img
title Xen 4.0 with Linux 2.6.32.25 dom0
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
        module /vmlinuz-2.6.32.25 ro root=/dev/mapper/vg00-lvroot rd_LVM_LV=vg00/lvroot nomodeset
        module /initramfs-2.6.32.25.img

Then let's disable KSM services, those don't work with Xen currently:


[root@el6 ~]# chkconfig ksm off
[root@el6 ~]# chkconfig ksmtuned off

And then it's time to reboot into Xen ! Remember to choose the Xen entry from grub:


[root@el6 ~]# reboot

Verifying Xen after reboot

[root@el6 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1024     4     r-----     93.9

[root@el6 ~]# xm info
host                   : el6.localdomain
release                : 2.6.32.25
version                : #1 SMP Sun Nov 14 21:50:39 EET 2010
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2826
hw_caps                : bfebfbff:20100800:00000000:00000940:0008e3fd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 8190
free_memory            : 7070
node_to_cpu            : node0:0-3
node_to_memory         : node0:7070
node_to_dma32_mem      : node0:3259
max_node_id            : 0
xen_major              : 4
xen_minor              : 0
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : dom0_mem=1024M loglvl=all guest_loglvl=all
cc_compiler            : gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC)
cc_compile_by          : root
cc_compile_domain      :
cc_compile_date        : Sun Nov 14 21:41:09 EET 2010
xend_config_format     : 4

[root@el6 ~]# uname -a
Linux el6.localdomain 2.6.32.25 #1 SMP Sun Nov 14 21:50:39 EET 2010 x86_64 x86_64 x86_64 GNU/Linux

From the output above we can see we're running Xen hypervisor version 4.0.1 (xen_major, xen_minor, xen_extra) with Linux 2.6.32.25 dom0 kernel.

Updating the dom0 kernel

Linux 2.6.32.x is the long-term supported and maintained kernel from both kernel.org and XenProject.org, so it gets pretty frequent updates. Here are the steps to update your kernel tree so you don't have to clone the whole tree again.

Clean up and update the git tree, then check the changelog for the recent changes:


[root@el6 ~]# mkdir /root/kernel/linux-2.6-xen
[root@el6 linux-2.6-xen]# make clean
[root@el6 linux-2.6-xen]# git pull
[root@el6 linux-2.6-xen]# git log | less

And then follow the steps in the previous chapters to build and install the new kernel version.

Online changelog for the "xen/stable-2.6.32.x" branch of the kernel git tree is located here: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x .

Using libvirt and virt-manager with Xen

Install libvirt and virt-manager:


[root@el6 ~]# yum install libvirt virt-manager xorg-x11-xauth

The default version of libvirt in RHEL6 is not compiled with Xen support. You can verify this by using ssh with X11 forwarding from your laptop/desktop to your dom0, and trying to run "virt-manager":


[root@el6 ~]# virt-manager &
[1] 2867

Unable to open connection to hypervisor URI 'xen:///':
no connection driver available for xen:///
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 992, in _try_open
    None], flags)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 111, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: no connection driver available for xen:///

This can be fixed by rebuilding libvirt rpms with Xen support enabled.

NOTE! Before rebuilding libvirt src.rpm you need to boot into non-Xen environment! This is because some of the automated RHEL6 libvirt Xen-specific tests seem to fail in Xen dom0 environment, but work OK in baremetal.

Verify libvirt version in RHEL6:


[root@el6 ~]# rpm -qa | grep libvirt

RHEL 6.0 seems to have libvirt version 0.8.1.

Let's download the src.rpm from Red Hat ftp server, patch the spec file to enable Xen support, and try rebuilding it.


[root@el6 ~]# cd /root/src
[root@el6 src]# wget ftp://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/libvirt-0.8.1-27.el6.src.rpm
[root@el6 src]# rpm -i libvirt-0.8.1-27.el6.src.rpm
[root@el6 src]# wget http://pasik.reaktio.net/xen/patches/libvirt-spec-rhel6-enable-xen.patch
[root@el6 src]# cd /root/rpmbuild/SPECS
[root@el6 SPECS]# cp -a libvirt.spec libvirt.spec.orig
[root@el6 SPECS]# patch -p0 < ~/src/libvirt-spec-rhel6-enable-xen.patch
patching file libvirt.spec

[root@el6 SPECS]# rpmbuild -bb libvirt.spec
error: Failed build dependencies:
        libnl-devel >= 1.1 is needed by libvirt-0.8.1-27.el6.x86_64
        xhtml1-dtds is needed by libvirt-0.8.1-27.el6.x86_64
        libudev-devel >= 145 is needed by libvirt-0.8.1-27.el6.x86_64
        libpciaccess-devel >= 0.10.9 is needed by libvirt-0.8.1-27.el6.x86_64
        yajl-devel is needed by libvirt-0.8.1-27.el6.x86_64
        libpcap-devel is needed by libvirt-0.8.1-27.el6.x86_64
        avahi-devel is needed by libvirt-0.8.1-27.el6.x86_64
        parted-devel is needed by libvirt-0.8.1-27.el6.x86_64
        device-mapper-devel is needed by libvirt-0.8.1-27.el6.x86_64
        numactl-devel is needed by libvirt-0.8.1-27.el6.x86_64
        netcf-devel >= 0.1.4 is needed by libvirt-0.8.1-27.el6.x86_64

Ok, so we need more packages installed:


[root@el6 SPECS]# yum install libnl-devel xhtml1-dtds libudev-devel libpciaccess-devel yajl-devel libpcap-devel avahi-devel parted-devel device-mapper-devel numactl-devel netcf-devel

After that let's try again:


[root@el6 SPECS]# rpmbuild -bb libvirt.spec

After a while you'll see:


Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-0.8.1-27.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-client-0.8.1-27.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-devel-0.8.1-27.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-python-0.8.1-27.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/libvirt-debuginfo-0.8.1-27.el6.x86_64.rpm

Then let's install the xen-enabled rpms (note you need to use --force because the same version of libvirt is already installed):


[root@el6 ~]# cd /root/rpmbuild/RPMS/x86_64/
[root@el6 x86_64]# rpm -Uvh --force libvirt-0.8.1-27.el6.x86_64.rpm libvirt-client-0.8.1-27.el6.x86_64.rpm libvirt-python-0.8.1-27.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:libvirt-client         ########################################### [ 33%]
   2:libvirt                ########################################### [ 67%]
   3:libvirt-python         ########################################### [100%]

Now make sure "libvirtd" is set to automatically start on boot:


[root@el6 ~]# chkconfig --list libvirtd
libvirtd        0:off   1:off   2:off   3:on    4:on    5:on    6:off

Then it's time to reboot back to Xen !

Verifying libvirt with Xen

[root@el6 ~]# virsh list
 Id Name                 State
----------------------------------
  0 Domain-0             running

[root@el6 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1024     4     r-----     52.8

So it seems to work. Now you're also able to use graphical "virt-manager" to manage Xen on RHEL6.

Bridge problems and errors with custom dom0 kernel

# brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0Â Â Â Â Â Â Â Â Â  /sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
8000.000000000000Â Â Â Â Â Â  yes


This problem happens because Red Hat added bridge IGMP snooping support to rhel6 2.6.32 kernel, and also to rhel6 bridge-utils. If you use mainline or custom Linux 2.6.32 kernel with el6 distro then you won't have that bridge IGMP snooping support in the kernel, and bridge-utils (brctl) will give errors like above.

There are two workarounds:

  • Use Xen dom0 kernel based on actual rhel6 kernel/patches, mayoung does have "xendom0" kernel available for el6 distros based on actual rhel6 kernel, with xen dom0 pvops patches added on top, so it includes bridge IGMP snooping support and the default el6 bridge-utils works OK without errors. See the end of this wiki tutorial page for links to xendom0 kernel rpms.
  • If using custom dom0 kernel (xen.git xen/stable-2.6.32.x) then rebuild the el6 bridge-utils src.rpm and remove the IGMP snooping patch from the spec file: http://lists.xensource.com/archives/html/xen-devel/2011-02/msg00760.html .

Using graphical virt-manager to manage Xen on RHEL6

Some screenshots of using virt-manager with Xen on RHEL6 and installing Fedora 14 Xen PV domU. Note that you don't need (and probably shouldn't) run Xserver on dom0! You can use ssh X11 forwarding to run virt-manager on dom0 but display the virt-manager GUI on your local laptop/desktop. See the Fedora 13 tutorial link in the end of this page for more information about using ssh X11 forwarding.

rhel6-virtmanager01.png

rhel6-virtmanager02.png

rhel6-virtmanager03.png

rhel6-virtmanager04.png

rhel6-virtmanager05.png

rhel6-virtmanager06.png

rhel6-virtmanager07.png

rhel6-virtmanager08.png

rhel6-virtmanager09.png

rhel6-virtmanager10.png

rhel6-virtmanager12.png

rhel6-virtmanager13.png

Network setup details and troubleshooting

See Fedora13Xen4Tutorial for more information about virbr0 setup/configuration for NAT/DHCP, routing, firewall rules, etc.

Links for more information

See this Fedora 13 Xen 4.0 tutorial for more information about installing different kinds of Xen guest VMs and other tips: Fedora13Xen4Tutorial .

More information about RHEL and Xen:

More information about Xen:

Additional links:

End of tutorial.