RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial

From Xen
Revision as of 18:11, 16 August 2012 by Pasik (talk | contribs)

Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen

Requirements:

  • You need at least RHEL 5.8 / CentOS 5.8. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.7 VF passthru works only once, and fails on the second time.
  • You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.

Components used in this tutorial:

  • Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
  • Intel Xeon CPU L5640.
  • Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
  • IOMMU/VT-d enabled in BIOS.
  • CentOS 5.8 x86_64 DVD1.
  • Stock Xen rpms from CentOS 5.8.
  • Stock kernel-xen from CentOS 5.8 as dom0 kernel.

EL5 Host installation and configuration

  • Install RHEL 5.8 or CentOS 5.8 x64 (64 bit) host using the "Server" profile from DVD1.
  • Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
  • After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
  • Do all the usual network/IP, hostname, DNS etc configuration.
  • Make sure there is free unallocated space in the LVM volume group:
[root@dom0 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  38
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                27
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               32.00 MB
  Total PE              59551
  Alloc PE / Size       55670 / 1.70 TB
  Free  PE / Size       3881 / 121.28 GB
  VG UUID               QuouSc-MXj2-Hhdo-Ut9k-RLLO-jk9N-RjQD6E
  • Note the "Free PE / Size" line on the vgdisplay command output above.
  • Update the system with "yum update".
  • Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off
chkconfig mcstrans off
chkconfig haldaemon off
chkconfig hidd off
chkconfig autofs off
chkconfig avahi-daemon off
chkconfig xfs off
chkconfig bluetooth off
chkconfig pcscd off
chkconfig iptables off
chkconfig ip6tables off
  • Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
  • Edit "/etc/xen/xend-config.sxp" and disable (comment out) Xen network-script line:
#(network-script network-bridge)
  • We want to configure dom0 networking settings and bridges ourselves.
  • Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title CentOS (2.6.18-308.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all
        module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-308.11.1.el5xen.img
  • Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
  • Make sure the Xen entry is the default in Grub
  • Reboot the system to Xen.
  • When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     2048    12 r-----    788.0
  • Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info
host                   : dom0.localdomain
release                : 2.6.18-308.11.1.el5xen
version                : #1 SMP Tue Jul 10 09:29:47 EDT 2012
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001
total_memory           : 49139
free_memory            : 46145
node_to_cpu            : node0:0-11
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-308.11.1.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Jul 10 08:40:13 EDT 2012
xend_config_format     : 2
  • Versions of xen related rpms:
[root@dom0 ~]# rpm -qa | grep xen
xen-libs-3.0.3-135.el5_8.4
xen-3.0.3-135.el5_8.4
xen-libs-3.0.3-135.el5_8.4
kernel-xen-2.6.18-308.11.1.el5
  • Note that the actual Xen hypervisor is version 3.1.2 (+ a lot of patches from later Xen versions), only the userland tools are 3.0.3 based. Xen hypervisor is included in kernel-xen rpm in RHEL5/CentOS5.

Intel SR-IOV NIC

  • In this tutorial we're using the following Intel SR-IOV capable 10 Gbit/sec 82599 NIC:
[root@dom0 ~]# lspci | grep 10-Gig
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  • which uses the Intel ixgbe driver, as can be seen from the "ethtool -i" output:
[root@dom0 ~]# ethtool -i eth2
driver: ixgbe
version: 3.4.8-k
firmware-version: 0.9-3
bus-info: 0000:03:00.0
[root@dom0 ~]# ethtool -i eth3
driver: ixgbe
version: 3.4.8-k
firmware-version: 0.9-3
bus-info: 0000:03:00.1

Enabling IOMMU support in Xen command line options

  • Edit "/boot/grub/grub.conf" and add "iommu=1" option for Xen, and "pci_pt_e820_access=on" option for dom0 Linux kernel (vmlinuz). Both options are required:
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title CentOS (2.6.18-308.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all iommu=1
        module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00 pci_pt_e820_access=on
        module /initrd-2.6.18-308.11.1.el5xen.img

Enable Intel SR-IOV NIC Virtual Functions (VFs) in the ixgbe driver options

  • Add the following line to end of "/etc/modprobe.conf":
options ixgbe max_vfs=8
  • 8 is just an example here, 82599 NIC supports up to 64 VFs.
  • Blacklist the Intel VF driver (ixgbevf) in dom0 so that dom0 kernel doesn't try to use the Virtual Functions (we want to PCI passthru them to Xen VMs!). Create "/etc/modprobe.d/blacklist-ixgbevf.conf" file with the following contents:
# intel ixgbe sr-iov vf (virtual function) driver
blacklist ixgbevf
  • Now reboot the system to activate the changes.
  • After reboot check that the IOMMU support gets enabled in Xen hypervisor from the Xen dmesg log:
[root@dom0 ~]# xm dmesg | grep -i vt-d | grep -i enable
(XEN) Intel VT-d has been enabled
(XEN) Intel VT-d snoop control enabled
(XEN) [VT-D]iommu.c:619: iommu_enable_translation: iommu->reg = ffff828bfff58000
  • If VT-d IOMMU doesn't get enabled, read the full "xm dmesg | less" log for more information.
  • Also verify you can see the VFs in "lspci" output:
[root@dom0 ~]# lspci | grep "Virtual Function"
03:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  • Why 16 VFs when we configured max_vfs=8? There's 8 per NIC/port, so 16 total for 2 ports.
  • Also check the dom0 Linux kernel dmesg for the ixgbe driver logs:
[root@dom0 ~]# dmesg | grep ixgbe
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
ixgbe: Copyright (c) 1999-2011 Intel Corporation.
ixgbe 0000:03:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe 0000:03:00.0: (PCI Express:5.0GT/s:Width x8) 00:3b:31:77:9e:1c
ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: E81283-002
ixgbe 0000:03:00.0: eth2: IOV is enabled with 8 VFs
ixgbe 0000:03:00.0: eth2: IOV: VF 0 is enabled MAC fe:71:c2:a8:4d:28
ixgbe 0000:03:00.0: eth2: IOV: VF 1 is enabled MAC a2:cd:8b:d7:a0:3e
ixgbe 0000:03:00.0: eth2: IOV: VF 2 is enabled MAC 2a:e4:d6:69:a2:9a
ixgbe 0000:03:00.0: eth2: IOV: VF 3 is enabled MAC 5a:56:e0:a4:7d:36
ixgbe 0000:03:00.0: eth2: IOV: VF 4 is enabled MAC ae:f5:bb:ff:f4:bc
ixgbe 0000:03:00.0: eth2: IOV: VF 5 is enabled MAC 2a:be:89:3e:53:3b
ixgbe 0000:03:00.0: eth2: IOV: VF 6 is enabled MAC 1a:61:ce:c8:a7:c0
ixgbe 0000:03:00.0: eth2: IOV: VF 7 is enabled MAC 96:d7:3f:c4:f2:aa
ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
ixgbe 0000:03:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe 0000:03:00.1: (PCI Express:5.0GT/s:Width x8) 00:3b:31:77:9e:1d
ixgbe 0000:03:00.1: MAC: 2, PHY: 8, SFP+: 4, PBA No: E81283-002
ixgbe 0000:03:00.1: eth3: IOV is enabled with 8 VFs
ixgbe 0000:03:00.1: eth3: IOV: VF 0 is enabled MAC ee:b0:2b:65:c4:6c
ixgbe 0000:03:00.1: eth3: IOV: VF 1 is enabled MAC 0a:f0:de:c0:b1:4d
ixgbe 0000:03:00.1: eth3: IOV: VF 2 is enabled MAC 9a:ab:7f:4a:d6:ce
ixgbe 0000:03:00.1: eth3: IOV: VF 3 is enabled MAC ee:f3:59:15:1b:02
ixgbe 0000:03:00.1: eth3: IOV: VF 4 is enabled MAC be:cd:68:3c:bc:f2
ixgbe 0000:03:00.1: eth3: IOV: VF 5 is enabled MAC a2:66:eb:00:25:80
ixgbe 0000:03:00.1: eth3: IOV: VF 6 is enabled MAC 6e:3e:da:74:84:63
ixgbe 0000:03:00.1: eth3: IOV: VF 7 is enabled MAC de:06:3d:af:8c:ce
ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection

Configuring Xen pciback for VF PCI passthru

  • First check available PCI devices for Xen PCI passthru:
[root@dom0 ~]# xm pci-list-assignable-devices
Error: pciback not loaded?
  • The error message makes sense, we haven't yet configured any PCI devices for passthru, and we haven't loaded the Xen pciback module in dom0 kernel.
  • Edit "/etc/modprobe.conf" and add options for the Xen pciback dom0 kernel module:
options pciback hide=(03:10.0)(03:10.1)(03:10.2)(03:10.3)(03:10.4)(03:10.5)(03:10.6)(03:10.7)(03:11.0)(03:11.1)(03:11.2)(03:11.3)(03:11.4)(03:11.5)(03:11.6)(03:11.7)
  • Here we are "hiding" all the Virtual Function PCI device IDs, so they can be used for Xen PCI passthru.
  • Now load the "pciback" module in dom0 kernel:
modprobe pciback
  • After loading "pciback" module check the dom0 Linux kernel dmesg:
pciback 0000:03:10.0: seizing device
pciback 0000:03:10.2: seizing device
pciback 0000:03:10.4: seizing device
pciback 0000:03:10.6: seizing device
pciback 0000:03:11.0: seizing device
pciback 0000:03:11.2: seizing device
pciback 0000:03:11.4: seizing device
pciback 0000:03:11.6: seizing device
pciback 0000:03:10.1: seizing device
pciback 0000:03:10.3: seizing device
pciback 0000:03:10.5: seizing device
pciback 0000:03:10.7: seizing device
pciback 0000:03:11.1: seizing device
pciback 0000:03:11.3: seizing device
pciback 0000:03:11.5: seizing device
pciback 0000:03:11.7: seizing device
PCI: Enabling device 0000:03:11.7 (0000 -> 0002)
PCI: Enabling device 0000:03:11.5 (0000 -> 0002)
PCI: Enabling device 0000:03:11.3 (0000 -> 0002)
PCI: Enabling device 0000:03:11.1 (0000 -> 0002)
PCI: Enabling device 0000:03:10.7 (0000 -> 0002)
PCI: Enabling device 0000:03:10.5 (0000 -> 0002)
PCI: Enabling device 0000:03:10.3 (0000 -> 0002)
PCI: Enabling device 0000:03:10.1 (0000 -> 0002)
PCI: Enabling device 0000:03:11.6 (0000 -> 0002)
PCI: Enabling device 0000:03:11.4 (0000 -> 0002)
PCI: Enabling device 0000:03:11.2 (0000 -> 0002)
PCI: Enabling device 0000:03:11.0 (0000 -> 0002)
PCI: Enabling device 0000:03:10.6 (0000 -> 0002)
PCI: Enabling device 0000:03:10.4 (0000 -> 0002)
PCI: Enabling device 0000:03:10.2 (0000 -> 0002)
PCI: Enabling device 0000:03:10.0 (0000 -> 0002)
  • Now check again what PCI devices are ready for Xen PCI passthru:
[root@dom0 ~]# xm pci-list-assignable-devices
0000:03:10.1
0000:03:10.0
0000:03:10.2
0000:03:10.3
0000:03:10.4
0000:03:10.5
0000:03:10.6
0000:03:10.7
0000:03:11.0
0000:03:11.1
0000:03:11.2
0000:03:11.3
0000:03:11.4
0000:03:11.5
0000:03:11.6
0000:03:11.7