Xen On Funtoo From Scratch

Hardware Configuration

• VGA: Nvidia GeForce 9800 GT
• CPU: Intel Core i7 3770
• Motherboard: DQ77MK

Used Software

• Dom0 OS: Funtoo GNU/Linux distro (derrived from gentoo).
• Kernel: 3.4.4
• Xen: 4.2.0-unstable rev 25099
• DomU OS: Windows XP SP3

Reference documentation

• Creating bootable USB-stick from gentoo livecd: http://www.gentoo.org/doc/en/liveusb.xml
• Funtoo installation: http://www.funtoo.org/wiki/Funtoo_Linux_Installation
• Xen 4.1 installation on gentoo: http://en.gentoo-wiki.com/wiki/Xen4.1
• Xen 4.2 installation on ubuntu: http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-%2042unstable-patches-for-vga-pass-through
• LVM installation guide on funtoo: http://www.funtoo.org/wiki/Rootfs_over_encrypted_lvm

Make bootable USB

Preparation

   # emerge -av dosfstools syslinux
   

Make partitions

   # fdisk -l /dev/sdb  
   
   Disk /dev/sdb: 8422 MB, 8422162432 bytes
   239 heads, 47 sectors/track, 1464 cylinders, total 16449536 sectors
   Units = sectors of 1 * 512 = 512 bytes
   Sector size (logical/physical): 512 bytes / 512 bytes
   I/O size (minimum/optimal): 512 bytes / 512 bytes
   Disk identifier: 0x0008aec7
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1   *        2048    16449535     8223744    b  W95 FAT32

Install MBR

   # dd if=/usr/share/syslinux/mbr.bin of=/dev/sdb
   0+1 records in
   0+1 records out
   440 bytes (440 B) copied, 0.0190098 s, 23.1 kB/s

Make fat32 filesystem

   # mkdosfs -F32 /dev/sdb1 
   mkdosfs 3.0.9 (31 Jan 2010)

Download Gentoo LiveCD iso

Go to http://www.gentoo.org/main/en/where.xml ; pick mirror you like and download install-amd64-minimal-[0-9]*.iso, for example - this one:

   wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-iso/install-amd64-minimal-20120621.iso

Mount gentoo LiveCD

   # mkdir -p /mnt/cdrom/
   # mount -o loop install-amd64-minimal-20120621.iso /mnt/cdrom/
   mount: warning: /mnt/cdrom/ seems to be mounted read-only.

Mount usb stick

   # mkdir -p /mnt/usb/
   # mount /dev/sdb1 /mnt/usb/

Copy contents of LiveCD to usb stick

   # cp -aR /mnt/cdrom/* /mnt/usb/
   # mv /mnt/usb/isolinux/* /mnt/usb
   # mv /mnt/usb/isolinux.cfg /mnt/usb/syslinux.cfg
   # rm -rf /mnt/usb/isolinux*
   # mv /mnt/usb/memtest86 /mnt/usb/memtest
   # sed -i \
        -e "s:cdroot:cdroot slowusb:" \
        -e "s:kernel memtest86:kernel memtest:" \
        /mnt/usb/syslinux.cfg

Unmount medias

   # umount /mnt/cdrom/
   # umount /mnt/usb/

Install syslinux

   note guide # syslinux /dev/sdb1

Boot from USB-stick

If in trouble, try following gentoo handbook

Install Funtoo

Once booted, do this:

Partition hard drive

   # gdisk -l /dev/sda  
   GPT fdisk (gdisk) version 0.8.4
   
   Partition table scan:
     MBR: protective
     BSD: not present
     APM: not present
     GPT: present
   
   Found valid GPT with protective MBR; using GPT.
   Disk /dev/sda: 1953525168 sectors, 931.5 GiB
   Logical sector size: 512 bytes
   Disk identifier (GUID): DDF92951-E1C2-4138-B01C-71051D08D1FB
   Partition table holds up to 128 entries
   First usable sector is 34, last usable sector is 1953525134
   Partitions will be aligned on 2048-sector boundaries
   Total free space is 2014 sectors (1007.0 KiB)
   
   Number  Start (sector)    End (sector)  Size       Code  Name
      1            2048           67583   32.0 MiB    EF02  BIOS boot partition
      2           67584         1091583   500.0 MiB   8300  Linux filesystem
      3         1091584        17868799   8.0 GiB     8200  Linux swap
      4        17868800      1953525134   923.0 GiB   8E00  Linux LVM

Setup LVM

   # pvcreate /dev/sda4 
     Writing physical volume data to disk "/dev/sda4"
     Physical volume "/dev/sda4" successfully created
   # vgcreate data /dev/sda4 
     Volume group "data" successfully created
   # lvcreate -L40G -n root data
     Logical volume "root" created
   # lvcreate -L40G -n winxp data
     Logical volume "winxp" created

Setup Filesystems

   # mkfs.ext3 /dev/sda2
   # mkswap /dev/sda3 
   # mkfs.ext4 /dev/data/root 

Mount filesystems

   # mkdir /mnt/funtoo
   # mount /dev/data/root /mnt/funtoo/
   # mkdir /mnt/funtoo/boot
   # mount /dev/sda2 /mnt/funtoo/boot/
   # cd /mnt/funtoo/

Setup networking inside of LiveUSB OS

   # /etc/init.d/dhcpcd start

Download Funtoo stage3 distribution

   # wget http://ftp.osuosl.org/pub/funtoo/funtoo-current/x86-64bit/corei7/stage3-current.tar.xz

Unpack Funtoo distribution

   # tar xJpf stage3-current.tar.xz

Chrooting

   # mount --bind /dev/ /mnt/funtoo/dev/
   # mount --bind /proc/ /mnt/funtoo/proc 
   # cp /etc/resolv.conf /mnt/funtoo/etc/
   # chroot /mnt/funtoo/

Syncing

   # emerge --sync

Configure Funtoo

/etc/fstab

   # <fs>			<mountpoint>	<type>		<opts>		<dump/pass>
   
   /dev/sda2		/boot		ext3		defaults	1 2
   /dev/sda3		none		swap		sw		0 0
   /dev/data/root		/		ext4		noatime		0 1

/etc/localtime

   # cp /usr/share/zoneinfo/Europe/Moscow /etc/localtime 

/etc/make.conf

   ACCEPT_KEYWORDS="~amd64"
   CHOST="x86_64-pc-linux-gnu"
   CFLAGS="-march=corei7 -O2 -pipe"
   CXXFLAGS="-march=corei7 -O2 -pipe"
   SYNC="git://github.com/funtoo/ports-2012.git"
   
   USE="X xinerama"
   
   VIDEO_CARDS="nvidia"
   
   MAKEOPTS="-j10"

Install system-wide services

   # echo "sys-boot/boot-update" >> /etc/portage/package.unmask
   # echo "sys-devel/automake" >> /etc/portage/package.unmask
   # emerge -av eix lvm2 grub bridge-utils vixie-cron syslog-ng logrotate =sys-boot/boot-update-1.6.2
   # eix-update
   # rc-update add lvm default
   # rc-update add vixie-cron default
   # rc-update add syslog-ng default
   # rc-update add sshd default
   # rc-update add dhcpcd default
   # passwd

Install Kernel

   # emerge -av gentoo-sources 

Configure kernel

   # cd /usr/src/linux
   # make menuconfig

   General setup  --->
      <*> Kernel .config support
         [*]   Enable access to .config through /proc/config.gz
   
   Bus options (PCI etc.)  --->
      <*> PCI Stub driver
   
   Processor type and features  ---> 
      [*] Paravirtualized guest support  ---> 
         [*]   Xen guest support
   
   [*] Networking support  --->
      Networking options  --->
         <*> 802.1d Ethernet Bridging    
   
   Device Drivers  ---> 
      [*] Block devices (NEW)  --->
         <*>   Xen virtual block device support
         <*>   Xen block-device backend driver
   
   Device Drivers  ---> 
      [*] Network device support  --->
         [*]   Ethernet driver support (NEW)  ---> 
            [*]   Intel devices (NEW)
            < >     Intel(R) PRO/100+ support
            < >     Intel(R) PRO/1000 Gigabit Ethernet support
            <*>     Intel(R) PRO/1000 PCI-Express Gigabit Ethernet support
            < >     Intel(R) 82575/82576 PCI-Express Gigabit Ethernet support
            < >     Intel(R) 82576 Virtual Function Ethernet support
            < >     Intel(R) PRO/10GbE support
            < >     Intel(R) 10GbE PCI Express adapters support
            < >     Intel(R) 82599 Virtual Function Ethernet support
            [ ]     Intel (82586/82593/82596) devices
            <*>     NetXen Multi port (1/10) Gigabit Ethernet NIC
   
   Device Drivers  ---> 
      [*] Network device support  --->
         <*>   Universal TUN/TAP device driver support 
         <*>   Xen network device frontend driver
         <*>   Xen backend network device
        
   Device Drivers  ---> 
      Xen driver support  ---> 
         [*] Xen memory balloon driver (NEW) 
         [*]   Scrub pages before returning them to system (NEW) 
         <*> Xen /dev/xen/evtchn device (NEW) 
         [*] Backend driver support (NEW) 
         <*> Xen filesystem (NEW) 
         [*]   Create compatibility mount point /proc/xen (NEW) 
         [*] Create xen entries under /sys/hypervisor (NEW) 
         <*> userspace grant access device driver (NEW) 
         <*> User-space grant reference allocator driver (NEW) 
         <*> xen platform pci device driver (NEW)
         <*> Xen ACPI processor
   
   File systems  --->
      <*> Ext3 journalling file system support
      <*> The Extended 4 (ext4) filesystem
      [*]   Use ext4 for ext2/ext3 file systems (NEW)
      [*]   Ext4 extended attributes (NEW)

   # make -j 10
   # make install
   # make modules_install

Install initramfs

   # mkdir /root/initramfs
   # cd /root/initramfs/
   # git clone git://github.com/slashbeast/better-initramfs.git
   # cd better-initramfs
   # bootstrap/bootstrap-all
   # make prepare
   # make image 
   # cp output/initramfs.cpio.gz /boot/

Configure bootloader

   # grub-install --no-floppy /dev/sda
   

/etc/boot.cfg

   boot {
       generate grub
       default "Funtoo Linux XEN"  
       timeout 3 
   }
   
   "Funtoo Linux" {
       kernel vmlinuz[-v]
       initrd /initramfs.cpio.gz
       params += root=/dev/data/root lvm
   }
   
   "Funtoo Linux XEN" {
       type xen
       xenkernel xen.gz
       kernel vmlinuz[-v]
       initrd /initramfs.cpio.gz
       params += root=/dev/data/root lvm
   }

Update

   # boot-update

Booting in new system

When booting, select regular (non-XEN) kernel at the first time. It is because we don't have XEN userspace tools to operate with yet.

   # reboot

Install xen 4.2-unsable

Get current Nvidia flash ROM

   # wget http://www.davidgis.fr/download/nvflash_5.100.1_usb.iso.tar.bz2
   # tar xvf nvflash_5.100.1_usb.iso.tar.bz2
   # dd if=nvflash_5.100.1_usb.iso of=/dev/sdb
   
   # reboot

Boot from USB stick

   C:\> nvflash.exe --save vgabios.rom

Boot back to the system

Choose non-xen kernel again

Install XEN 4.2

   # emerge mercurial yajl dev86 bin86 iasl
   # rev=25099; hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/ xen-unstable.hg-rev-${rev}
   # cd xen-unstable.hg-rev-25099/tools/
   # ./configure
   # sed '/Werror/d' -i tests/mce-test/tools/Makefile
   # make FLEX=flex -j 10
   # make clean
   # cd ../
   # wget http://www.davidgis.fr/download/xen-4.2_rev24798_gfx-passthrough-patchs.tar.bz2
   # tar -xvf xen-4.2_rev24798_gfx-passthrough-patchs.tar.bz2
   # for file in xen-4.2_rev24798_gfx-passthrough-patchs/*; do patch -p1 < $file; done

Now the tricky part. You have to tune one of asl-files to support your very own VGA card memory ranges. It goes like there:

   # lspci | grep VGA
   01:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce 9800 GT] (rev a2)
   # dmesg | grep 01:00.0 | grep mem | head -n3
   [    0.281769] pci 0000:01:00.0: reg 10: [mem 0xf6000000-0xf6ffffff]
   [    0.281776] pci 0000:01:00.0: reg 14: [mem 0xe0000000-0xefffffff 64bit pref]
   [    0.281783] pci 0000:01:00.0: reg 1c: [mem 0xf4000000-0xf5ffffff 64bit]

These three ranges goes to tools/firmware/hvmloader/acpi/dsdt.asl by formula

   MaxRange - MinRange + 1

(consider using some hex calculator (emerge wcalc), for example - `wcalc -h`)

In this example this would be:

• 0xf6ffffff - 0xf6000000 + 1 = 0x01000000
• 0xefffffff - 0xe0000000 + 1 = 0x10000000
• 0xf5ffffff - 0xf4000000 + 1 = 0x02000000

Then edit tools/firmware/hvmloader/acpi/dsdt.asl file as follows:

                   /* reserve MMIO BARs of gfx for 1:1 mapping */
                   DWordMemory(
                       ResourceProducer, PosDecode, MinFixed, MaxFixed,
                       Cacheable, ReadWrite,
                       0x00000000,
                       0xf6000000,
                       0xf6ffffff,
                       0x00000000,
                       0x01000000)
   
                   DWordMemory(
                       ResourceProducer, PosDecode, MinFixed, MaxFixed,
                       NonCacheable, ReadWrite,
                       0x00000000,
                       0xe0000000,
                       0xefffffff,
                       0x00000000,
                       0x10000000)
   
                   DWordMemory(
                       ResourceProducer, PosDecode, MinFixed, MaxFixed,
                       Cacheable, ReadWrite,
                       0x00000000,
                       0xf4000000,
                       0xf5ffffff,
                       0x00000000,
                       0x02000000)

Remember USB stick with vgabios.rom on it we dumped before? Now we need it.

   # mkdir /mnt/usb
   # mount /dev/sdb1 /mnt/usb/
   # cp /mnt/usb/VGABIOS.ROM tools/firmware/vgabios/vgabios-pt.bin

Thats it. Now simply (and dirty) make && make install all needed components:

   # make xen && make tools && make stubdom    
   # make install-xen && make install-tools PYTHON_PREFIX_ARG= && make install-stubdom

Add following script to /etc/init.d/xen

   #!/sbin/runscript
   # Copyright 1999-2011 Gentoo Foundation
   # Distributed under the terms of the GNU General Public License v2
   # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/files/sshd.rc6.3,v 1.2 2011/09/14 21:46:19 polynomial-c Exp $
       
   start() {
       /etc/init.d/xencommons start
       /etc/init.d/xend start
   }
   
   stop() {
       /etc/init.d/xend stop
       /etc/init.d/xencommons stop
   }

   # chmod +x /etc/init.d/xen
   # rc-update add xen default

Now you can safely choose XEN kernel - it should be bootable.

   # reboot

• install winxp
• integreate