Xen Profiling: oprofile and perf

By Konrad Rzeszutek Wilk from this email message

Summary

I've been toying with some perf tools using the v3.5 kernel to get an idea of what is working and what not. The executive summary is that we should write a perf driver to interact with perf Linux subsystem; and also to enhance the perf library to support the --xen extensions (similar to how the --kvm exists). It might also require adding some new hypercalls, though the existing ones (the oprofile variant) could be retooled to use with perf.

perf

What currently exists are two ways of collecting performance data in both dom0 and domU (and by that I also mean PVHVM). The easiest one is the 'perf' toolkit which works nicely - except that it is using a timer mode - so not very accurate. However, if Xen is booted with 'vpmu=1' parameter and perf is used within a PVHVM guest - the data is much more accurate. So what would be good is to implement a non-timer variant (so some model of APIC emulation combined with collecting data in a oprofile ring and avoiding MSRs).

oprofile

The good news is that the old performance system oprofile works with the non-timer variant with some patches. You could also run the oprofile in timer mode and get the same data that the 'perf' does. The patches for the oprofile have been in the past posted by Michael Petullo, and I created a git branch with them with some little changes (basically to make them work with the new kernel - they are still buggy though - rmmoding caused a crash). They are now at:

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/oprofile.v1

I plan to tinker a bit with them so they aren't completely hacked in with #ifdef CONFIG_XEN but I do not have any plans to upstream them. I think the right direction is to make a new shiny perf driver.

Anyhow, if you want to play and get some performance data, the steps are:

• Get the latest kernel from Linus's tree. Merge the above mentioned git branch
• Install oprofile. If you are using Fedora 17, the oprofile RPM already has the Xen enablement patches thanks to William Cohen (see [1]). If you are using a different distro, get oprofile and apply the patch mentioned in the URL.

• Run it. This is what I've been doing:

 opcontrol --start --passive-domains=2,3 
 --passive-images=/mnt/lab/nettest/vmlinux-pvhvm,/mnt/lab/nettest/vmlinux-pvhvm 
 --xen=/data/xen-syms --vmlinux=/data/vmlinux --callgraph=5

A co-worker mentioned that to get the exact data from the guest (which is not what I was using it for), your guest config file must contain this "magic":

 cpuid=['0xa:eax=0x07300403,ebx=0x00000004,ecx=0x00000000,edx=0x00000603'

for HVM guests. For PV guests the data shows up just fine.