Debugging Xen
This page describes a set of things to look at when trying to debug what appears to be errant Xen Behavior. Providing as much information as possible below with bugs reported to the Xen project will make it much easier for developers to figure out what has gone wrong, and fix the issue.
If you run into a problem in Xen, please file a bug in the Xen Bugzilla. You might want to browse the mailing lists and the bugzilla first to verify if it's a new issue. Posting a query on the IRC channels might be a good idea. If it is an open issue, create an account in bugzilla (if you do not have one) and file a new bug. Please be as detailed in your information as possible, and assist the assignees in resolving it. Note that you only need an collected by xen-bugtool, account to file, not to search or query the bugzilla.
Please use xen-bugtool to collect the logs from your system, and attach them to that bug that you've just created. These logs are very useful when it comes to diagnosing any problems.
Sources of Information
Please include the following information in all bugzilla reports (unless completely inappropriate):
- Dom0 Operating System (including distribution, and version)
- Hardware Platform (UP, SMP, number & type of disk/nics)
- Kernel config for dom0 and domU (if modified from default)
- Output from dmesg under dom0
- Output from dmesg under domU
- Output from serial port if Xen oops
- Complete oops message or kernel panic
- System.map if Linux oops/panic for appropriate kernel
- grub entry for domain
The following will be collected by xen-bugtool, if you use it:
- Architecture (x86, x86 w/ pae, x86_64, vt/svm)
- Output from xm dmesg under dom0
- Output of xm info under dom0 (this will include xen revision number)
- Lots of log files
If this is a networking bug, please try to include the following if possible: (before and after failure, if applicable)
- ifconfig -a output
- brctl output (if using bridging)
- netstat -s
- netstat -rn
- your domain configuration file
- your interface configuration mechanism (dhcp, private, ifcfg-* files, etc)
- details on what works/doesn't (can you ping internal, external? is only tcp broken? etc)
- a tcpdump or ethereal trace capture of the connection, or a brief snapshot if too large
Debugging Xenstore Problems
If you suspect that the wrong values are being written to or read from Xenstore, then you may turn on xenstore tracing to help debug the problem:
- Set XENSTORED_TRACE=1 into /etc/{sysconfig,default}/xencommons.
- Reboot.
- xenstored will then write a trace file, giving every read and write on the store. Use xen-bugtool to attach that and all your other logs to a Bugzilla entry.
Some easy steps to enable xend/xenstored tracing
# service xend stop # killall -9 xenstored # export XEND_DEBUG=1 # export XENSTORED_TRACE=1 # /usr/sbin/xend trace_start >/dev/null 2>&1 &
- xend pid file: /var/run/xend.pid
- xend trace file: /var/log/xen/xend.trace
- xend log file (check /etc/xen/xend-config.sxp: (logfile ...) default: /var/log/xen/xend.log
- xend debug log: /var/log/xen/xend-debug.log
- xenstored pid file: /var/run/xenstore.pid
- xenstored trace file: /var/log/xen/xenstored-trace.log
Note: on some system, xend trace doesn't work. As a workaround, you can enable xenstored trace only by applying this patch:
--- /usr/sbin/xend.orig 2008-11-24 16:09:16.000000000 +0800
+++ /usr/sbin/xend 2008-11-20 11:59:20.000000000 +0800
@@ -78,7 +78,8 @@ def check_user():
def start_xenstored():
XENSTORED_TRACE = os.getenv("XENSTORED_TRACE")
cmd = "xenstored --pid-file /var/run/xenstore.pid"
- if XENSTORED_TRACE:
+ #if XENSTORED_TRACE:
+ if True:
cmd += " -T /var/log/xen/xenstored-trace.log"
s,o = commands.getstatusoutput(cmd)
And then:
# service xend stop # killall -9 xenstored # service xend start
Note: in some system, kill xenstored will cause dom0 not work well(no hotplug event generated). You should restart dom0 to resolve this problem.
