Xen Serial Console
Configuring Xen to use and log to a serial console
Serial console is a very good and important tool for debugging Xen and also Linux kernel problems. If your system doesn't start up, the display goes blank, the whole computer crashes during startup or during normal operation, serial console allows you to see and log everything that's happening to a text file and allows you to troubleshoot the issue much more easily. You can also paste the logged messages (as text) for online troubleshooting with other people.
What actually is a serial console? Basicly you configure Xen and/or Linux kernel to write the boot and console messages to a serial port as text. Then you setup a cable (or nowadays IP connection) between the Xen server and for example a laptop. Then you configure the laptop to listen on the serial port, and log all the Xen and Linux dom0 console boot messages to a file giving you a full log of the boot process and system operation.
You can also leave the serial console up and running allowing you to log any errors that might show up later.
If your server is having problems and it crashes, you usually just see an empty VGA console when you go look at it. That doesn't help you at all troubleshooting the actual problem. Serial console to the rescue! Using a serial console you can log the full error and crash messages, and if you set up serial console logging to a file, you'll be able to easily check the logs later when you need them. Messages won't disappear from the serial console logs when the server powers off or reboots.
Components required for the serial console
First of all you need two computers:
1) The actual computer running Xen to debug/troubleshoot. It needs to have a serial port for the serial console. See below for types of supported serial ports. Remember USB-serial dongles are NOT usable for serial console on the computer running Xen.
2) Another computer that views and logs the messages from the computer running Xen. This can be another server, a desktop machine, or a laptop. On this another computer you can use USB-serial adapters.
Types of serial ports on the computer running Xen
There are four common types of serial ports for setting up a serial console for the computer running Xen:
1) The computer running Xen has a built-in rs232 serial port (COM1) on the motherboard, with a DB9 serial cable connector. Most servers have real serial ports on the motherboard, and also older desktop computers and laptops have it. Most current (as of 2010) deskop and laptops don't include a serial port anymore.
2) If your computer running Xen doesn't have a real physical serial port you can add a PCI add-on serial card with a DB9 serial connector. This is a good option for many computers.
3) If you are using a laptop that doesn't have a physical serial port, you can get a ExpressCard serial card, for example "SYBA SD-EXP15005" (http://www.newegg.com/Product/Product.aspx?Item=N82E16839328018&Tpk=SDEXP15005), which works in the same way as PCI serial cards for desktop computers. Also some laptops offer a SOL (Serial Over LAN), usually when the laptop has Intel vPro (AMT) remote management feature built in.
4) Virtual serial port, usually called SOL (Serial Over LAN). Many servers have a SOL serial port on the management processor, or on IPMI card. Basicly SOL looks like a real serial port to the operating system (Xen/Linux), but actually it can be accessed and used over the network using an IP connection, instead of the 'oldskool' method of using a serial cable between the computers. Some new desktop and laptop computers also have a SOL provided by the Intel vPro or AMT (Active Management Technology) management features.
Note that for serial console you CAN'T use USB-serial dongles on the computer running Xen! (usb-serial dongles are not seen as real serial ports by the hardware or Xen, so usb-serial ports are NOT available in the beginning of boot process).
Connection between the computers
You also need a connection between the computers:
1) If you're using a physical serial port on the computer running Xen then you need to have a DB9-DB9 serial cable ("null-modem cable") between the two computers. For example serial cables that HP ships with their Procurve Ethernet Switches can be usually used between the computers for the serial console. The other computer needs to have a physical serial port aswell. If the other computer used for logging the serial console lacks a built-in serial port, you can get a PCI serial card for it, or you can get an USB serial adapter for it. USB serial adapter doesn't work for the computer running Xen (*), but it works for the other computer that is only used to view and log the serial console.
(* unfortunately USB serial adapter is not a "real" serial port, it's not available at boot time so you cannot write boot-time messages to an USB serial adapter).
2) If you're going to use a SOL virtual serial port then the other computer doesn't need a physical serial port, and there's no need for a serial cable between the two computers. Viewing and logging a SOL serial console only requires a network (IP) connection to the management processor of the server running Xen, and possibly a SOL viewer or terminal application on the other computer.
Getting required information to set up the serial console on the computer running Xen
You need to know the following things to get the serial console set up in the computer running Xen:
- IO port of the serial port
- IRQ (interrupt request) of the serial port
- Name of the serial device in Linux (ttyS*)
You can get this information from the operating system. On Linux (dom0, if already running Xen) run "dmesg | grep ttyS". The output should be something like:
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
The example output above shows the computer has two serial ports (COM1 and COM2), called ttyS0 and ttyS1 in Linux. IOport for the first serial port is 0x3f8 and the IRQ is 4, and IOport for the second serial port is 0x2f8 and the IRQ is 3. These are the default (legacy) IOports and IRQs reserved for serial ports on the motherboard.
If you have a PCI serial card, or a SOL device, or an Intel AMT card, you might need to run "lspci -vvv" and find the serial port information from there.
NOTE: With Xen 4.2, you do _not_ have to find the serial information and Xen can find it automatically. You pass the extra argument pci in the com1 line, as so: com1=38400,8n1,pci. If you have an Intel AMT, then the extra argument is amt.
Setting up the serial console in grub.conf
Using Grub legacy:
Here's an example grub1 grub.conf for configuring the serial console with Xen and pv_ops dom0 kernel (for example Debian 6.0 Squeeze uses pvops dom0 kernel):
title Xen 4.0 / dom0 Linux 2.6.32.26 pvops with a serial console root (hd0,0) kernel /xen-4.0.gz dom0_mem=512M loglvl=all guest_loglvl=all com1=38400,8n1 console=com1 module /vmlinuz-2.6.32.26 ro root=/dev/vg00/lv01 console=hvc0 earlyprintk=xen nomodeset module /initrd-2.6.32.26.img
In this example we're configuring Xen to use the serial console on onboard legacy serial port COM1, we set the serial console speed to 38400 bits per second, and settings to 8 databits, no parity and 1 stopbit. If you use the standard serial ports (COM1, COM2) you don't usually need to specify the IOport or the IRQ.
Pv_ops dom0 Linux kernel is configured to use the Xen (hvc0) console. Dom0 Linux kernel console output will go to the serial console through Xen, so both Xen hypervisor and dom0 linux kernel output will go to the same serial console.
Different dom0 Linux kernel versions might require a bit different settings.
grub1 grub.conf example for RHEL5/CentOS5 Xen with Dell DRAC SOL (this example also displays GRUB on the serial console):
serial --unit=0 --speed=57600 terminal --timeout=10 console serial default=0 timeout=5 title Red Hat Enterprise Linux Server (2.6.18-92.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-92.el5 console=tty0 console=com2 com2=57600,8n1 module /vmlinuz-2.6.18-92.el5xen ro root=LABEL=/1 xencons=ttyS1 console=ttyS1 module /initrd-2.6.18-92.el5xen.img
On this system the Dell DRAC SOL serial port is seen as "COM2". It's a standard legacy serial port, so no IOport or IRQ configuration is necessary in grub.conf.
Using Grub 2:
If you run with Debian and Grub 2 (which should be the case if you are running Squeeze), then you should edit /etc/default/grub as below:
GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1,0x3e8,5 console=com1,vga" GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"
The above setup of the serial device parameters are for a Supermicro X8STi-F motherboard, where IPMI SOL is configured for COM3 from the BIOS. Note that you need to use "com1=" option for Xen, even when the SOL is actually COM3. If you specify "com3=" for Xen the serial console won't work for some reason. When you are done with your grub config file, do:
update-grub2
to generate the /boot/grub/grub.cfg
Viewing and logging the serial console on another computer using a serial cable
Connect the serial cable between the computers, prepare the computer running Xen based on the grub.conf examples above, and then configure the other computer.The other computer needs to have the exact same serial port speed, databits, stopbit, parity and flow control settings than the computer running Xen!
If you're using Windows on the other computer to view and log the serial console:
First you need to know which COM port you've connected the serial cable to. Onboard serial ports are usually either COM1 or COM2. USB serial adapters might be what ever COM port, and often you're able to change the assigned USB serial adapter COM port from Windows Device Manager. So first use Windows Device Manager to figure out what serial ports you have and what you're using.
Then you need a terminal emulator program to access the serial port. Windows includes Hyperterminal, but that's not a very good program. You can also use Putty, or Teraterm. There are many other programs for accessing serial ports. Usually any program that can be used for calling with a modem, or to configure an ethernet switch also works for accessing a serial console. Start the terminal emulator program, and configure the correct serial (COM) port, correct speed, correct settings (8n1), and flow control.
If you're using Linux on the other computer to view and log the serial console:
In linux you can get a list of the serial ports by running "dmesg | grep ttyS". USB serial adapters are usually named "ttyUSB*", so you can check for them with "dmesg | grep ttyUSB".
You can use for example "minicom", "kermit" or many other tools to view the serial console. You can start minicom with "minicom -s" to enter the setup mode to configure the serial settings. Another option is to press "Ctrl-A p" when minicom is running to enter the settings dialog. After configuring the settings remember to save them, and then restart minicom. "Ctrl-A x" can be used to quit minicom. Read the minicom man-page for more information about the usage.
You can also use simple "screen" command to view the serial console: "screen /dev/ttyUSB0 115200".
If you're using SOL and ipmitool:
Before connecting, make sure nobody else is using the serial console (only one person at a time can connect). You can disconnect a currently connected session using:
ipmitool -I lanplus -H server-ip-address -U ADMIN sol deactivate
Then connect using:
ipmitool -I lanplus -H server-ip-address -U ADMIN sol activate
If you want to log the output, simply using the "tee" utility is fine:
ipmitool -I lanplus -H server-ip-address -U ADMIN sol activate | tee my-log-file.txt
Starting up the computer running Xen and viewing the serial console
When you have connected the computers with a serial cable, you have prepared grub.conf on the computer running Xen, and configured the terminal emulator on the another computer making sure you have matching speed and settings on both computers, feel free to power on the computer running Xen and select the Xen with a serial console boot entry from grub. Now you should first see Xen boot messages on the serial console followed by dom0 Linux kernel boot messages.
How does the serial console log look like?
Take a look at this example: http://pasik.reaktio.net/xenserialconsolelog.txt
Connecting to SOL virtual serial port over the network
Some servers allow you to telnet or ssh into the management processor (HP iLO, DELL DRAC), and view the SOL serial console from there.
Some Intel vPro or AMT SOL implementations require you to use a special 'SOL remote console utility' to view the serial console. There's for example "amtterm" for Linux. Check the documentation of your server/computer for more information.
Configuring serial console for non-standard PCI serial ports (for example Intel AMT)
If your serial port is on a non-standard IOport, for example 0xe000, you can use a grub1 grub.conf configuration like this:
title Xen 3.4.2 / pv_ops Linux dom0 2.6.31.6 with a serial console root (hd0,0) kernel /xen-3.4.gz dom0_mem=512M loglvl=all guest_loglvl=all sync_console console_to_ring com1=115200,8n1,0xe000,0 console=com1 module /vmlinuz-2.6.31.6 ro root=/dev/vg00/lv01 console=hvc0 earlyprintk=xen nomodeset module /initrd-2.6.31.6.img
Important part being this:
com1=115200,8n1,0xe000,0
"0xe000" is the IOport of the serial port. Last ",0" means what IRQ to use.. some setups requires 0 there instead of the real IRQ. If you have problems, try using either the real IRQ, or then just 0.
NOTE: With Xen 4.2, you do _not_ have to find the serial information and Xen can find it automatically. You pass the extra argument pci in the com1 line, as so: com1=115200,8n1,pci. If you have an Intel AMT, then the extra argument is amt.
No getty (login prompt) on the serial console so I can't login?
You need to configure a getty in /etc/inittab to listen on the console device, or on the serial port. This will enable you to login to Xen dom0 from the serial console.
Example /etc/inittab line to run getty on RHEL5/CentOS5 for COM2 (ttyS1) at speed 57600 bits per second:
s1:2345:respawn:/sbin/agetty ttyS1 57600
Example /etc/inittab line to run agetty on Debian 6.0 using Xen 4.1.1 for hvc0 at speed 38400 bits per second:
s1:2345:respawn:/sbin/agetty 38400 hvc0
Can I get GRUB to show up on the serial console?
Yes, that's possible. Some servers allows you to configure "BIOS redirection", which makes the full boot process starting from BIOS messages to show up on a serial console, including GRUB. If your BIOS doesn't support this, there's also another option. Example grub1 grub.conf:
# Show grub on the Intel AMT serial-over-lan console serial --port=0x3440 --speed=115200 terminal --timeout=10 console serial default=0 timeout=10 title Xen 3.4.2 / pv_ops Linux dom0 2.6.31.6 with a serial console root (hd0,0) kernel /xen-3.4.gz dom0_mem=512M loglvl=all guest_loglvl=all sync_console console_to_ring com1=115200,8n1,0x3440,0 console=com1 module /vmlinuz-2.6.31.6 ro root=/dev/vg00/lv01 console=hvc0 earlyprintk=xen nomodeset module /initrd-2.6.31.6.img
Xen com1= option for non-standard serial ports (IPMI SOL, Intel AMT, PCI serial)
Note that even if your SOL device is, for example, COM3, you still need to specify "com1=<foo> console=com1" options for Xen. If you specify "com3=" the serial console won't work! Remember to list the correct (actual) serial port IOport and IRQ in the Xen "com1=" parameters!
It doesn't work, how to troubleshoot?
- Are you using correct type of serial cable between the computers? You need a "null-modem" serial cable. For example serial cables shipped with HP Ethernet Switches usually work for serial consoles.
- Do you have matching serial speed/parity/databit/stopbit settings on both computers? You need to make sure the settings match.
- Try using different flow control settings.
- If possible, you can run for example "minicom" or "putty" on both computers, and test the serial cable and connection first like that. You should be able to chat over the serial connection, see the text you write on the other computer.
- Did you specify the optional IRQ on the Xen comX= line? Try specifying the correct IRQ. If that doesn't work, then try specifying 0 as the IRQ. Some setups require 0 as IRQ to make it work. Some setups don't require any IOport or IRQ settings on the comX= line.
I'm using IPMI SOL serial console (on a Blade), and pv_ops dom0 kernel seems to hang?
You might need this extra patch: http://lists.xensource.com/archives/html/xen-devel/2010-01/msg00773.html
This patch was committed to xen-unstable on 2010-02-08 so it'll be part of Xen 4.0.0 release. http://xenbits.xen.org/xen-unstable.hg?rev/12fc55dffb6b
I can't use a serial console, are there other methods to capture the boot time messages?
Well.. you could always record a video of the boot process from the console display, using a phone or a real video camera.
Links for more information
AMT-HOWTO: http://manpages.ubuntu.com/manpages/jaunty/man7/amt-howto.7.html
DELL DRAC SOL configuration with RHEL5+Xen: http://blog.coolzero.info/blogs/index.php/2008/06/04/drac-5-rhel-5-2-centos-xen-based-and-ser?blog=5
Using Intel AMT Serial-Over-LAN: http://software.intel.com/en-us/articles/using-intel-amt-serial-over-lan-to-the-fullest/
HP iLO Integrated Lights-Out Virtual Serial Port configuration and operation HOWTO, 4th edition: http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00263709/c00263709.pdf
Linux serial console HOWTO: http://www.vanemery.com/Linux/Serial/serial-console.html
Enabling serial console in OpenSolaris with Xen: http://hub.opensolaris.org/bin/view/Community+Group+xen/serial-console
OpenSUSE How to Capture Xen Hypervisor and Kernel Messages using a Serial Cable:
Novell Configuring a Remote Serial Console for SLES: http://www.novell.com/support/viewContent.do?externalId=3456486&sliceId=1
Ubuntu Serial Console howto: https://help.ubuntu.com/community/SerialConsoleHowto
Ubuntu IPMI SOL usage: https://help.ubuntu.com/community/IPMI
