Event Channel Internals

General idea

Event channels are the basic primitive provided by Xen for event notifications. An event is the Xen equivalent of a hardware interrupt. They essentially store one bit of information, the event of interest is signalled by transitioning this bit from 0 to 1.

Notifications are received by a guest via an upcall from Xen, indicating when an event arrives (setting the bit). Further notifications are masked until the bit is cleared again (therefore, guests must check the value of the bit after re-enabling event delivery to ensure no missed notifications).

Event notifications can be masked by setting a flag; this is equivalent to disabling interrupts and can be used to ensure atomicity of certain operations in the guest kernel.

There are four kinds of events which can be mapped into event channels in Linux:

• Inter-domain notifications. This includes all the virtual device events, since they're driven by front-ends in another domain (typically dom0).
• VIRQs, typically used for timers. These are per-cpu events.
• IPIs.
• PIRQs - Hardware interrupts.

Events are stored in a bitmap shared between guest and hypervisor. Several tricks such as N-level search path and per-cpu mask are used to speed up search process.

All versions of Xen support the 2-level event channel implementation. All recent versions of Xen (4.4 and later) also support the FIFO channel implementation.

During the 4.3 cycle, a 3-level event channel implementation was discussed but did not go forward. The FIFO based implementation addresses the scaling concerns that motivated the 3-level implementation. The rationale behind extended event channel ABI is to scale better, as nowadays hardware is getting more and more powerful and can support more DomUs running at the same time. Another approach to address scalability is to disaggregate Dom0 functionalities to separate domains. Here is a write-up done at that time (2013):

2-level event channel ABI

This is the de-factor implementation for event channel mechanism. It is supported across all Xen releases for compatibility.

32 bit domain supports up to 1024 event channels, 64 bit domain supports up to 4096 event channels.

This implementation utilizes 2-level search path to speed up searching. The first level is a bitset of words which contain pending event bits. The second level is a bitset of pending events themselves.

FIFO-based event channel ABI

A new FIFO-based event channel is implemented in Xen 4.4 and later. This new ABI utilizes lock-less queues between Xen and guests to achieve FIFO property. Concurrent access in Xen to the event queues is protected by spin lock. This approach has nice features such as event priority, flexible configuration for number of event channels.

Here is a historical discussion thread.