Event Channel Internals

General idea

Event channels are the basic primitive provided by Xen for event notifications. An event is the Xen equivalent of a hardware interrupt. They essentially store one bit of information, the event if interest is signalled by transitioning this bit from 0 to 1.

Notifications are received by a guest via an upcall from Xen, indicating when an event arrives (setting the bit). Further notifications are masked until the bit is cleared again (therefore, guests must check the value of the bit after re-enabling event delivery to ensure no missed notifications).

Event notifications can be masked by setting a flag; this is equivalent to disabling interrupts and can be used to ensure atomicity of certain operations in the guest kernel.

There are four kinds of events which can be mapped into event channels in Linux:

• Inter-domain notifications. This includes all the virtual device events, since they're driven by front-ends in another domain (typically dom0).
• VIRQs, typically used for timers. These are per-cpu events.
• IPIs.
• PIRQs - Hardware interrupts.

Events are stored in a bitmap shared between guest and hypervisor. Several tricks such as N-level search path and per-cpu mask are used to speed up search process.

Xen prior to 4.3 supports 2-level event channel implementation, 3-level event channel implementation is planned for 4.3.

2-level event channel

This is the de-factor implementation for event channel mechanism. It is supported across all Xen releases for compatibility.

32 bit domain supports up to 1024 event channels, 64 bit domain supports up to 4096 event channels.

This implementation utilizes 2-level search path to speed up searching. The first level is a bitset of words which contain pending event bits. The second level is a bitset of pending events themselves.

3-level event channel

This is planned to be introduced in Xen 4.3. However this is supposed to be only used by Dom0 or driver domains as normal DomU will not need so many event channels.

32 bit domain supports up to 32K event channels, 64 bit domain supports up to 256K event channels.

The implementation utilizes 3-level search path to speed up searching. The fist level is a bitset of words which indicates words containing pending bits in the second level. The second level is a bitset of words which indicates words in third level which contain pending events. The third level is a bitset of pending events themselves.

A naive stress test (64 bit Dom0 with 4G RAM) shows that Dom0 gets Out-of-Memory error after allocating ~45K event channels.

Towards N-level event channel and beyond

The rationale behind N-level (N>=3) event channel is to scale better, as nowadays hardware is getting more and more powerful and can support more DomUs running at the same time. Another approach to address scalability is to disaggregate Dom0 functionalities to separate domains.

The interface for registering N-level is defined when implementing 3-level event channel. Please see Xen headers for detail.