Contributions

Michal Orzel (Arm)

Luca Fancellu (Arm)

Bertrand Marquis (Arm)

Introduction

This document aims to evaluate Xen hypervisor real time capacities by conducting an investigation to assess and analyze interrupt forwarding path inside Xen. The next goal is to try to compute a WCET (Worst Case Execution Time) for the most standard use case which is handling an interrupt in a real time guest.

Investigation is done on an example of generating a virtual timer interrupt from within a Zephyr guest.

Configuration

The analysis is based on Xen 4.16.0 (tag RELEASE-4.16.0).

Compiler

gcc version 9.4.0 (target: aarch64-linux-gnu)

Target

FVP_Base_RevC-2xAEMvA

Fast Models [11.14.21 (Mar 16 2021)]

Xen Configuration process

$ ./configure
–prefix=/usr –exec_prefix=/usr –bindir=/usr/bin –sbindir=/usr/sbin
–libexecdir=/usr/lib –datadir=/usr/share –sysconfdir=/etc
–sharedstatedir=/com –localstatedir=/var –libdir=/usr/lib
–includedir=/usr/include –oldincludedir=/usr/include
–infodir=/usr/share/info –mandir=/usr/share/man –disable-stubdom
–disable-ioemu-stubdom –disable-pv-grub –disable-xenstore-stubdom
–disable-rombios –disable-ocamltools –disable-qemu-traditional
–disable-pvshim –disable-static –disable-seabios
–disable-sdl –disable-systemd –disable-xsmpolicy

Xen Build process

$ make dist-xen && make dist-tools

Kconfig options that must be disabled

CONFIG_NEW_VGIC

If this option is set, the new vGIC will be used and enter_hypervisor_from_guest will additionally call the functions: vtimer_update_irqs and vcpu_update_evtchn_irq. This will result in executing more instructions. Also CONFIG_NEW_VGIC is not a security supported option.

CONFIG_IOREQ_SERVER

If this option is set, check_for_vcpu_work function will additionally call vcpu_ioreq_handle_completion which will result in more instructions and additional loops.

CONFIG_DEBUG_LOCK_PROFILE

If this option is set, we can see how often locks are taken and blocked using serial console or xenlockprof tool. This will result in more instructions executed.

CONFIG_DEBUG_LOCKS

This option enables debugging features of lock handling. Additional checks will be performed when acquiring and releasing locks which results in more instructions executed.

CONFIG_HAS_ITS

This option enables GICv3 ITS MSI controller support. ITS should be avoided as it can generate calls to the RCU system in Xen. Also this option is currently in unsupported state.

CONFIG_XSM

XSM code is based on RCU which could cause cross core IPI to be generated.

Additional recommendations for the real time guests

do not specify vpl011=“sbsa_uart” in the guest config file
do not access virtual distributor/re-distributor
other guests should not generate SGI to real time guest
real-time guest should be allowed to make use of maximum 16 interrupts

Runtime Configuration

Target is running using 2 CPUs.

dom0 is running using single vCPU assigned to pCPU 0.

Guest (Zephyr) is running using single vCPU assigned to pCPU 1.

	Scheduler	pCPU no.	vCPU no.
dom0	credit2	0	0
guest(Zephyr)	null	1	0

Interrupt Forwarding Path Analysis

About

In this analysis, the number of assembly instructions is taken to act as a benchmark/performance value.

Process of tracing included 1000 iterations. During each iteration a guest was setting up a virtual timer to fire an interrupt immediately. For each iteration the number of assembly instructions, needed to handle a vtimer interrupt, executed in EL2 was measured starting from a context switch from EL1 (guest) to EL2 (Xen) and ending with a switch back to EL1.

With the configuration described in the previous chapter, each iteration resulted in the same number of assembly instructions being 1090. The generated trace file was divided into blocks and analyzed in the next chapter.

A graph showing the complete interrupt execution flow:

Blocks Analysis

Saving state on entry to hypervisor

Loops	Code	No. of ASM ins.	Spinlocks
No	ASM	35 (fixed)	No

guest_irq -> guest_vector;
guest_vector -> entry;
entry -> entry_guest;

Consuming pending SErrors generated by the guest if any

Loops	Code	No. of ASM ins.	Spinlocks
No	ASM	10 (fixed)	No

Executed by default unless “serrors=panic” cmd line option is set. In a real-time system it is recommended to use “serrors=panic” so that a guest’s serror will cause the whole system to crash.

entry -> check_pending_guest_serror;

Actions to be done after entering Xen and before unmasking interrupts

Loops	Code	No. of ASM ins.	Spinlocks
No	C	7 (fixed)	No

entry -> enter_hypervisor_from_guest_preirq;
enter_hypervisor_from_guest_preirq -> needs_ssbd_flip -> check_workaround_ssbd;

Actions to be done after entering Xen and before handling any request

Loops	Code	No. of ASM ins.	Spinlocks
Yes	C	304 (not fixed)	Yes

Loops: Iterating through LR registers Max. no. of iterations: The architecture limits the number of List registers to 16.
Instructions: The number of instructions depends mainly on number of LR registers. In our case there is only one LR to be cleared which results in a block containing 304 instructions. The worst case is clearing all 16 LR registers. GICH_VTR.ListRegs defines the number of implemented LR registers.

entry -> enter_hypervisor_from_guest;
enter_hypervisor_from_guest -> vgic_sync_from_lrs;
vgic_sync_from_lrs -> gic_get_nr_lrs;
vgic_sync_from_lrs -> gicv3_hcr_status;
/* spinlock save : arch.vgic.lock */
vgic_sync_from_lrs -> _spin_lock_irqsave;
/* Iterating through each LR - start */
vgic_sync_from_lrs -> find_next_bit;
vgic_sync_from_lrs -> gic_update_one_lr;
gic_update_one_lr -> gicv3_read_lr;
gicv3_read_lr -> gicv3_ich_read_lr;
gic_update_one_lr -> irq_to_pending;
gic_update_one_lr -> int_test_and_clear_bit;
gic_update_one_lr -> int_clear_bit;
gic_update_one_lr -> test_bit;
/* Remove from inflight list */
gic_update_one_lr -> list_del_init;
/* Iterating through each LR - end */
/* spinlock restore : arch.vgic.lock */
vgic_sync_from_lrs -> _spin_unlock_irqrestore;

GIC IRQ handler entry

Loops	Code	No. of ASM ins.	Spinlocks
Yes	C	144 (not fixed)	Yes

Loops: Iterating through pending IRQs The maximum number of IRQs is limited in GICv3 to 1024. However the max. no. of interrupt sources is device specific and it is usually discovered either from DT or bus specific mechanisms.

entry -> do_trap_irq;
/* Accept an interrupt from the GIC and dispatch its handler */
do_trap_irq -> gic_interrupt;
gic_interrupt -> gicv3_read_irq;
/* do_IRQ is called in case of PPI or SPI interrupt */
gic_interrupt -> do_IRQ;
do_IRQ -> irq_to_desc;
/* spinlock lock: desc->lock */
do_IRQ -> _spin_lock;
do_IRQ -> gicv3_irq_ack;
do_IRQ -> test_bit;
/* Setting IRQ status to _IRQ_INPROGRESS */
do_IRQ -> int_set_bit;
/* spinlock unlock: desc->lock */
do_IRQ -> _spin_unlock_irq;
/* Call vtimer IRQ hanlder */
do_IRQ -> vtimer_interrupt;

Virtual Timer IRQ handler

Loops	Code	No. of ASM ins.	Spinlocks
No	C	360 (fixed)	Yes

/* Inject interrupt into the guest */
vtimer_interrupt -> vgic_inject_irq;
/* spinlock save: arch.vgic.lock */
vgic_inject_irq -> _spin_lock_irqsave;
/* Set IRQ to pending */
vgic_inject_irq -> irq_to_pending;
/* Check if vcpu is offline */
vgic_inject_irq -> test_bit;
/* Set IRQ status to GIC_IRQ_GUEST_QUEUED */
vgic_inject_irq -> int_set_bit;
vgic_inject_irq -> vgic_get_virq_priority;
vgic_get_virq_priority -> vgic_rank_irq -> vgic_get_rank;
/* Raise IRQ */
vgic_inject_irq -> gic_raise_guest_irq;
gic_raise_guest_irq -> gic_get_nr_lrs;
gic_raise_guest_irq -> irq_to_pending;
/* Find unused LR to insert IRQ into */
gic_raise_guest_irq -> gic_find_unused_lr;
gic_find_unused_lr -> gic_get_nr_lrs;
gic_find_unused_lr -> test_bit;
gic_find_unused_lr -> find_next_zero_bit;
gic_raise_guest_irq -> int_set_bit;
gic_raise_guest_irq -> gic_set_lr;
gic_set_lr -> int_clear_bit;
gic_set_lr -> gicv3_update_lr;
gicv3_update_lr -> gicv3_ich_write_lr;
gic_set_lr -> int_set_bit;
gic_set_lr -> int_clear_bit;
/* Add IRQ to inflight list */
vgic_inject_irq -> list_add_tail;
/* spinlock restore: arch.vgic.lock */
vgic_inject_irq -> _spin_unlock_irqrestore;
/* Wake up vcpu if it is blocked - not needed in our case */
vgic_inject_irq -> vcpu_kick;
vcpu_kick -> vcpu_unblock;
vcpu_unblock -> int_test_and_clear_bit;

GIC IRQ handler exit

Loops	Code	No. of ASM ins.	Spinlocks
No	C	97 (fixed)	Yes

/* spinlock lock: desc->lock */
do_IRQ -> _spin_lock_irq;
/* Clear bit _IRQ_INPROGRESS of IRQ status */
do_IRQ -> int_clear_bit;
do_IRQ -> gicv3_host_irq_end;
/* Lower the priority and deactivate the IRQ */
gicv3_host_irq_end -> gicv3_eoi_irq;
gicv3_host_irq_end -> gicv3_dir_irq;
/* spinlock unlock: desc->lock */
do_IRQ -> _spin_unlock;
/* Check if there are any pending interrupts */
gic_interrupt -> gicv3_read_irq;

Actions to be done before entering the guest

Loops	Code	No. of ASM ins.	Spinlocks
Yes	C	100 (not fixed)	Yes

Loops: Iterating through pending work for vcpu before it can be safely resumed Max. no. of iterations: Only 1 iteration possible, provided CONFIG_IOREQ_SERVER is disabled.

Iterating through pending softIRQs Max. no. of iterations: Number of iterations depends on a number of softIRQs which is dependent on a lot of factors (mostly when using RCU). As long as the real-time core is not doing any hypercalls, none should be generated.

guest_vector -> exit;
exit -> leave_hypervisor_to_guest;
/* Process pending work for the vCPU */
leave_hypervisor_to_guest -> check_for_vcpu_work;
/* Call softIRQs if pending */
leave_hypervisor_to_guest -> check_for_pcpu_work;
/* Synchronise vGIC state with the GIC hardware */
leave_hypervisor_to_guest -> vgic_sync_to_lrs;
vgic_sync_to_lrs -> gic_restore_pending_irqs;
gic_restore_pending_irqs -> gic_get_nr_lrs;
/* spinlock lock: arch.vgic.lock */
gic_restore_pending_irqs -> _spin_lock;
gic_restore_pending_irqs -> list_empty;
/* spinlock unlock: arch.vgic.lock */
gic_restore_pending_irqs -> _spin_unlock;
leave_hypervisor_to_guest -> needs_ssbd_flip;
leave_hypervisor_to_guest -> check_workaround_ssbd;

Restoring state on exit from hypervisor

Loops	Code	No. of ASM ins.	Spinlocks
No	ASM	33 (fixed)	No

exit -> exit_guest;
exit -> return_from_trap;

Spinlock Analysis

The goal of this chapter is to analyze the spinlocks inside the interrupt forwarding path that was a subject of the previous chapter.

Interrupt Forwarding Path Analysis showed that there are 2 spinlocks in the path:

arch.vgic.lock from struct vgic_cpu

This spinlock is initialized per vCPU.

desc->lock from struct irq_desc

This spinlock is initialized per IRQ (so 1024 spinlocks).

SPINLOCK: arch.vgic.lock

The following functions make use of locking/unlocking arch.vgic.lock spinlock.

arch/arm/gic-vgic.c

vgic_sync_from_lrs
gic_restore_pending_irqs
vgic_vcpu_pending_irq

arch/arm/vgic-v3-its.c

its_handle_clear
its_handle_inv
its_handle_invall
its_discard_event
its_handle_movi

The above can be discarded provided that CONFIG_HAS_ITS is not set.

arch/arm/vgic-v3.c

__vgic_v3_rdistr_rd_mmio_read
__vgic_v3_rdistr_rd_mmio_write

The above can be triggered only by a guest trying to access virtual redistributor. This is something pointless when running on a single vCPU that is statically assigned to a pCPU so a guest should not do this.

arch/arm/vgic.c

domain_vgic_init
vcpu_vgic_init
vgic_migrate_irq
vgic_disable_irqs
vgic_enable_irqs
vgic_clear_pending_irqs

vgic_migrate_irq is executed if the target vCPU has changed which cannot occur when a guest is running on a single vCPU that is statically assigned to a pCPU.

vgic_disable_irqs and vgic_disable_irqs are executed only if the guest is trying to access the virtual distributor. This is something pointless when running on a single vCPU that is statically assigned to a pCPU so a guest should not do this.

vgic_inject_irq Possible callers:
virt_timer_expired (not possible when using NULL scheduler)
phys_timer_expired (not possible when using NULL scheduler)
vtimer_update_irq (only if CONFIG_NEW_VGIC is set)
vpl011_update_interrupt_status (only if we use vpl011=“sbsa_uart” in guest)
vgic_set_irqs_pending (if guest is accessing virtual distributor)
vgic_to_sgi (if guest is trying to generate SGI; guest should not do this)
vtimer_interrupt
do_IRQ
vgic_vcpu_inject_lpi (only if CONFIG_HAS_ITS is set)
dm_op (when calling hypercall dm_op with XEN_DMOP_set_irq_level)
vcpu_update_evtchn_irq (only if CONFIG_NEW_VGIC is set)

Correlation between structures: vgic_cpu and pending_irq

struct vgic_cpu

inflight_irqs list This list is used to keep track of the IRQs that the vGIC injected into the guest. It is ordered by IRQ priority.
lr_pending list This list is used to queue IRQs (struct pending_irq) that the vGIC tried to inject into the guest, but there was no LR available.

The above lists are per vCPU and initialized in vcpu_vgic_init.

struct pending_irq

inflight list This list is used to append instances of structure pending_irq to the list inflight_irqs from structure vgic_cpu.
lr_queue list This list is used to append instances of structure pending_irq to the list lr_pending from structure vgic_cpu.

The above lists are per IRQ and initialized in vgic_init_pending_irq.

SPINLOCK: desc->lock

The following functions make use of locking/unlocking desc->lock spinlock.

arch/arm/irq.c

do_IRQ
release_irq
setup_irq
route_irq_to_guest
release_guest_irq
irq_set_spi_type
irq_local_set_type

arch/arm/vgic.c

vgic_disable_irqs
vgic_enable_irqs
vgic_set_irqs_pending

common/irq.c

init_one_irq_desc

To achieve the best WCET a configuration in which a guest is running on a single vCPU statically assigned to a pCPU using NULL scheduler shall be used. This way we are avoiding additional locks related to: -migrating vCPUs (thanks to single vCPU/pCPU) -performing timer softirqs (thanks to NULL scheduler)

To limit the worst case scenario, a real-time guest should be allowed to make use of maximum 16 interrupts.

Correlation between IRQ and VGIC spinlocks

Tracing and analysis done in the chapter Interrupt Forwarding Path Analysis showed that:

-> spinlock desc.lock from irq_desc is only taken/released from: - do_IRQ Sections protected by this spinlock are marked blue in the Interrupt Execution flow graph.

-> spinlock arch.vgic.lock from vgic_cpu is only taken/released from: - vgic_sync_from_lrs (section 1) - vgic_inject_irq (section 2) - gic_restore_pending_irqs (section 3) Sections protected by this spinlocks are marked red in the Interrupt Execution flow graph.

-> both of these spinlocks are never locked together. If one spinlock is taken the other is already released.

IPI Analysis

About

This analysis identifies all possibles cases where Xen can generate an IPI to another core. The goal is to define what could influence the response time of a real-time guest running in the configuration described in the chapter Configuration.

The analysis starts from the actual function generating an IPI and looks for the reverse calling tree to find out all conditions of generation.

On arm there is one single function used to generate IPIs: gic_hw_ops->send_SGI.

The function is implemented by the interrupt controller driver and is using the hardware to generate an interrupt to one or several cores of the system.

A graph showing all the places from which an IPI can be triggered:

File:Rta-ipi-call-graph.jpg

IPI call graph

Callers of gic_hw_ops->send_SGI:

The function is called by: - send_SGI_mask - send_SGI_self - send_SGI_allbutself

send_SGI_mask is called by:
- send_SGI_one
- smp_send_event_check_mask
- smp_send_call_function_mask
send_SGI_self is never called
send_SGI_allbutself is called by:
- cpu_up_send_sgi

Level 2 callers:

cpu_up_send_sgi is called by:
- exynos5_cpu_up: This function is only called during boot by the exynos5 platform to signal to the main CPU that a secondary core is up and alive. Conclusion: Xen Init.
- plat->cpu_up: This function is only called during boot by platforms to signal to the main CPU that a secondary core is up and alive. Conclusion: Xen Init.
smp_send_event_check_mask is called by:
- vcpu_kick: This function is waking up/signaling another vcpu of a guest and is only actually firing an interrupt if the destination vcpu of the guest is different than the current vcpu.
  - vgic_queue_irq_unlock: Used only in new VGIC Conclusion: Do not use new VGIC.
  - vgic_kick_vcpus: Used only in new VGIC Conclusion: Do not use new VGIC.
  - vgic_inject_irq:
    - vtimer: raise interrupt on core itself Conclusion: real time process could be impacted by timer on the core but this is the same behaviour as normal hardware.
    - vpl011: generate serial interrupt to guest Conclusion: do not use vpl011.
    - XEN_DMOP_set_irq_level: hypercall to set an interrupt level. Conclusion: Control domain should not use this to generate interrupts to the real time guest.
    - vcpu_mark_events_pending: part of xen even channels Conclusion: do not use event channels between real time guest and other guests on the system.
    - vcpu_update_evtchn_irq: same as previous
    - do_IRQ: this can be triggered when an hardware interrupt is to be forwarded to the guest. This can in fact be a wanted scenario for a real time guest wanting to handle an hardware interrupt.
cpumask_raise_softirq is called by:
- credit2 __runq_tickle: This function is raising an interrupt if the next task to schedule is modified on a core. Conclusion: Do not use credit2 for real-time guests. Conclusion: Should not happen if real-time core is not using credit2.
- nullsched null_unit_wake: Wake up cpu when a task is scheduled on it. Conclusion: Single core guest alone on its CPU
- sched/core schedule: Deschedule current domain and reschedule a new one. TODO: Generic code, can this happen if using null scheduler ?
- rcupdate rcu_barrier: This function is called by:
  - cpu_hotplug_begin: This function is called by:
    - cpu_down: Function to stop a core. Conclusion: Xen halt/reboot.
    - cpu_up: Function called when a core is started. Conclusion: Xen Init.
- rcupdate force_quiescent_state: This function is called when the callback queue on a cpu is above 1000 (parameter defined in the module). This function is called by:
  - call_rcu: This function is called by:
    - free_percpu_area: This function is called when a CPU is DEAD. Conclusion: Xen halt/reboot.
    - rcu_node_free is called by:
      - radix_tree_node_free is called by:
        radix_tree_shrink is called by:
        radix_tree_delete: Deletes a node from the radix tree.
        
        radix_tree_delete is called by:
        p2m_mem_access_radix_set: Conclusion: It happens only if mem access is active. Don’t use mem access for any domain.
        __p2m_set_entry: Insert (or remove) an entry in the p2m.
        p2m_set_entry
        p2m_set_mem_access: Set access type for a region of pfns
        p2m_mem_access_check
        do_trap_stage2_abort_guest
        do_trap_guest_sync (Instruction/data abort from lower EL, permission fault)
        guest_sync (Synchronous exception from 64bit guest)
        
        guest_sync_compat (Synchronous exception from 64bit guest)
        
        mem_access_memop
        do_memory_op (XENMEM_access_op)
        compat_memory_op (only x86)
        
        do_trap_hypercall
        do_trap_guest_sync
        guest_sync (Hypercall from 64bit guest)
        
        guest_sync_compat (Hypercall from 32bit guest)
        
        p2m_insert_mapping
        map_regions_p2mt
        map_range_to_domain (Called at init)
        handle_device
        handle_node
        prepare_dtb_hwdom
        construct_dom0
        create_dom0*
        
        map_device_children
        handle_device*
        
        pci_host_bridge_mappings
        construct_dom0*
        
        handle_passthrough_prop
        handle_prop_pfdt
        scan_pfdt_node
        domain_handle_dtb_bootmodule
        prepare_dtb_domU
        construct_domU
        create_domUs*
        
        try_map_mmio
        do_trap_stage2_abort_guest Conclusion: This branch is taken only on a permission fault access, done by an HW domain when Xen boots using ACPI and the HW domain must have access to the region.
        
        acpi_map_other_tables (only ACPI at init)
        prepare_acpi*
        
        prepare_acpi (only ACPI at init)
        construct_dom0*
        
        map_mmio_regions
        vgic_v2_domain_init (Xen init)
        domain_vgic_init
        arch_domain_create
        domain_create*
        
        gicv2_map_hwdown_extra_mappings (Dom0 construction)
        gic_map_hwdom_extra_mappings
        construct_dom0*
        
        vgic_v2_map_resources (domain creation)
        domain_vgic_init*
        
        do_domctl (XEN_DOMCTL_memory_mapping)
        do_trap_hypercall
        do_trap_guest_sync
        guest_sync (Hypercall from 64bit guest)
        
        guest_sync_compat (Hypercall from 32bit guest)
        
        map_range
        rangeset_consume_ranges
        vpci_process_pending
        check_for_vcpu_work
        leave_hypervisor_to_guest
        
        apply_map (Called only on Xen boot [system_state < SYS_STATE_active])
        modify_bars
        cmd_write
        vpci_write_helper
        vpci_write
        vpci_ecam_write
        vpci_mmcfg_write
        handle_write
        try_handle_mmio
        do_trap_stage2_abort_guest
        do_trap_guest_sync (data abort from lower EL)
        guest_sync (Synchronous exception from 64bit guest)
        
        guest_sync_compat (Synchronous exception from 64bit guest)
        
        rom_write
        vpci_write_helper
        vpci_write
        vpci_ecam_write
        vpci_mmcfg_write
        handle_write
        try_handle_mmio
        do_trap_stage2_abort_guest
        do_trap_guest_sync (data abort from lower EL)
        guest_sync (Synchronous exception from 64bit guest)
        
        guest_sync_compat (Synchronous exception from 64bit guest)
        
        init_bars
        vpci_add_handlers
        pci_add_device
        pci_physdev_op
        do_physdev_op
        do_trap_hypercall
        do_trap_guest_sync
        guest_sync (Hypercall from 64bit guest)
        
        guest_sync_compat (Hypercall from 32bit guest)
        
        pci_add_device
        
        setup_one_hwdom_device
        _setup_hwdom_pci_devices
        setup_hwdom_pci_devices (called only on x86)
        
        init_bars
        vpci_add_handlers
        pci_add_device
        pci_physdev_op
        do_physdev_op (PHYSDEVOP_pci_device_add)
        do_trap_hypercall
        do_trap_guest_sync
        guest_sync (Hypercall from 64bit guest)
        
        guest_sync_compat (Hypercall from 32bit guest)
        
        pci_add_device*
        
        setup_one_hwdom_device
        _setup_hwdom_pci_devices
        setup_hwdom_pci_devices (called only on x86)
        
        map_dev_mmio_region
        xenmem_add_to_physmap_one
        xenmem_add_to_physmap
        do_memory_op* (XENMEM_add_to_physmap)
        
        xenmem_add_to_physmap_batch
        do_memory_op* (XENMEM_add_to_physmap_batch)
        
        guest_physmap_add_entry
        xenmem_add_to_physmap_one*
        
        create_grant_host_mapping
        map_grant_ref
        gnttab_map_grant_ref
        do_grant_table_op (GNTTABOP_map_grant_ref)
        compat_grant_table_op (only x86)
        
        do_trap_hypercall
        do_trap_guest_sync
        guest_sync (Hypercall from 64bit guest)
        
        guest_sync_compat (Hypercall from 32bit guest)
        
        set_foreign_p2m_entry
        acquire_resource
        do_memory_op* (XENMEM_acquire_resource)
        
        arm_iommu_map_page
        iommu_map
        iommu_legacy_map
        map_grant_ref*
        
        unmap_common
        unmap_grant_ref
        gnttab_unmap_grant_ref
        do_grant_table_op* (GNTTABOP_unmap_grant_ref)
        
        unmap_and_replace
        gnttab_unmap_and_replace
        do_grant_table_op* (GNTTABOP_unmap_and_replace)
        
        p2m_remove_mapping
        unmap_regions_p2mt Conclusion: Not called anywhere
        
        unmap_mmio_regions
        do_domctl* (XEN_DOMCTL_memory_mapping)
        
        map_range*
        
        guest_physmap_remove_page
        replace_grant_host_mapping
        map_grant_ref*
        
        unmap_common*
        
        gnttab_transfer
        do_grant_table_op* (GNTTABOP_transfer)
        
        guest_remove_page
        decrease_reservation
        do_memory_op* (XENMEM_decrease_reservation)
        
        memory_exchange
        do_memory_op* (XENMEM_exchange) Conclusion: Do not use memory_op hypercall on any guest using XENMEM_exchange as argument.
        
        do_memory_op* (XENMEM_remove_from_physmap) Conclusion: Do not use memory_op hypercall on any guest using XENMEM_remove_from_physmap as argument.
        
        gnttab_unpopulate_status_frames
        gnttab_set_version
        do_grant_table_op (GNTTABOP_set_version)
        
        gnttab_map_frame
        xenmem_add_to_physmap_one*
        
        relinquish_p2m_mapping
        domain_relinquish_resources
        domain_kill
        do_domctl* (XEN_DOMCTL_destroydomain) Conclusion: Domain destroy
        
        its_discard_event Conclusion: ITS can’t be used by any guest as it generates RCU subsystem calls that may lead to cross core IPI handling.
        its_unmap_device
        its_handle_mapd
        vgic_its_handle_cmds
        vgic_v3_its_mmio_write
        handle_write
        try_handle_mmio (Deliberate action to write on a certain part of memory)
        do_trap_stage2_abort_guest
        do_trap_guest_sync (data abort from lower EL)
        guest_sync (Synchronous exception from 64bit guest)
        
        guest_sync_compat (Synchronous exception from 64bit guest)
        
        its_handle_discard
        vgic_its_handle_cmds*
        
        map_grant_ref* Conclusion: Do not use grant table hypercall with GNTTABOP_map_grant_ref operation on any guest.
        
        unmap_common* Conclusion: Do not use grant table hypercall with GNTTABOP_unmap_grant_ref operation on any guest.
        
        gnttab_release_mappings Conclusion: called only on domain destroy.
        domain_teardown
        domain_create* (on error path)
        
        domain_kill*
        
        radix_tree_node_destroy
        radix_tree_destroy
        p2m_teardown Conclusion: Only on domain destroy.
        arch_domain_destroy
        domain_create* (on error path)
        
        complete_domain_destroy
        
        vgic_v3_domain_free Conclusion: Only on domain destroy.
        domain_vgic_free
        arch_domain_destroy*
        
        domain_create Conclusion: Only on domain creation.
        create_domUs
        start_xen (init of Xen)
        
        create_dom0
        start_xen (init of Xen)
        
        setup_system_domains
        start_xen (init of Xen)
        
        do_domctl* (XEN_DOMCTL_createdomain)
        
        scheduler_init
        init_idle_domain
        start_xen (init of Xen)
        
        complete_domain_destroy (callback of call_rcu, issued by domain_destroy)
        
        gnttab_release_mappings*
    - domain_destroy: This is called whenever a domain is destroyed to wake up the other cores of the domain to stop them. Conclusion: Domain create/destroy.
    - free_pirq_struct is called by:
      - domain_create: Called if domain creation failed. Conclusion: Domain create/destroy.
      - complete_domain_destroy: Called when a domain is destroyed. Conclusion: Domain create/destroy.
      - pirq_get_info: only on error case
        evtchn_bind_pirq: Called during EVTCHNOP_bind_pirq Conclusion: Do not create event channels
    - cpu_schedule_down is called by:
      - sched_rm_cpu:
        cpu_schedule_callback: in CPU_DEAD case only to remove a cpu from the scheduler. Conclusion: Xen halt/reboot.
        
        cpupool_cpu_remove_forced: only called during resume if a cpu available before is not coming up anymore. Conclusion: Xen halt/reboot.
      - cpu_schedule_callback: in CPU_UP_CANCELLED which is only if something goes wrong during cpu bring up. Conclusion: Xen Init.
    - schedule_cpu_add: Add a cpu not assigned to any pool to a pool. Conclusion: Cpupool with fixed hardware core.
    - rcu_barrier_action is called by:
      - rcu_process_callbacks
        __do_softirq*
    - avc_node_delete (xsm): do not use XSM
    - avc_node_replace (xsm): do not use XSM
- cpu_raise_softirq_batch_finish: This is never called on arm.
smp_send_call_function_mask is called by:
- on_selected_cpus is called by:
  - scrub_heap_pages: This function is scrubbing the memory to remove any old data from it during init. Conclusion: Xen Init.
  - cpu_down: Stops a xen cpu. Conclusion: Xen halt/reboot.
  - smp_call_function is called by:
    - setup_virt_paging: Setup Xen mappings in the mmu during boot. Conclusion: Xen Init.
    - machine_halt: Stops the computer. Conclusion: Xen halt/reboot.
    - machine_restart: Restart the computer: Conclusion: Xen halt/reboot.
    - read_clocks: Get clock information from other core to generate some debug information in the debug console. Conclusion: Debug console usage.
    - gdb_smp_pause: Debugger pausing all cores. Conclusion: Debugging support.
smp_send_state_dump: Request through the console to dump cpu state. Conclusion: Debug console usage.

RCU callback stack

Any callback added to the list by a call to call_rcu()
- rcu_do_batch
  - __rcu_process_callbacks
    - rcu_process_callbacks
      - __do_softirq
        process_pending_softirqs
        do_idle
        idle_loop
        continue_new_vcpu (PC address for a new vcpu)
        
        startup_cpu_idle_loop
        start_secondary (Xen init)
        
        init_done (Xen init)
        
        __cpu_up
        cpu_up
        start_xen
        
        core_parking_helper (only on x86)
        
        enable_nonboot_cpus (only on x86)
        
        __cpu_die
        cpu_down
        core_parking_helper (only on x86)
        
        enable_nonboot_cpus (only on x86)
        
        disable_nonboot_cpus (only on x86)
        
        dump_domains (only for debug purpose)
        do_tasklet
        idle_loop*
        
        run_all_nonirq_keyhandlers (only for debug purpose)
        do_tasklet*
        
        livepatch_printall (only for debug purpose)
        do_tasklet*
        
        scrub_heap_pages
        heap_init_late
        start_xen
        
        rcu_barrier_action (rcu barriers are used only on cpu up/down)
        rcu_process_callbacks
        __do_softirq*
        
        rcu_barrier
        cpu_hotplug_begin (only when cpu system config changes)
        cpu_down*
        
        cpu_up*
        
        warning_print
        console_endboot
        start_xen
        
        _setup_hwdom_pci_devices
        setup_hwdom_pci_devices (called only on x86)
        
        apply_map* (Called only on Xen boot [system_state < SYS_STATE_active])
        
        do_softirq
        idle_loop

Conclusions

Cases out of scope

Some parts of Xen are temporary (init, halt, reboot) or simply do not make sense for a real-time usage and can lead to some IPI generation so are out of scope of the analysis:

Xen Init Several cases can lead to a generation of an IPI to another core:
- Secondary core startup signaling to main core that it has started and is alive.
- Scrubbing of guest memory which is signaling other cores to make sure cache is synced on all cores.
- Setup Xen virtual memory (configure and start MMU).
Xen halt/reboot When the complete platform is to be rebooted or stopped, Xen is signaling secondary cores to also stop. Conclusion: Stopping or rebooting the system is out of scope.
Debug console usage Some of the commands available in the debug console are requesting information from other cores and to make sure the information is stable an IPI is sent and some operations are done directly in secondary cores before reporting it to the user. Conclusion: The debug console should be disabled.
Debugging support (ie gdb) To do debugging there is a need to synchronize cores (for example to stop all of them from executing code synchronously). Conclusion: GDB support should be disabled.
Domain create/destroy Whenever a domain is created, cores are assigned and scheduled which might need to signal the other core they will be running on. When it is destroyed the other cores of the guest need to be wake up to properly stop. Conclusion: Guest Init/Destroy phase is out of scope.
Domain configuration Whenever a domain configuration is altered or modified, and in particular the cpupool it is assigned to is modified, an IPI might be generated to inform the affected cores. Conclusion: Guest configuration should not be modified at runtime.
Event channel creation Event channel should not be used between real time domain and others domain.
Hardware interrupts Hardware interrupts handlers must be analysed to derive a worst case execution time and check if it’s feasible with the real time system. This depends on the device being used and because of that it’s out of scope from this analysis.

Restrictions in configuration

Some conditions on which IPI can be generated can be prevented by putting some restrictions on how the system can be configured and the attributes/options of the guest that must be real-time:

Single core guest Several conditions could lead to Xen generating an interrupt to wake up an other vcpu of a guest (injection of an interrupt, guest starting the virtual interrupt controller) and the guest itself if it wants to send an IPI to one of its other cores. Conclusion: Only one core should be assigned to real-time guests.
Cpupool with fixed hardware core In a generic case, Xen can move a guest vcpu between hardware cores depending on the load of the system. As this might alter the guest timings and have other bad side effects (cache or TLB flushes required), a real time guest should have a configuration using cpupools and cores dedicated to it. Conclusion: Real-time guests must have a dedicated cpupool with dedicated cores.
Avoid using ITS Use of ITS can generate calls to the RCU system in Xen, manipulating a radix tree used to store the pending irq and subsequently adding callbacks that will be executed on idle loop or on context switch. The RCU system can raise cross core IPI when there are too many callbacks queued. So don’t use ITS on any guest of the system.
Don’t use PV driver on any guest Use of PV driver needs the usage of grant tables. Hypercalls with GNTTABOP_map_grant_ref and GNTTABOP_unmap_grant_ref operations trigger modifications to a radix tree and subsequently uses the RCU that could lead to cross core IPI.
Don’t use mem access to monitor any guest on the system Use of mem access leads to modifications to a radix tree that subsequently trigger RCU, this could cause cross core IPI to be raised.
XSM must be disabled Compile Xen with CONFIG_XSM not set or start Xen with “flask=disabled” to make Xen use the dummy xsm module. XSM code is based on RCU to protect access and modification of some internal list, RCU could cause cross core IPI to be raised.

Xen on ARM Real Time Analysis

Contributions

Introduction

Configuration

Compiler

Target

Xen Configuration process

Xen Build process

Kconfig options that must be disabled

Additional recommendations for the real time guests

Runtime Configuration

Interrupt Forwarding Path Analysis

About

Blocks Analysis

Saving state on entry to hypervisor

Consuming pending SErrors generated by the guest if any

Actions to be done after entering Xen and before unmasking interrupts

Actions to be done after entering Xen and before handling any request

GIC IRQ handler entry

Virtual Timer IRQ handler

GIC IRQ handler exit

Actions to be done before entering the guest

Restoring state on exit from hypervisor

Spinlock Analysis

SPINLOCK: arch.vgic.lock

Correlation between structures: vgic_cpu and pending_irq

SPINLOCK: desc->lock

Correlation between IRQ and VGIC spinlocks

IPI Analysis

About

Callers of gic_hw_ops->send_SGI:

Level 2 callers:

RCU callback stack

Conclusions

Cases out of scope

Restrictions in configuration

Navigation menu

Search