Nested Virtualization in Xen: Difference between revisions
Line 130: | Line 130: | ||
:Nested fails to work with L1 shadow mode |
:1.Nested fails to work with L1 shadow mode |
||
:Virtual Box fails to boot on top of Xen (L1 panic while booting L2) |
:2.Virtual Box fails to boot on top of Xen (L1 panic while booting L2) |
||
:Hyper-v screen flush issue: |
:3.Hyper-v screen flush issue: |
||
::Current the screen flush is not correct after booting hyper-v. Two workarouds: |
::Current the screen flush is not correct after booting hyper-v. Two workarouds: |
||
::using remote desktop to connect guest |
::using remote desktop to connect guest |
Revision as of 02:59, 13 February 2014
Introduction
As the virtualization becomes popular, there are more and more requirements for nested virtualization, which provides the capability to enable the virtual machines to start second level of virtual machines. For example, with nested virtualization, cloud vendors are able to migrate their own cloud to others without any modification.
Currently, we enabled the nested virtualization on Xen. We can boot up all popular VMMs as normal virtual machine (Xen, KVM, VMware Workstation, VMware ESX, Hyper-V, etc.) and all those VMMs are able to boot its own VM. Besides, with adding several critical nested features, we made a good progress in performance that the second level guest is almost same with first level guest: Performance optimization (virtual EPT, VMCS shadowing, etc.) has boosted Xen nested virtualization performance by 30%-500%.
Abbreviation
Abbreviation | Description |
---|---|
L0 | Xen hypervisor |
L1 | First level VM which is able to start second level VM |
L2 | The second level VM which boot up by L1 |
XEN_on_XEN | L1 XEN based on L0 XEN |
VMware_ESX_on_XEN | L1 VMware ESX Server on L0 XEN |
VMware_Workstaion_on_XEN | L1 VMware Workstation on L0 XEN |
Hyperv_on_XEN | L1 WIN8 with Hyper-v feature enabled on L0 XEN |
XP_mode | L1 WIN7 VirtualPC XP_mode on L0 XEN |
How to use nested
Booting Xen:
- Nested feature is added to Xen long time ago. But it is really ready to use only from last year. So it’s better to use the latest Xen,
git clone git://xenbits.xen.org/xen.git xen.git
- NB: When I write this Wiki, there are still two patches are missing in upstream Xen if you want to boot up VMware. You can get those patches from:
Booting Dom0:
- No special requirement for Dom0. Just use the stable one is enough.
QEMU:
- Both Qemu-xen and Qemu traditional are OK.
L1 configuration:
- To use nested virtualization, you need to add the following two lines in your guest configuration file:
hap=1 nestedhvm=1
- NB: We don’t want to support L1 guest with shadow mode, since the performance is very poor. So I recommend you to enable the EPT(hap = 1) in the L1 guest. Besides, there is a known issue that nested is failing to work with L1 shadow mode.
- To boot L1 VMware-ESX, VMware-Workstation, Win8-hyperv, you needs to mask CPU bit.
cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
Current status
Testing environment:
Version | |
---|---|
L0 XEN | git commit:58f5bcaf05621810f06bf5b3592e2ae87475053d |
L1 XEN | git commit:e423b5cd60ff95ba3680e2e4a8440d4d19b2b13e |
L1 KVM | 3.12.-rc2 |
L1 VMware Workstation | 10.0 |
L1 VMware ESX | 5.1 |
L1 Hyperv | Win8 with hyperv feature enabled |
L1 configuration:
hap=1 nestedhvm=1 cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
Result:
L1\L2 | XP_x86 | XP_x64 | RHEL6U4_x86 | RHEL6U4_x64 | WIN7_x86 | WIN7_x64 | WIN8_x86 | WIN8_x64 |
---|---|---|---|---|---|---|---|---|
XEN | V | V | V | V | V | V | V | V |
KVM | V | V | V | V | V | V | V | V |
VMware-ESX | V | V | V | V | V | V | V | V |
VMware-Workstation | V* | V | V | V | V* | V | V | V |
Hyperv | V | V | V | V | V | V | V | V |
- V means guest boots well
- V* means it works before, but buggy with latest Xen
Know Issues
- 1.Nested fails to work with L1 shadow mode
- 2.Virtual Box fails to boot on top of Xen (L1 panic while booting L2)
- 3.Hyper-v screen flush issue:
- Current the screen flush is not correct after booting hyper-v. Two workarouds:
- using remote desktop to connect guest
- or set full update mode in Qemu.
Not Tested
- L1 stress test, performance test
- L2 stress test performance test
- L2 save restore, live migration
- L1 save restore, live migration