Compressing VM Exports
Note: this has been implemented and is documented in the XenAPI reference
Compressing VM exports
There are two kinds of VM exports in XCP:
- metadata only
- full VM: metadata plus disk blocks
An export is a tar format archive where the metadata is always the first file and has name "ova.xml". The metadata is stored in XML format and includes a bunch of existing version numbers:
The CLI commands involved are:
xe vm-export vm=... filename=... xe vm-import filename=...
Exports and imports can also be processed by HTTP GET and PUT.
The "full VM" exports can be very large since they contain raw disk blocks. Their size makes them difficult to store and distribute over the network.
The disk blocks in a "full VM" export often contain very little user data and compress very well with gzip. I propose to:
- add a --compress option to the CLI which will cause VM exports to be compressed via gzip
- on import, auto-detect whether a VM requires decompression and decompress it first.
- Read the first 512 bytes (a tar header length)
- Check if the tar header is well-formed and refers to an ova.xml file
- f so: consider as uncompressed
- If not: open a pipe to zcat, in a background thread write the first 512 bytes and then the rest of the file. Read from the output of zcat and look for a tar header
Version number discussion
The export format has two layers:
- the outermost ("envelope"?) layer which currently contains: tar with an xml file containing version numbers
- the innermost ("payload"?) layer which currently contains: the rest of the VM metadata and optional disk blocks
The changes anticipated in the future all concern the "innermost" layer and hence are covered by the existing version numbers. For example I anticipate
- adding new VM metadata
- changing the disk block format e.g. to vhd
Adding another version number to cover the outermost layer seems like overkill because:
- it is simple enough to tell the difference between tar and tar.gz automatically
- no future changes to this layer are anticipated
- if we change the outermost layer then we'll simply extend the auto-detect code.
Old (uncompressed) exports will continue to import on new servers, thanks to the auto-detection.
New compressed exports will not import on old hosts. They will fail with a generic "IMPORT_ERROR" exception (probably containing the text 'Failure "int_of_string"'). A simple workaround is to uncompress the file first and then import it.