Proposal: Disk import/export

Proposal to improve disk import/export

There are various ways to move a disk into or out of a system via the XenAPI, both "official" (ie. via an API) and "unofficial" (ie. by working around missing APIs). Many of the "official" ways to move disks around use non-standard and poorly-documented protocols and formats, which inhibits interoperability with other services such as CloudStack and OpenStack. The "unofficial" ways are often risky since they potentially conflict with running operations (e.g. vhd coalesce).

Disk import/export is a fundamental ability of a virtualisation platform, and good support for it is demanded by all users: traditional server virt users need off-site incremental backup; cloud orchestration layers need to deploy images from central repositories; everyone benefits from being able to quickly move gold image disk volumes around.

This document starts with an overview of what we currently have; followed by an analysis of use-cases we'd like to support (or improve our support for); followed by a set of principles to govern our designs; and finally by a proposed set of APIs and CLI commands.

What do we currently have?

The following sections describe the current mechanisms, who is known to use them, advantages and disadvantages of each.

HTTP PUT raw disk contents

1. (Optionall) The client calls XenAPI Task.create if it wants to be able to tell if the operation succeeded or failed. This is only optional to allow quick uploads entirely over HTTP.
2. The client sends an authenticated HTTP PUT to /import_raw_vdi?vdi=(ref or uuid). If a the client called Task.create it can add a "task_id=task reference" query parameter or cookie. Authentication can be either
   1. basic auth (convenient for commandline usage with wget/curl)
   2. a pre-created session_id query parameter or cookie header
3. The server
   1. if the authentication cannot be verified then returns an HTTP 403 forbidden
   2. if the VDI query parameter (or cookie) is not present then an uncaught exception causes the server to return HTTP 500 Internal server error
   3. if the VDI ref or uuid doesn't exist then an uncaught exception causes the server to return HTTP 500 Internal server error
   4. if the VDI is only accessible on a remote host then an HTTP 302 redirect is returned using the Host.address field (this won't work if a client is behind a NAT)
   5. If the client requests any HTTP transfer-encoding, the server returns HTTP 403 forbidden
   6. If all looks ok, the server returns HTTP 200 OK with headers
      1. content-type: application/octet-stream
      2. connection: close
      3. task-id: XenAPI task ID
4. The client
   1. writes the unencoded disk contents
   2. closes the connection at the end
5. (Optionally) The client waits until Task.get_finished is true, and then checks the value of Task.get_status to find out whether the operation succeeded or failed. Reasons the task may fail include
   1. insufficient space in the VDI for the data provided
   2. I/O errors writing to the backend substrate
6. If the client called Task.create then it now calls Task.destroy

This can all be driven through the xe CLI command:

 xe vdi-import uuid=<target VDI> filename=<raw image>

This command takes care of authentication, Task handling and error reporting.

Advantages:

1. simple, can be driven entirely through a wget/curl invocation (without error reporting) or via the CLI

Disadvantages:

1. only supports raw format images
2. no support for uploading deltas
3. requires you to pre-create an image of the right size
4. doesn't allow an aborted transfer to be resumed
5. no progress monitoring

Principles

This section proposes some guiding principles to help guide the shape of the API.

Possibilities include:

1. use of a nominated standard format for all exported and imported disk data (e.g. vhd now; qcow2 or vhdx later?). Note this doesn't say anything about the runtime format of the disk.
2. always supporting resumption of interrupted transfers, since vhds are large