XCP Archive/XCP diagnostic messages

In future we would like to have a diagnostic console which can help understand the state of an individual host (not pool). We would like to see:

1. instantaneous load information, including
   1. disk and network throughput (bytes per sec)
   2. memory usage
   3. CPU usage
   4. number of messages per second across various internal control interfaces
   5. message latency distribution per internal service
2. the states of "alarms", where an "alarm" is set when one or more of the load metrics crosses some threshold for some period of time. Perhaps we could 3 states and use red/amber/green (people love dashboards with traffic lights)
3. the contents of message queues containing JSON control messages
4. a live stream of messages, filtered with some expression
5. a live stream of logs, filtered with some expression

Ideally the console would be entirely web-based, and the API should be designed to make that easy.

Example messages

The message format is not fixed, it can still be adjusted. Everything here is just an example.

Load information example

The load metrics are grouped by the type of thing they refer to:

$ wget http://server/metrics

{
"Host": {"instances": "/metrics/host", "description": "host"},
"VM":   {"instances": "/metrics/VM", "description": "virtual machine"}
}