Add ability to continuously stream metrics

[cdahlqvist]

Based on tests it looks like all metrics currently are written to the metrics store at the end of the challenge. For long-running benchmarks this is impractical as it gives a potentially vary large amount of data to index at the end and makes it hard to see how the benchmark is doing during the execution.

It would be very useful to be able to stream metrics through bulk requests to the metrics store as the benchmark progresses.

[danielmitterdorfer]

This is a tricky one. The reason why Rally does not stream metrics is to minimize any overhead. For our own long-running benchmarks, we use multiple iterations (--laps). Rally does store metrics between laps. I think we could get down to the operation level (i.e. send metrics after each operation has finished) without too much hassle but streaming them without creating any bottlenecks will be tricky.

[cdahlqvist]

I agree it is a tricky one. Writing metrics directly to a Elasticsearch could make the benchmark execution sensitive to back-pressure applied from the metrics store. What I have done in the past is to stream the records to a file in JSON format as they are generated. This has allowed me to set up filebeat/Logstash to tail this file and index them as quickly as possible. Would it be possible to have 'file' as a third mode in order to add flexibility? This also opens up for post-processing, which would be a useful option.

[danielmitterdorfer]

The work done in #278 should help when implementing this. That change makes it relatively easy to send request and maybe even system metrics once a task has finished (instead of doing all this at the end of a benchmark). However, streaming still remains extremely tricky given that the throughput calculation is done via a sliding window. So I think we will implement this in two steps:

1. Provide metrics after a task has finished
2. "Stream" metrics (probably in micro-batches covering 5 seconds of data; this is how Rally works internally today)

[danielmitterdorfer]

Rally will store request metrics in microbatches every 30 seconds (not configurable). This includes service time, latency and throughput samples and all system metrics in case Rally is not only used as a load generator. All postprocessing is done on the coordinator machine. Thus, we don't influence performance of load generators.