Allow to benchmark multi-machine clusters

[danielmitterdorfer]

Currently, it is only possible to benchmark on a single machine (except for the special case where we run with --pipeline=benchmark-only which (a) puts the burden of provisioning on the user and (b) does not gather system metrics (like CPU usage or index size). Rally should be able to run benchmarks also on clusters across multiple machines.

In this ticket we want to collect the high-level ideas around making this possible. Work should be done in smaller, more focused tickets. At least these areas need to change:

• Provisioning (consider also Evaluate provisioning of benchmark machines with Ansible #42 and Allow setting cluster config #117)
• Gathering metrics (we need to gather them on the machines that are being benchmarked)
• Reporting: We currently report system level metrics only once (e.g. CPU usage) and need to reconsider that (although we will prepare for that case already in Gather benchmark environment data #54)

Note: Distribution of the load generator is handled separately in #257.

[siennathesane]

What's the status of this issue?

[danielmitterdorfer]

It's in the early concept stage. We use Github's milestone feature and it's planned for 0.5.0 (no release date).

The current "workaround" is to use the pipeline "benchmark-only" as I've mentioned in the description of this ticket provided you are able to apply enough load with a single Rally instance. I'd expect that you can apply a bit more load when #108 is ready on which I am currently working on.

[siennathesane]

Gotcha, thanks!

[danielmitterdorfer]

We'll implement this gradually and implement support for single-machine clusters in #184 first. Then we can extend Rally to multi-machine clusters. This will reduce the risk of adding too many changes at once (as we then also need an ability to define cluster topology).

[danielmitterdorfer]

For the first implementation we'll assume that every node has the same configuration (i.e. there is no possibility to define dedicated node roles (master, data, coordinator, ...) and different node configurations). We will allow multiple nodes per host though.

[danielmitterdorfer]

Having Rally set-up multi-node clusters can be done now by specifying the respective nodes with --target-hosts. It is possible to use multiple nodes per host by specifying the same IP/port pair twice.

Examples:

• --target-hosts=127.0.0.1:39200 will set up a single ES node on the local machine which runs on port 39200.
• --target-hosts=127.0.0.1:39200,127.0.0.1:39200 will set up two ES nodes on the local machine.
• --target-hosts=192.168.2.2:9200,192.168.2.3:9200,192.168.2.4:9200 will set up three ES nodes running on the machines with the IPs 192.168.2.2, 192.168.2.3 and 192.168.2.4 on port 9200. Note that in order for this to work, the Rally daemon process must be started on each machine (see docs).