relative-time should be reset per task

[russmac]

Rally version (get with esrally --version):
esrally 0.5.3

Invoked command:
esrally --track=logging --pipeline=benchmark-only --target-hosts=${target_hosts}

Configuration file (located in ~/.rally/rally.ini)):

[meta]
config.version = 8

[system]
env.name = rally_zero

[node]
root.dir = /root/.rally/benchmarks

[runtime]
java8.home = /usr/lib/jvm/java-8-openjdk-amd64

[benchmarks]
local.dataset.cache = ${node:root.dir}/data

[reporting]
datastore.type = elasticsearch
datastore.host = localhost
datastore.port = 9200
datastore.secure = False
datastore.user = elastic
datastore.password = *****************************

[tracks]
default.url = https://github.com/elastic/rally-tracks

[defaults]
preserve_benchmark_candidate = False

JVM version:
openjdk-8-jre-headless:amd64 8u121-b13-1~bpo8+1

OS version:
Debian 8.6

Description of the problem including expected versus actual behavior:
relative-time should be relative to each challenge

Steps to reproduce:

1. Execute logging track.
2. View elasticsearch metrics in Kibana, relative-time is relative to the start of entire track not each challenge making combined graphs between trials/races far less useful if not unusable. Its not due to warmup off as the first warmup is also relative to the beginning of the track executing not the challenge.

Describe the feature:
Reset relative-time on each challenge so combined graphs work.

I made some edits, still getting used to the rally lingo

[danielmitterdorfer]

Thanks for the report @russmac! You know, naming is one of the hard problems in computer science and I have yet to come up with a better name for "challenge"... . For each invocation, Rally runs (or rather: races) on one track with one challenge. For example, you want to run on the "logging" track, or the "geonames" track or ... . Then you need to decide what you actually want to do on this track and that's captured by the challenge, i.e. it describes the concrete workload for this track, like indexing or indexing and then executing queries etc.. . And Rally can only execute one challenge per invocation. So I think this is not the problem.

But I think you're referring to another problem (and please correct me if I'm wrong). The default challenge for the "logging" track is "append-no-conflicts" (you can see it by issuing esrally list tracks) and it executes multiple operations:

1. index-append
2. force-merge
3. index-stats
4. node-stats
5. default (a match-all query)
6. term
   ...

I guess the problem is that relative-time is not reset when a new operation starts, so you see for example "term" starting for one execution at say 1200 seconds and for the next execution starting at 1250 seconds. But you'd like that relative-time always starts at zero for each operation individually (which is not the case today - I think - but would make sense to me). Correct?

[russmac]

Hi @danielmitterdorfer , Thanks for the reply and for degarbling my rallyspeak. Your right I had been using "challenge" in place of "operation".

You hit the nail on the head, I think the "operations" should be resetting the relative time so they can be compared side by side in Kibana when I create a visualization for each "operation".

[danielmitterdorfer]

Thanks for the confirmation @russmac. The change makes definitely sense to me.

[danielmitterdorfer]

Just an update on this issue: As Rally gathers not only request metrics (in the load test driver) but also system metrics (on the target host(s)) it is not sufficient to reset a local timer whenever a new step starts. Instead, we need to broadcast the status change also to the target host(s) so relative time is also properly reset for system metrics. We're looking into different approaches how to implement this best.