Rally tables should always output the same rows

[dakrone]

When running rally benchmarks, a nice summary table is generated at the end, however, between subsequent runs, different rows can be generated, for example, one might have:

|   All |         50th percentile latency |         index-append |    7794.54 |     ms |
|   All |         90th percentile latency |         index-append |    12663.5 |     ms |
|   All |         99th percentile latency |         index-append |    17143.2 |     ms |
|   All |        100th percentile latency |         index-append |    20297.1 |     ms |

And the next time you run it, it'll output:

|   All |         50th percentile latency |         index-append |    7794.54 |     ms |
|   All |         90th percentile latency |         index-append |    12663.5 |     ms |
|   All |         99th percentile latency |         index-append |    17143.2 |     ms |
|   All |       99.9th percentile latency |         index-append |    19243.2 |     ms |
|   All |        100th percentile latency |         index-append |    20297.1 |     ms |

I like to put colums side-by-side to do change comparisons, but it is more difficult if they don't have consistent rows

[danielmitterdorfer]

That's a bit tricky because Rally will show you as many percentiles as possible but will also not provide you with a false sense of accuracy: You will simply not get a 99.999th percentile from Rally if you have just 100 samples. It looks as if you have a different number of samples in both races and that's the reason for the difference in output.

About which output do you care?

• The command line output at the end of a race?
• The comparison output of two races (i.e. esrally compare --baseline=... --contender=...)?
• The output of the report file?

[dakrone]

About which output do you care?

• The command line output at the end of a race?

This is usually what I grab, because it happens to be a tabular format I can easily import into the tool I use.

• The comparison output of two races (i.e. esrally compare --baseline=... --contender=...)?

Almost never this, since when I'm comparing things there's usually setup involved so I have to run two separate tests, rather than use the baseline/contender settings.

• The output of the report file?

To be 100% honest, I haven't actually ever looked at the report file, since the tabular command line output works for me, that's what I've been using.

[danielmitterdorfer]

Thanks for your feedback. As discussed offline, it would then make sense to allow users to force a consistent output. I'd default to the current (a bit more condensed) output though.

[danielmitterdorfer]

There will be a new command line option, called --report-values with the following possible options:

• available (default): Will only show the available lines.
• all-percentiles: as above but will show all possible values for percentile metrics (latency and service time).
• all: Will show all lines regardless whether the corresponding value is defined.

I have to admit, I am not totally happy with the current name and open for suggestions. ;)

[dakrone]

Sounds good, thanks @danielmitterdorfer

[danielmitterdorfer]

Now that I slept on it, I decided to change the parameter's name to --show-in-report, e.g.: --show-in-report="all-percentiles". This will land soon in master.