Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow explicit specification of the steps Rally is executing for a benchmark #61

Closed
danielmitterdorfer opened this issue Feb 6, 2016 · 0 comments
Labels
:Usability Makes Rally easier to use
Milestone

Comments

@danielmitterdorfer
Copy link
Member

This is an enabler ticket to use Rally in more situations. Currently, we assume a very specific order of steps:

  1. Source code is checked out
  2. Issue gradle clean, then gradle assemble
  3. Copy the resulting ZIP, adapt the config and the log file
  4. Launch the benchmark candidate
  5. Run benchmark
  6. Print command line report

The only "flexibility" so far is the --skip-build flag, which skips the second step allowing to iterate faster. With this ticket, we introduce the concept of a "pipeline", similar to build pipelines. With this new structure we want to enable use cases like:

  • Downloading a prebuilt distribution from the official repositories instead of building Elasticsearch from source (changing steps 1 - 3)
  • Leverage other tools for provisioning (e.g. Ansible) (changing step 3)
  • Just benchmark a prepared Elasticsearch instance that is already running (eliminating steps 1 - 4) although this will not allow us to attach telemetry (or only in a very limited fashion) and we lose effectively all control over the benchmark candidate risking correctness issues.
  • etc. etc.

In the scope of this ticket, we will also adapt the command line flags:

  • Old: esrally --skip-build (will be gone then)
  • New: esrally --pipeline from-sources-skip-build

We'll strive to keep all defaults, so if users just invoke esrally it will still do what they expect (namely: run all the steps above).

We will also overhaul the list subcommand. Currently, there is a bolted-on esrally list-telemetry. In the scope of this ticket, we will generalize the list command to allow:

  • esrally list telemetry (behaves similar to the previous esrally list-telemetry)
  • esrally list pipelines (prints all available pipelines)
  • esrally list tracks (prints all available tracks)

In summary, we provide a uniform list subcommand interface to users.

To be clear, we just introduce the pipeline concept in this ticket but we will not (yet) support the use cases above. We will also not allow users to specify their own pipelines. This does not make sense with the current API and we should defer this decision until the API is much more mature.

@danielmitterdorfer danielmitterdorfer added the :Usability Makes Rally easier to use label Feb 6, 2016
@danielmitterdorfer danielmitterdorfer self-assigned this Feb 6, 2016
@danielmitterdorfer danielmitterdorfer added this to the 0.1.1 milestone Feb 6, 2016
@danielmitterdorfer danielmitterdorfer modified the milestones: 0.1.0, 0.1.1 Feb 11, 2016
@danielmitterdorfer danielmitterdorfer removed their assignment Feb 19, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Usability Makes Rally easier to use
Projects
None yet
Development

No branches or pull requests

1 participant