Increased flexibility for termination of parallel tasks

[cdahlqvist]

At the moment is appears that Rally waits for all parallel tasks to complete before continuing. For some workloads, e.g. when one wants to index a specific number of documents and run periodic queries while doing so, I can be difficult to ensure that these parallel tasks terminate at roughly the same time.

It would be useful if it was possible to specify which of the parallel tasks that should be considered for termination, e.g. through a flag on the task. The default for all tasks would be that they are considered and at least one of the tasks need to be considered.

[danielmitterdorfer]

One aspect is missing from the original description (or rather vague): Consider task N is marked as the task that determines termination of a parallel structure and is executed by M number of clients. We will terminate other tasks in the same parallel structure if and only if M clients have successfully finished the execution of task N (i.e. we will not terminate everything as soon as the first client executing task N has finished).

[danielmitterdorfer]

As a concrete example of what will be possible consider this example challenge (it's based on geonames):

{
  "name": "parallel",
  "description": "Parallel test",
  "index-settings": {
    "index.number_of_replicas": 0
  },
  "schedule": [
    {
      "parallel": {
        "completed-by": "index-append",
        "tasks": [
          {
            "operation": "index-append",
            "warmup-time-period": 0,
            "time-period": 45,
            "clients": 8
          },
          {
            "operation": "index-stats",
            "clients": 1,
            "time-period": 120,
            "target-throughput": 10
          },
          {
            "operation": "node-stats",
            "clients": 2,
            "time-period": 120,
            "target-throughput": 10
          }
        ]
      }
    },
    {
      "operation": "force-merge",
      "clients": 1
    },
    {
      "parallel": {
        "completed-by": "default",
        "tasks": [
          {
            "operation": "default",
            "clients": 2,
            "warmup-iterations": 50,
            "iterations": 100,
            "target-throughput": 50
          },
          {
            "operation": "scroll",
            "clients": 2,
            "warmup-iterations": 5,
            "iterations": 500,
            "target-throughput": 25
          }
        ]
      }
    }
  ]
}

Rally will terminate the first structure after roughly 45 seconds (after all 8 clients executing the index-append operation have finished) regardless whether index-stats and node-stats have finished.

Then it will run force-merge with one client.

Finally, it runs the operation default and scroll in parallel. As soon as the operation default has finished, scroll will be completed and the benchmark terminates.

Note that Rally supports only one task in completed-by.

[cdahlqvist]

@danielmitterdorfer Great. I think that should cover what I was planning of using it for.