Improve response checks in bulk API runner

[danielmitterdorfer]

At the moment, Rally checks for bulk errors by inspecting the errors property of the bulk response (and adds it to the request meta-data).

On errors, we should improve the inspection and also report more meta-data (like items with failed shards etc.)

[bleskes]

note that failing replicas will not bubble up as an error. You need to look at the failures section of the _shards header of each item.

[danielmitterdorfer]

Thanks for the pointer. In that case I think we should add this as an option which needs to be enabled by the user in the track.

[bleskes]

maybe we should improve the bulk response to bubble this up. I think it has merit for exactly the reason you think of making it optional - people won't have to parse the response.

[danielmitterdorfer]

maybe we should improve the bulk response to bubble this up.

This would definitely help in this case.

The reason I wanted to make response parsing optional is to avoid adding any bottlenecks in the load test driver. I did not check yet how long it would take to extract these data (for different bulk sizes) and my gut feeling tells me that the overhead is negligible but I'd rather measure it first and base my decision on hard numbers. :)

[danielmitterdorfer]

Rally will return a structure like:

{
  "weight": 5000,
  "unit": "docs",
  "bulk-size": 5000,
  "success": true,
  "success-count": 5000,
  "error-count": 0,
  "ops": {
    "index": {
      "item-count": 5000,
      "created": 5000
    }
  },
  "shards_histogram": [
    {
      "item-count": 5000,
      "shards": {
        "total": 2,
        "successful": 2,
        "failed": 0
      }
    }
  ]
}

when the new parameter detailed-results is set to true in the track specification. If the value is false, it will return instead the following meta-data:

{
  "weight": 5000,
  "unit": "docs",
  "bulk-size": 5000,
  "success": true,
  "success-count": 5000,
  "error-count": 0
}

The default value will be false as this feature adds very significant overhead due to the need to iterate over the complete response structure. The benchmarks (which can be run with make benchmark) indicate a slowdown by a factor greater than 1.000: Whereas we can return within single digit microseconds when detailed-results is false, the call needs several milliseconds when detailed-results is true. Whether this is acceptable needs to be decided on a case-by-case basis (by looking at typical response times of the bulk operation in a benchmark) and to avoid that users run unintentionally into trouble I decided to set it to false by default.

[danielmitterdorfer]

By the way, the pydoc comment has a few more examples that might also be interesting.

[bleskes]

@danielmitterdorfer does it mean you gave up on:

maybe we should improve the bulk response to bubble this up. I think it has merit for exactly the reason you think of making it optional - people won't have to parse the response.

[danielmitterdorfer]

@bleskes I did not gave up but Rally works with ES 1.x, 2.x, 5.x and 6.x and I needed to implement a solution that works for all of them. We can still implement this in ES though. I can create a ticket later.

[bleskes]

@danielmitterdorfer makes total sense. Thanks for explaining.

[danielmitterdorfer]

I've created elastic/elasticsearch#23143 now as a follow-up.