Support datastreams in Rally

[gingerwizard]

Propose the following features be core operations in Rally due to the applicability to common workloads:

• Create a datastream - https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-a-data-stream.html#manually-create-a-data-stream
• Delete a datastream - https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-a-data-stream.html#delete-a-data-stream
• Bulk indexing - https://www.elastic.co/guide/en/elasticsearch/reference/master/use-a-data-stream.html#data-streams-bulk-indexing-requests I'm not sure we need changes here as the structure is the same as a bulk request @dliappis but we might want to limit to create actions only.
• Stats API support - Create datastreams-stats telemetry device #1161
• Ability to rollover a datastream - https://www.elastic.co/guide/en/elasticsearch/reference/master/use-a-data-stream.html#manually-roll-over-a-data-stream We can potentially support this through a new runner. Note the event data supports this via this runner. We want this rollover to occur during indexing, however - so we may want to incorporate it into the bulk action itself. To discuss with @dliappis. This is low priority and likely to be used infrequently if we recommend using ILM for rollover. Dropped in support of ILM. Support ILM Policies #1087

Datastreams support a number of other operations e.g. Updating documents by query. I don't believe these will be used regularly in workloads however and can be implemented as custom runners if needed.

[hub-cap]

I have hit an issue on the a track that also needs create-datastream, and would love to see this done as a good first step. Lets chat tomorrow @gingerwizard about timing of this first step.

[gingerwizard]

Code merged. Pending IT and docs.

[gingerwizard]

@danielmitterdorfer we will need data streams stats. My initial instinct is to add as a runner like https://esrally.readthedocs.io/en/stable/track.html?highlight=index%20stats#index-stats? Any other approaches you'd like to consider? Happy to own.

[danielmitterdorfer]

If we want custom metrics I suggest we implement a so-called telemetry device similar to:

rally/esrally/telemetry.py

Line 1243 in 4d4f863

class IndexStats(InternalTelemetryDevice):

.

[gingerwizard]

👍 appreciate the guidance.