Add Support for Datastreams #1092
Conversation
danielmitterdorfer
added
:Load Driver
esrally/track/params.py
Outdated
| @@ -344,10 +408,14 @@ def __init__(self, track, params, **kwargs): | |||
| super().__init__(track, params, **kwargs) | |||
| if len(track.indices) == 1: | |||
| default_index = track.indices[0].name | |||
| elif len(track.data_streams) == 1: | |||
| default_index = track.data_streams[0].name | |||
There was a problem hiding this comment.
I think a more generic name, such as default_target would be less confusing. This applies similarly also to other parts of this PR such as ForceMergeParamSource.
There was a problem hiding this comment.
Agreed. Should we also prevent the use of index when data streams are in use and vise versa? This would be consistent with our proposal for the track i.e. data-streams and indices cannot co-exist.
|
@dliappis ready for review. Assuming this is good, i'd like to hold off docs + int tests until we have Index Templates. These are really needed to exploit this feature properly and any int tests wouldn't be representative. |
Unrelated changes are formatting on long lines which my editor had configured. I'll revert but i do think we need a max line length and wrap appropriately. |
Addresses #1053
This PR adds support for data streams. I've tried to address this in a backwards compatible way i.e. it should in no way impact current use of indices. The PR itself is of limited use until we have #1054 as we would be entirely reliant on dynamic mappings otherwise. Assuming this PR is good, i will address this functionality next and then document together ie. i propose we don't document until #1054 is addressed.
This PR supports creating and deleting data streams through new explicit operations. Furthermore, we attempt to provide the same support for data streams as indices. More specifically,
data-streamsis a top level key in the track andtarget-data-streamcan be specified on a corpus documents. These definitions will be re-usable in:In all cases we prefer the specification of an index over a data stream if both are specified e.g. we don't error on the specification of both.
Finally, there are cases where internally a data stream value is used to set an
indexconstruct. This can appear confusing but the ES api doesn't extinguish also e.g. a bulk operation would contain{"create": {"_index": "logs-k8-application.log-default"}}when indexing into a data streamI've held off implementing statistics. I suspect we will want to implement the stats endpoint for datastreams as a telemetry device.
Finally, PR doesn't include integration tests. We can either wait for #1054 or add limited now e.g. creation and deletion but not indexing.