Support data streams in create-track

[gizas]

• Have you signed the contributor license agreement? Yes

• Have you followed the contributor guidelines?

  • Yes
  • Squashed branch
  • Relevant issue exists here: Support data streams in create-track #1360

• Have you run make check-all successfully?

  • Format and linting fixes applied.
  • I see junit tests failing

• Did you choose a descriptive title and description for your PR? Yes

• (Only for maintainers) Did you apply appropriate labels and a milestone?

Info to validate and run PR:
https://github.com/elastic/observability-dev/issues/2194#issuecomment-1167531749

Error of junit testing:

 except BaseException as e:
>           raise AssertionError("Failed to install Elasticsearch {}.".format(distribution_version), e)
E           AssertionError: ('Failed to install Elasticsearch 7.6.0.', JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))

pick 0338008 Commit to allow data-stream usage for create-track command
# This is a combination of 5 commits.
it/__init__.py:174: AssertionError
--------------------------------------------- generated xml file: /Users/andreasgkizas/elastic/rally/fork/rally/junit-py38-it.xml ---------------------------------------------
=========================================================================== short test summary info ===========================================================================
ERROR it/basic_test.py::test_run_without_arguments[in-memory-it] - AssertionError: ('Failed to install Elasticsearch 7.6.0.', JSONDecodeError('Expecting value: line 1 colum...

• Relates #2194

Additional story created to track creation of test: #1538

[michaelbaamonde]

Thanks for taking this on @gizas! I left some comments after a first pass.

I also think it'd be good to come up with an integration test for this new functionality, similar to this one. I'll give this some more thought come up with some options for seeding Elasticsearch with datastreams in such a test. Conveniently, elastic/rally-tracks#275 is in review and we may be able to use that once it's merged.

[j-bennet]

There seems to be a little bit of a mismatch between what the code in doing in datastreamextract, and what the comments are saying. It would be good to have that clarified.

[gizas]

I also think it'd be good to come up with an integration test for this new functionality, similar to this one. I'll give this some more thought come up with some options for seeding Elasticsearch with datastreams in such a test. Conveniently, elastic/rally-tracks#275 is in review and we may be able to use that once it's merged.

I have added a dummy placeholder for a new test here ba02999
Can we sync on how to ingest datastream inside Elasticsearch?

[gizas]

Story for test created #1538

[michaelbaamonde]

This is looking good to me, but since @pquentin had the most recent feedback, let's make sure he's good with the latest commits before merging. Thanks for all the iteration on this @gizas!

[pquentin]

Thanks for itereating! We're nearly there.

So adding an integration test is tough, but can you please add an unit test for datastreams support? See test_extract_index_create in tests/tracker/index_test.py for the existing index test.

[gizas]

Some more tests to allow hidden indices

❯ esrally create-track  --indices=".ds-metrics-kubernetes-*" 

[ERROR] Cannot create-track. Failed to extract any indices for track!.

❯ esrally create-track --indices=".ds-metrics-kubernetes*" 
Extracting documents for index [.ds-metrics-kubernetes...     1000/1000 docs [100.0% done]
Extracting documents for index [.ds-metrics-kubern...     447468/1522057 docs [29.4% done]

❯ esrally create-track --indices=".ds-logs*" 
Extracting documents for index [.ds-logs-elastic_agent...     1000/1000 docs [100.0% done]
Extracting documents for index [.ds-logs-elastic_ag...     154360/908119 docs [17.0% done]

❯ esrally create-track --track=test  --indices=".ds-logs*"  --data-streams="metrics-*"

usage: esrally create-track [-h] --track TRACK (--indices INDICES | --data-streams DATA_STREAMS) --target-hosts TARGET_HOSTS [--client-options CLIENT_OPTIONS] [--output-path OUTPUT_PATH] [--quiet] [--offline]
esrally create-track: error: argument --data-streams: not allowed with argument --indices

❯ esrally create-track --data-streams="metrics-*" 
[INFO] Connected to Elasticsearch cluster [ha-es-zone-d-0] version [8.3.0-SNAPSHOT].

Extracting documents for index [.ds-metrics-elastic_ag...     1000/1000 docs [100.0% done]
Extracting documents for index [.ds-metrics-elastic_a...     12850/257181 docs [5.0% done]

[gizas]

So adding an integration test is tough, but can you please add an unit test for datastreams support? See test_extract_index_create in tests/tracker/index_test.py for the existing index test.

Added in 39450ec

Also one additional use case added NOT to allow hidden indices when filter is "_all"
Overall hidden indices are supported with --indices and data-streams and blocked when filter is "_all"

❯ esrally create-track --track=test --indices="_all" 

[ERROR] Cannot create-track. Failed to extract any indices for track!.

[pquentin]

Thank you! This looks great now.