Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added Time-series indexing (TSDB) to Integrations Experimental Indexi…
…ng settings (elastic#144974) ## Summary Closes elastic#144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here elastic#144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]>
- Loading branch information