Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tsdb metrics builtin component template #97602

Merged

Conversation

martijnvg
Copy link
Member

@martijnvg martijnvg commented Jul 12, 2023

Fleet is currently hard coded to set index.codec to best_compression (deflate compression). This is good for most data streams, except for data streams were tsdb is enabled. Ideally Fleet doesn't need to set this setting at all and Elasticsearch's default would be good. But unfortunately this isn't the case. It default to default (lz4 - optimised for speed), which in would mean much higher disk space usage. Ideally the default would be default when synthetic source is enabled and otherwise best_compression. Changing this now, would mean a breaking change.

Instead Fleet like to depend on Elasticsearch's internal component templates. To at least abstract some of the internal details away. The metrics-settings is ok for non tsdb, but there is no component template for tsdb metrics. This PR adds this.

Relates to elastic/kibana#160288

@github-actions
Copy link
Contributor

Documentation preview:

@martijnvg martijnvg added :Data Management/Indices APIs APIs to create and manage indices and templates :StorageEngine/TSDB You know, for Metrics >enhancement v8.1.0 and removed v8.10.0 labels Jul 12, 2023
@elasticsearchmachine
Copy link
Collaborator

Hi @martijnvg, I've created a changelog YAML for you.

Copy link
Member

@felixbarny felixbarny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll probably need to update

And add metrics-tsdb-settings.

Otherwise LGTM

@martijnvg martijnvg marked this pull request as ready for review July 13, 2023 08:20
@elasticsearchmachine elasticsearchmachine added Team:Data Management Meta label for data/management team Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Jul 13, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@martijnvg martijnvg added v8.10.0 and removed v8.1.0 labels Jul 13, 2023
@martijnvg
Copy link
Member Author

@elasticmachine update branch

@ruflin
Copy link
Member

ruflin commented Jul 18, 2023

@eyalkoren New built in template here. Can you check that it aligns with the naming convention?

@eyalkoren
Copy link
Contributor

@eyalkoren New built in template here. Can you check that it aligns with the naming convention?

Is there a naming convention already? 🤔

@ruflin
Copy link
Member

ruflin commented Jul 31, 2023

Lets align it with the hopefully soon to be reached conclusion in #96267

@martijnvg martijnvg requested a review from andreidan August 10, 2023 07:15
Copy link
Contributor

@andreidan andreidan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Martijn

"template": {
"settings": {
"index": {
"lifecycle": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a reminder to override this in stateless to use data stream lifecycle (assuming this component template will be used in data streams only)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iirc this will be used with data streams.

@martijnvg martijnvg merged commit 297a013 into elastic:main Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >enhancement :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Data Management Meta label for data/management team v8.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants