Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REQUEST]: Discourage cloning index templates - offer easier option to customize settings if not using namespaces + Warning/Disclaimer page #1395

Open
lucabelluccini opened this issue Oct 17, 2024 · 5 comments · May be fixed by #1408

Comments

@lucabelluccini
Copy link
Contributor

Description

We're aware of the issues cloning index templates of Fleet integration leads to.

We need to attempt to warn users / push users to not use index template cloning or edit the index & component templates.

The page https://www.elastic.co/guide/en/fleet/current/data-streams-ilm-tutorial.html should be reviewed to display 2 options:

  • one in case the user wants to edit the ILM policy (this would involve creating a @custom component template to override the ILM policy or edit the default ILM policy coming with the integration) - point to https://www.elastic.co/guide/en/fleet/current/data-streams.html#data-streams-ilm
  • one in case the user wants to have a namespace specific ILM policy (this will require index template cloning)
    • for the namespace specific customizations, we need to point to a new "warnings/disclaimer" page

Further enhance https://www.elastic.co/guide/en/fleet/current/integrations-assets-best-practices.html

  • Never edit managed assets of Fleet (e.g. the Index template, the @package component templates and/or ingest pipelines bundled with integrations). Any change will be overwritten upon upgrade/install.

  • When the user creates a @custom component template for a package integration, while allowed and legit, the user can incur in some risks which might prevent the correct ingestion of data.

    • It can lead to broken indexing, data loss, broken integration packages upgrades.
      • Example: the @package component template of the integration switches from a "normal" data stream to TSDB or LogsDB and some settings/mappings introduced by the user are not compatible with such indexing modes.
      • Example: the user overrides the type of an ECS field from keyword to text - this might prevent the aggregation on such field on built-in dashboards.
      • We warn about this on this page
  • When users create a @custom ingest pipeline as we document here, Elastic is not responsible of the correct indexing/behavior of the ingest pipeline. It becomes a custom processing of the data and it's user responsibility to test it.

    • We might add the warning here
  • When a user clones the index template of an integration package, the user can incur in the risk of not picking up index template major changes upon upgrades. The "structure" of the index template will be "frozen" at the moment it was cloned.

  • Cloning index templates to add/inject additional component templates cannot be tested by Elastic, so we cannot guarantee it can work in future release and it can break at any minor release.

Resources

Referenced above

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @lucabelluccini

I can help with a draft.

@lucabelluccini
Copy link
Contributor Author

FYI @jsoriano / @kilfoyle

Some of those warnings are already in the docs - but we need to start to better frame the message and raise more warnings.

@kilfoyle kilfoyle linked a pull request Oct 24, 2024 that will close this issue
@kilfoyle
Copy link
Contributor

@lucabelluccini to get us started, I've opened this PR to add the new warnings you suggest for the Best practices for integration assets page. Please let me know whatever you think we should change.

For the first part, about updating the Customize data retention policies tutorial with two different options, I haven't worked much with component templates, etc., and would really appreciate if you or anyone can suggest exactly what steps should be added/changed.

@lucabelluccini
Copy link
Contributor Author

For the first part, about updating the Customize data retention policies tutorial with two different options, I haven't worked much with component templates, etc., and would really appreciate if you or anyone can suggest exactly what steps should be added/changed.

Hello @kilfoyle

The key is the current guide we have at https://www.elastic.co/guide/en/fleet/current/data-streams-ilm-tutorial.html is focusing on customizing the ILM policy for a specific namespace.

We need to discourage it as we want the majority of users want to customize the ILM policy for all namespaces. And customizing the ILM policy for all namespaces should not involve cloning the index template. It's a pitfall a lot of users are following generating very painful recovery procedures.

@lucabelluccini
Copy link
Contributor Author

If you need assistance please book a slot next week so we can go over it together.

@kilfoyle
Copy link
Contributor

Thanks @lucabelluccini!

I updated my pull request so that it adds a prominent warning in that "Customize data retention policies" tutorial. If you can have a look at my comment here, and let me know if we need to rephrase things, I'd appreciate it. Whatever changes you have in mind I'm happy to add in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants