[ILM] Support custom ILM policies

[simitt]

APM Server allows users to configure custom policies per event type since 7.5. Before that, users had to follow a manual ILM setup guide to customize their policies. We have had a couple of user requests to automatically support this.

Beats removed the versions of ILM policies in 7.6 since custom ILM policies have been overwritten on every upgrade, leading to bad surprises and unwanted behavior.

Let's ensure that users don't experience the same pain when switching to managed beats/APM Server.
(1) As long as custom ILM policies are not supported as part of fleet, we have to ensure that users' custom ILM policies are not overwritten on version upgrade.
(2) Add support for custom ILM policies per data stream in packages and Fleet.

cc @sorantis would be great to get your input on this.

[sorantis]

@simitt I'm currently working on the overall structure for all integrations, and looks like APM could fit the proposed structure. I'll share the doc with you and we can chat about details next week.

[simitt]

@sorantis do you have an update on this? APM is working on integrating with Elastic Agent and Fleet in 7.14.

[sorantis]

@simitt This document outlines the user journeys related to choosing ILM policies for integrations. cc @mostlyjason

Specifically to package-spec, I don't think there's any defined section today for ILM policies. I've been checking the recent work for adding ML jobs to integrations . I wonder if we can follow a similar pattern here and add an ILM asset to package spec, as we have done for ML job. @ruflin @mtojek thoughts?

[ruflin]

We already support ILM policies as part of the package-spec today AFAIK. We have that in the endpoint package but there it is a fully "managed" ILM policy meaning we always overwrite it. APM is different here. It is not necessarily about shipping and ILM policy but how to make sure the ILM policy setting is not overwritten on each package upgrade.

[simitt]

APM is different here. It is not necessarily about shipping and ILM policy but how to make sure the ILM policy setting is not overwritten on each package upgrade.

Allowing users to customize their ILM policies will probably be interesting for other integrations too. APM shouldn't be different here, it just already supported this feature. The default ILM policies should be overwritten if they change on an upgrade, but if users provide custom policies then they should not be overwritten.

[ruflin]

Sorry, what I meant is that APM is different from the ILM part we do for endpoint where it is an "internal" ILM policy which is not expected to be modified by any user and always overwritten.

Searching for this issue I stumbled over elastic/kibana#72986 which is also more similar to the package spec change. Instead what I think we need for APM is likely to be purely in Fleet. I'll follow up shorty with a potential proposal.

[sorantis]

My two cents on custom ILM policies (and perhaps also dashboards). I can see two distinct cases that I think should be clarified.

1. The user can customize/edit and existing default ILM policy. - Do we want this? Allowing users to change prebuilt, Elastic owned ILM policies can make upgrade scenarios complicated. Perhaps making such assets immutable and allowing users to clone them for further customizations is a better option.
2. The user can assign a different/custom ILM policy to an integration. - This case is more about letting the user assign a different ILM policy when configuring an integration, which is already accounted for in the Data Management section of the Integrations Granularity doc. Should the user choose a different ILM policy, then all future updates of the Elastic owned policies will have no impact on the existing configurations. The (unlikely) downside is that the user won't get updated ILM policies with future updates. This can be highlighted to the user during the integration upgrade workflow. It will be up to the user to take an action or not.

[mukeshelastic]

Following up on Dimitri's comment above:

I am assuming the specific use case here is to allow different solution users have a control on the data retention. For example, for an APM solution in a single elastic deployment, thee are N datastreams that get created and users want a retention policy other than what apm-server integration ships by default. In current UX, they create a new ILM policy and assign it to the datastreams, somewhere in stack management UI. In future, we will simplify this experience by integrating assigning ILM policies within integrations UI.
Once the user assigns a different ILM policy to the apm datastreams, the apm-server package upgrade should not overwrite them with the default ILM policies and this needs to be supported in fleet. So we should create an issue for that.

The question of what should be the experience for editing the default assets - ILM policies, dashboards, index templates, ML jobs - that ship with a package is something that has been under discussion in the package lifecycle management google group.

So to summarize, fleet should support retaining modified assets ( we can restrict that to be ILM policies for now) during package upgrade workflow.

[simitt]

With the support of custom component templates elastic/kibana#90454, this feature is generally supported, therefore closing this issue.