[EPM] [Endpoint] Disable dynamic mapping on a template

[jonathan-buttner]

The endpoint team would like to disable dynamic mappings for all the endpoint indices. The easiest way to do this I think is to disable it at the template level that is generated by the ingest manager.

See this ticket for more detail: elastic/kibana#68380

[ruflin]

I'm wondering if we should introduce a mapping section or similar in the dataset manifest? I could forsee that there will be some requests in the future to have very specific template settings for a dataset.

[jonathan-buttner]

I'm wondering if we should introduce a mapping section or similar in the dataset manifest? I could forsee that there will be some requests in the future to have very specific template settings for a dataset.

What would happen if multiple dataset manifest files defined the same id field? Would the datasets get merged together? I wonder if the ingest manager would remove the previous template and only define a template for the last dataset or something.

If the hope is that datasets should not be merged, then having the mapping section per dataset manifest sounds good to me. I think you're right that requests might come in the future to have specific template settings for a dataset.

[ruflin]

TBH I'm not sure what would happen. But now that you mention it I think if someone tries to do it, the registry should complain and tell the user the package is not valid.

[ruflin]

To put out a specific proposal, I'm thinking of supporting the following in the dataset manifest:

template:
  mappings:
    dynamic: false
  settings:
    number_of_shards: 7
    index.lifecycle.name: foo

On the Kibana side, it would read the template settings in and either merge it with the existing template or probably even better, creates a third component with these in, so we leave it up to Elasticsearch to merge it correctly. @skh @neptunian Would that work? If yes, we need to find a name for this component.

@ycombinator I assume the above would also solve your ILM issue?

[ycombinator]

@ycombinator I assume the above would also solve your ILM issue?

Yes, this should do it for letting the Stack Monitoring datasets set a custom default ILM policy. Thanks!

[ruflin]

I filed #552 as a follow up, please have a look.

[ruflin]

Closing this as #552 was merged. @jonathan-buttner I assume you follow up on the Kibana side.

[neptunian]

@ruflin @jonathan-buttner Sorry I didn't get a chance to look at this closer sooner. Wondering if there was a particular reason (I saw there was some conversation here) we decided to build component templates based on a manifest file instead of having component template files that act as assets similar to how we handled the base package component templates? . We already have a function that checks for the existence of component templates and installs them. We just need to put the component templates in the dataset directory (and maybe a few other tweaks).

I think it's preferable to keep the assets in json files that we load through the asset paths we get back from the registry to keep it as consistent as possible with how we install other assets.

In the Kibana implementation, It seems that the component templates assets created are not being tracked as part of the installation of assets in the saved object, which means when the package is removed, all the component templates will not be removed. We didn't track initially because only the base package had these which were not removable. Will open an issue for this.

If the manifest file is the best way to go, I can remove the functionality that handles installing components in the component_templates directory that only the base package was utilizing since we no longer install the base package.

[jonathan-buttner]

@ruflin @jonathan-buttner Sorry I didn't get a chance to look at this closer sooner. Wondering if there was a particular reason (I saw there was some conversation here) we decided to build component templates based on a manifest file instead of having component template files that act as assets similar to how we handled the base package component templates? . We already have a function that checks for the existence of component templates and installs them. We just need to put the component templates in the dataset directory (and maybe a few other tweaks).

I think it's preferable to keep the assets in json files that we load through the asset paths we get back from the registry to keep it as consistent as possible with how we install other assets.

In the Kibana implementation, It seems that the component templates assets created are not being tracked as part of the installation of assets in the saved object, which means when the package is removed, all the component templates will not be removed. We didn't track initially because only the base package had these which were not removable. Will open an issue for this.

Oops yeah, I completely forgot about that.

If the manifest file is the best way to go, I can remove the functionality that handles installing components in the component_templates directory that only the base package was utilizing since we no longer install the base package.

hmm I keeping them as json files would work just as well. We'd have to make sure they were parsed before the _index_template is installed and grab the names to place it in the array here: https://github.com/elastic/kibana/blob/6cf796a4fb2a00165b8bf1a33047dd3688fc3e43/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts#L322

[neptunian]

@ruflin Same goes for the ILM policy. We have a function that installs ILM policies based on paths.

[ruflin]

@neptunian Good question. It all started with just configuring the ILM policy so I thought the manifest is an easy place to specify it. And for the mapping the initial problem was that fields.yml does not support the global setting. But you are correct, we could also use and load a json/yaml object instead. In any case, we need to track the assets we install.

One reason I keep coming back for the ILM policy to specify it in the manifest is that I think it is also someone could specify when building a package in Kibana in the future. Just writing it into a manifest sounds simpler than building a template for it, but both will work. And we don't support building packages in Kibana yet :-)

In summary, I think you are right that we could have solve the above with existing features :-(