[Fleet] Support composable templates #90454
Assignees
Labels
Comments
simitt
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
16 tasks
|
Dropping some ideas on how this could be achieved: Today the index template itself contains as little information as possible and settings and mapping should happen through the component templates. Taking this model, we could automatically always add a third component called If we let the user add their own component and reference it, we have the problem that we need to keep track of it an update the array of components each team, user needs to put it in in the right order etc. With the above, we would have done all the work for them and the user has a single existing place to update things. |
|
Reserving a fixed name does not give the users the flexibility to create multiple smaller templates and then mix and match by referencing them per data stream. I'd say the suggestion sounds reasonable though to start with, in case the implementation gets too complicated otherwise. |
|
What is the use case that the same component templates are used across multiple data streams? Settings? It might be even beneficial to not allow to much flexibility as otherwise it becomes too easy to break things accidentally. 1 component template is change and affects 20 data streams and one of them breaks. Hard to find the root cause later on. |
|
Yes I was thinking of deployment specific settings that might be the same for all data streams. |
20 tasks
|
@jen-huang @jfsiii I see this is in progress on the 7.14 board. I assume we'll need to collaborate on definition? |
2 tasks
jfsiii
pushed a commit
that referenced
this issue
Jun 23, 2021
## Summary Closes #90454 Closes #72959 * Rename the component templates which are [installed for some packages](https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L197-L213) from `${templateName}-mappings` and `${templateName}-settings` to `${templateName}@mappings` and `${templateName}@settings` * When any package is installed, add a component template named `${templateName}@custom` * Any of above templates also include a `_meta` property with `{ package: { name: packageName } }` * On package installation, add any installed component templates to the `installed_es` property of the `epm-packages` saved object * On package removal, remove any installed component templates from the `installed_es` property of the `epm-packages` saved object <details><summary>Kibana logs showing component templates added for package</summary> ``` │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@custom] ``` </details> <details><summary>screenshot - component templates are editable in the Stack Management UI</summary> <img width="1342" alt="Screen Shot 2021-06-17 at 4 06 24 PM" src="https://user-images.githubusercontent.com/57655/122465421-1502bb80-cf86-11eb-94f4-9880cb3ea844.png"> </details> ### Checklist Delete any items that are not applicable to this PR. - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jun 23, 2021
## Summary Closes elastic#90454 Closes elastic#72959 * Rename the component templates which are [installed for some packages](https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L197-L213) from `${templateName}-mappings` and `${templateName}-settings` to `${templateName}@mappings` and `${templateName}@settings` * When any package is installed, add a component template named `${templateName}@custom` * Any of above templates also include a `_meta` property with `{ package: { name: packageName } }` * On package installation, add any installed component templates to the `installed_es` property of the `epm-packages` saved object * On package removal, remove any installed component templates from the `installed_es` property of the `epm-packages` saved object <details><summary>Kibana logs showing component templates added for package</summary> ``` │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@custom] ``` </details> <details><summary>screenshot - component templates are editable in the Stack Management UI</summary> <img width="1342" alt="Screen Shot 2021-06-17 at 4 06 24 PM" src="https://user-images.githubusercontent.com/57655/122465421-1502bb80-cf86-11eb-94f4-9880cb3ea844.png"> </details> ### Checklist Delete any items that are not applicable to this PR. - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
added a commit
that referenced
this issue
Jun 23, 2021
…3126) ## Summary Closes #90454 Closes #72959 * Rename the component templates which are [installed for some packages](https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L197-L213) from `${templateName}-mappings` and `${templateName}-settings` to `${templateName}@mappings` and `${templateName}@settings` * When any package is installed, add a component template named `${templateName}@custom` * Any of above templates also include a `_meta` property with `{ package: { name: packageName } }` * On package installation, add any installed component templates to the `installed_es` property of the `epm-packages` saved object * On package removal, remove any installed component templates from the `installed_es` property of the `epm-packages` saved object <details><summary>Kibana logs showing component templates added for package</summary> ``` │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@mappings] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@custom] │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@custom] ``` </details> <details><summary>screenshot - component templates are editable in the Stack Management UI</summary> <img width="1342" alt="Screen Shot 2021-06-17 at 4 06 24 PM" src="https://user-images.githubusercontent.com/57655/122465421-1502bb80-cf86-11eb-94f4-9880cb3ea844.png"> </details> ### Checklist Delete any items that are not applicable to this PR. - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: John Schulz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature:
Support configuring composable templates from within Fleet. We should allow users to configure component templates per data stream. The component templates should have a higher priority than the index template that is created when a package is installed. I believe this would be a sufficient solution in combination with component template docs.
Users could probably break some functionality by defining incompatible mappings - but I'd say that's acceptable if they add custom templates.
Describe a specific use case for the feature:
Users are used to configuring template settings and/or additional fields from within the APM and beats yml files. When switching to Agent and Fleet they should still have the option to customize some parts of the template.
Relevant settings are e.g.
template.settings.index.number_of_replicasortemplate.append_fields(we see that coming up fairly often for APM).Some of the config settings are also currently supported for APM on Cloud.
In case this issue better fits the
package-specrepo I can transfer.cc @ruflin
The text was updated successfully, but these errors were encountered: