[Discussion] Input packages as "templates"

[kpollich]

As an aside on elastic/elastic-package#949, @ruflin proposed the idea of using our new concept of input packages as building blocks or templates for other packages. @jsoriano also had some input so pinging him here 🙂

For example, many packages are - at their core - an extension on top of logfile input or an httpjson input. Could we crystalize this concept by allow packages to "extend" an input package as starting point? With functionality like this, we could support things like base fields, processor support, etc across all packages without needing to duplicate/copy-paste config.

One way this might work would be a field like template_package in a package's manifest, and the elastic-package CLI would be able to read this field to generate the final package archive containing the "extended" set of fields. Another option would be to bake knowledge of these "base packages" into Fleet, but I'm more in favor of keeping all the information a package needs self-contained in its own archive.

This could represent an interesting evolution in how we think about and build packages. Eager to discuss further.

[jsoriano]

Input packages are a good building block for this. They could contain all the specific settings related to an input, and be used in combination with other packages, even already released ones. Current design (#319) has this kind of uses in mind.
For example a user could replace the input of any package with a custom one, what would give them the option to configure any setting specific of this input. This is a feature available in Filebeat that would be nice to have in Fleet too. Making Fleet aware of this would allow it to install the additional mappings of the input package, if any.

One way this might work would be a field like template_package in a package's manifest, and the elastic-package CLI would be able to read this field to generate the final package archive containing the "extended" set of fields.

This creates a build time dependency between packages, and can create different results for example if two packages are built with different versions of a template package. Keeping this on sync may force package developers to release their packages when the underlying input has changed or includes new fields.

This build time dependency is also limiting, as it only allows to use one input on each data stream, and this decision is made by the package developer. This is probably ok for the defaults, but wouldn't cover slightly more advanced use cases.

If we allow linking data streams with inputs in configuration time, we can use whatever input is available at the moment of configuring the data stream, as happens now with Filebeat.

Another option would be to bake knowledge of these "base packages" into Fleet, but I'm more in favor of keeping all the information a package needs self-contained in its own archive.

I think that input packages already provide a solution to where the knowledge about an input should be. Fleet gains this knowledge when an input package is installed. I think that we should progress on the implementation of input packages and evolve the use cases that they can offer us.

[endorama]

This creates a build time dependency between packages, and can create different results for example if two packages are built with different versions of a template package. Keeping this on sync may force package developers to release their packages when the underlying input has changed or includes new fields.

If we decide to implement any dependency between input and integration packages I think we must ensure that developers can express version constraints, otherwise updating any input package (upon which an unknown number of dependencies would be built upon) will create nightmares for package maintainers.

I think that input packages already provide a solution to where the knowledge about an input should be. Fleet gains this knowledge when an input package is installed.

I agree here but I think there should be a way for integration packages to say "I'm depending on input package ABC version x.y.z". This would help the overall experience in multiple ways: developers could lock versions, maintainers could perform bulk actions, users could know about the dependencies from the integration package and Fleet could automate installation.

These consideration brought me to a question I don't know the answer for: does Fleet allow installing different versions of the same package?

[ruflin]

These consideration brought me to a question I don't know the answer for: does Fleet allow installing different versions of the same package?

It doesn't today and for integration packages, there would be a conflict for assets like ingest pipeline, index templates, dashboards. But as input packages do not have these assets, this could be possible.