[Discuss] Support stored scripts in Fleet packages

[ajosh0504]

package-spec currently only supports inline scripts. Larger, more sophisticated scripts tend to be used as stored scripts instead of inline. Stored scripts are also known to reduce compilation time. This issue is to discuss if loading stored scripts should be supported in package-spec.

cc: @ruflin

[ruflin]

Is it correct to assume that a single stored script always belongs to a single ingest pipeline? Today we version ingest pipelines so I would assume we would also version the stored scripts attached to it. As the stored script is a reference in the ingest pipeline, it would mean on installation we have to adjust this reference. On upgrade, the stored script would have to be put in place first, the ingest pipeline as otherwise reference would not work yet.

[ajosh0504]

I'm not sure the assumption that a stored script belongs to a single ingest pipeline is right? I have worked on cases where the same script is shared but multiple ingest pipelines.

[peteharverson]

Note that two integrations PRs currently being worked on to package up ML supervised models and anomaly detection jobs for security use cases - Problem Child (elastic/integrations#2115) and DGA (elastic/integrations#2352), are having to include large inline scripts as part of inference and ingest pipeline definitions. As suggested by @ajosh0504 , it would be nice if packages supported stored scripts, rather than having to include them as part of the pipelines.

[ruflin]

@mtojek @jsoriano Would be great to get your input / thoughts on this topic.

[jsoriano]

+1, I think it would make sense to allow to include stored scripts on packages. It would also make sense to allow using them on different pipelines, while these pipelines belong to the same package. We don't support at the moment inter-package dependencies.

[SourinPaul]

Just giving it a friendly 👍 bump.

This came up during our conversations today re: adding transform to spec

As mentioned above, we need the ability to support stored scripts for most transform-based packages.

[szeitlin]

Do we think this is going to get on the roadmap in 2023-?