Improve support for transforms

[joshdover]

We merged basic transforms support in #307, but there are several outstanding and related problems that remain to be solved.

Short-term - hard blockers for next security packages

• Add transform to spec #307
• [Fleet] Implement new transform installation mechanism kibana#134321
  • This implements the package-spec definition above so transforms in packages can actually be installed. Until this is finished, packages can contain transforms, but they will not be installed.
• [Fleet] Support installing transforms without granting kibana_system index privileges kibana#137278
  • kibana_system is currently used to install all transforms. This is necessary in the Endpoint case where the package must be upgraded along with Kibana. CSP may need this too.
  • The problem with this is that it requires kibana_system to be granted read/write privileges to transform src/dest indices

Medium-term - likely not hard blockers for next security packages

• Better upgrade strategy for transforms
  • Rebuilding transform destination indices can be very expensive, but is necessary in some cases
  • Should avoid it in obvious upgrade scenarios (eg. a visualization was updated)
  • Updating the frequency or delay is supported by the transform API, Fleet could leverage this to avoid re-building transforms unnecessarily so.
  • Will definitely need to destroy and re-create a transform when the aggregation or fields have changed. This could be improved, but likely lower priority at this time.
• [Discuss] Support stored scripts in Fleet packages #202

Long-term

• Space-specific transforms #460
• Enforce the data stream naming convention for transform src and dest indices

[sophiec20]

Thanks for a good summary of improvements.

Re space-specific transforms .. Future changes to the transforms privileges model have a strong dependency on a better understanding of the plans for a unified platform security model.

Within the current limitations of the kibana and elasticsearch split privileges model - transforms supports Spaces as follows. The transform destination index is a normal elasticsearch index. When using Kibana Spaces, create a Kibana Data View on the destination index for the required Space. Spaces are supported. The long running transform persistent task is not space aware (in the same way that an elasticsearch index or an ILM policy is not space aware). This is by design.

Ideally this item should be de-scoped (or shown as long term) from Fleet integrations work as this has caused confusion in the past.

[szeitlin]

@sophiec20 Thanks for clarifying. I agree, I don't think space-specific transforms is a high priority for us, though installing package assets with the appropriate privileges I'm told has been/could be a blocker. @joshdover wrote some detailed suggestions re: API keys here, but I definitely don't know enough about the trade-offs re: auto-upgrade or how important that should be. #293 (comment)

[sophiec20]

@szeitlin Your comment mostly refers to Better privilege model for transforms (which should remain in this list).

[szeitlin]

@szeitlin Your comment mostly refers to Better privilege model for transforms (which should remain in this list).

@sophiec20 Correct, but it doesn't seem to be a ticket or linked to something that's obviously roadmapped to be working on in 8.4 or 8.5? Is there a plan for how to address that issue? It looked like it was still under discussion, and from what I've read so far, it seems potentially complex to decide how we want the privilege model to work, nevermind how to implement it.

[joshdover]

doesn't seem to be a ticket or linked to something that's obviously roadmapped to be working on in 8.4 or 8.5? Is there a plan for how to address that issue?

I've opened a new issue for this: elastic/kibana#137278

[joshdover]

As of 8.5, Fleet now supports installing transforms based on the new spec, thanks to @qn895's work in elastic/kibana#134321. Next steps as I understand it:

• (maybe) Revisit the upgrade behavior, we likely need to at least support the naive approach of deleting the destination index and rebuilding the transform. See discussion here: [Fleet] Implement new transform installation mechanism kibana#134321 (comment)
• [Fleet] Support installing transforms without granting kibana_system index privileges kibana#137278
  • First, align on technical definition, including how this impacts other asset types, like ILM. We need a common, scalable approach.
  • Implement

[qn895]

Another item for 8.6+ is to support installing transforms in order of dependency elastic/kibana#142891. Originally in the spec, we have it such that we can and should determine the order automatically. However, one thought is perhaps the author of the package should be able to specify the order of which transforms should be installed?