[Fleet] Add default variables to all integration data streams

[kpollich]

Ref elastic/elastic-package#949

There are several common variables that are defined across many integrations that have broad use cases and generic appeal to users, e.g.

• tags - string[]
• processors - string (yml box)
• ignore_older - string
• preserve_original_event - boolean

These variables are almost always copy/pasted between various data streams that want to support them. This is a manual process, and incurs a maintenance burden for integrations that want to adopt these universally applicable variables. If we wanted to adopt these across all integrations today, we'd also need to publish a new version of every single integrations or accept a slow pace of adoption - neither of which is particularly appealing.

Instead, we should add logic to Fleet to append these fields to all data streams across all integrations.

When a user opens policy editor to create/edit a package policy, Fleet will check a hardcoded list of global variable definitions and append those variables to the policy editor section for each data stream.

The nginx integration's access dataset is a good example of what the "end state" would look like for all data streams:

The highlighted section of "advanced fields" are defined by the nginx integration, but with this change in Fleet they'd appear in the same spot for all data streams across all integrations.

There's a few open questions and caveats to consider, e.g:

• What should we do when an integration already defines one of these fields?
  • Defer to the integration manifest
• What does an upgrade look like between package versions where these fields have moved from the integration source to Fleet?
• What is the long term migration path? Should this logic always live in Fleet, or should this eventually be handled by "integration templates" or a similar concept?

To start, let's limit the approach here to only the processors variable. This will let us validate the approach and iron out any of the edge cases around upgrades, deferring to integration manifests, etc.

Proposed configuration schema

Each variable definition should conform to an object that extends the package spec's variable definition. However, we should enforce a few caveats for the v1 of this implementation in the interest of avoiding complexity:

• Default variables are always placed in the Show advanced collapsible region
• Default variables are never marked as required

interface DefaultIntegrationVariable {
  name: string;
  type: string;
  title: string;
  description: string;
  multi: boolean;
  default: any; // can probably do some TypeScript wizardry to make this typesafe
  data_stream_types: Array<"logs" | "metrics">
}

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ishleenk17]

The highlighted section of "advanced fields" are defined by the nginx integration, but with this change in Fleet they'd appear in the same spot for all data streams across all integrations.

@kpollich: Not all of these fields would be valid for all datastreams. For Eg: ignore_older,tags is true only for logs datastream.

[kpollich]

Thanks, @ishleenk17

Not all of these fields would be valid for all datastreams. For Eg: ignore_older,tags is true only for logs datastream.

We could define a baseline compatibility condition in Fleet by adding a condition to each variable definition like data_stream_types: [logs, metrics].

[ishleenk17]

@kpollich : On the same lines as ignore_older, there is another variable "Preserve Original Events" which is meant to be present for all the logs datastream by default.

cc: @lalit-satapathy

[kpollich]

Thanks @ishleenk17, I've updated the description to capture that variable.

[kpollich]

@nimarezainia - Can you confirm that we're okay to start here with just the processors variable to validate the implementation? I think we had talked about reducing our scope here to just processors for now but would like to confirm. Thanks.

[nimarezainia]

@kpollich yes we should go with processors first. add_fields or add_tags can be used to achieve the tagging requirement if needed.

[nimarezainia]

@kpollich could we revisit this? Does it need to be in sprint 41? ranking is 20 and also I wonder if we really want to do this now given OTel. I also haven't seen any customer impacted as yet. Not discounting the need for the topic.