Negative performance impact on wildcard field changes

[epixa]

As wildcard field changes were due to be released in ECS 1.8, we updated a ton of keyword fields to be wildcard instead from 7.11 onward. However, we've identified that there are significant performance implications that we need time to review - specifically we've seen storage increases of up to 33% and indexing throughout impacted by 25%. ECS is rolling back the wildcard field changes and will not be releasing them in 1.8, so we should effectively rollback the addition of wildcard fields in Beats modules for 7.11 as well.

We'll be moving forward with wildcard fields in the near future, it's just not clear to what extent.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[simitt]

@elastic/apm-server we probably also need to pull this into 7.11 once rolled back.

[ph]

@epixa @andrewstucki How are integrations impacted by these problems? I've created elastic/integrations#568 this on the integration side. I see a few integrations that use wildcards like the zeek integration.

[andrewstucki]

@ph commented on the integration side of things, but I'll cross-post here too just for the sake of tracking:

@ph -- these fields were changed to bring the packages to parity with the beats modules. We're using them "correctly" in the sense that these fields are all marked as wildcard fields in the experimental/1.8 ECS schema.

Personally I'd be fine keeping these as wildcard since the changes are slightly more constrained than beats, as @ruflin mentioned, but if we want to revert them in the same way the 7.11 beats modules are getting reverted, I'm fine with that too.

WRT zeek the reason the integration has so many fields that are wildcard is because the package itself is huge--both the filebeat module and the package contain ~30 data streams IIRC, so they have a lot of fields in general.