[1.6] [Tooling] Add --exclude flag to Generator to support field removal testing (#1411)

[djptek]

Backports the following commits to 1.6:

• [Tooling] Add --exclude flag to Generator to support field removal testing ([Tooling] Add --exclude flag to Generator to support field removal testing #1411)

[djptek]

I'm backporting this to 1.6 since the goal is to use --exclude flag for testing with e.g. Beats and that's the current version

@ebeahan @kgeller make sense if I backport to the intermediate versions too?

[kgeller]

@djptek if we are going back to 1.6, then I think it makes sense to include in everything since

[djptek]

OK thx @kgeller 👍

[ebeahan]

Where in Beats is using 1.6.0? More recent releases of Beats are using a more recent version of ECS.

For example, Beats 7.13.0 uses ECS 1.9.0: https://github.com/elastic/beats/blob/v7.13.0/libbeat/_meta/fields.ecs.yml#L2

It's possible certain modules may still be using an older version of ECS.

Regardless, if you're trying to target an older version of ECS when running the generator script, you should use the --ref flag:

python scripts/generator.py --ref v1.6.0 --exclude ./custom/fields/exclude.yml

The --ref flag will generate artifacts for the targeted version.

[ebeahan]

@djptek I was a bit late with my thoughts before you merged 😅

I don't think this change is necessary, and I propose we revert. Let me know if I'm misunderstanding what you're trying to accomplish.

[djptek]

Hi @ebeahan - the goal of the --exclude flag was to generate e.g. fields.ecs.yml for testing with e.g. Beats with 1 or more fields removed. Running this within the context of the current ECS version creates a file that is in sync with the latest version of ECS which is several versions ahead of Beats (currently ECS 1.6), so there are far more differences than just the one removed field and it is no longer possible to assess the impact of building beats with the one field removed, because everything else has changed. By Backporting, we can checkout e.g. ECS 1.6, exclude the field(s) and use the generated files in the context of Beats.

If you're not happy with this approach let me know and I'll revert the backports on elastic main while leaving them on my fork so I can still do the impact evaluation from there

Sorry, hadn't seen the --ref flag. That make more sense, I'll revert these and go with that.