Config template "processors" section shadows custom ECS processors

[chris-counteractive]

As noted by @SecBear, the default config file contains a processors section that gets merged into the o365beat.yml and shadows the custom ECS processors. To fix that, you have to manually remove the template section, or merge the two. From my reply to @SecBear's PR:

This is definitely true: the second processor section "shadows" the first, and has to be removed or merged before use. The problem is, when building the beat the build tools actually create o365beat.yml dynamically by merging _meta/beat.yml with a config file template within the libbeat framework (libbeat/_meta/config.yml.tmpl). It's that .tmpl file that has the extra processor section, and I haven't had time to figure out how to suppress the inclusion of that section. Instead I do what you did in the PR, which is combine them or remove the second processors section altogether.

Unfortunately, if I merge this, it'll be clobbered by the build for the next release and I'll have to re-insert it by hand, which may be the best bet until we sort out a durable fix. Any thoughts on how to suppress the inclusion of the processors section from the libbeat template would be much appreciated! Or, if there's a smarter way to think about the issue, I'm happy to hear that too.

Thanks again for the contribution, I really appreciate the engagement, I'm sure we can sort out a long-term fix for what is definitely a real issue and inconvenience.

Originally posted by @chris-counteractive in #7 (comment)

[chris-counteractive]

The concatenation of o365beat's _meta/beat.yml and the libbeat _meta/config.yml.tmpl happens in beats/dev-tools/mage/config.go. There doesn't appear to be any configuration that'll suppress the inclusion of that second processors section, as the merge is just a simple file concat and the .tmpl file conditionals don't have any option for leaving it out entirely.

We'll have to consider removing it later in the build, which will be tricky because we won't have any control of future changes to that file. Maybe there's a more robust way to pre-process the config file, merge duplicate sections, and re-write it. Research ongoing.

[chris-counteractive]

Opened a discussion at https://discuss.elastic.co/t/config-template-processors-section-shadows-custom-processors-in-custom-beats/215578, the first step to an issue or pull request to correct the underlying problem.

[chris-counteractive]

Fixed this by re-writing the concatenated config files in the build magefile (see 0074704). It's not the "right answer," but it works and shouldn't be too brittle so long as we keep an eye on any updates to libbeat (the upstream issue remains, so any custom beat that wants to add a processors section will have to do something similar - either ask their users to fix/merge the configs, or rewrite during the build process like we did here).