[HTTP_Endpoint] Integration maps incoming field as json.* no option to map under root

[toddferg]

Currently it is allowed to add a prefix (listed as optional) but some may want to index the field as root.
Specifically the http endpoint input in filebeat has: fields_under_root as an option to allow this behavior.

If prefix is left blank, it adds json to the subfield.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

I don't think it's possible to do this nicely without a breaking change. The options are to add a fields_under_root option to the config, but this is non-orthogonal to the prefix field, so what does a prefix mean when this is true? The nicest way to do this would be to default to rooted when the option is not present, but this would be a breaking change. There are other issues relating to handlebars behaviour, but they are maybe not insurmountable.

After investigation, the only way to do this in a way that is not a breaking change appears to be via an additional toggle.

[andrewkroh]

Perhaps we could make prefix: . signify the fields should be added to the root of the event. The only semi-related prior art I can think of was where aws-s3 accepts expand_event_list_from_field: .[] to mean expand the array at the root of the doc.

Other than that idea, I agree that an additional config option to "nullify" the prefix would be needed.

[efd6]

Yeah, I was also thinking of either . or $ given their use in other context. Dot is nicer, though neither are good; but are better than a another knob. I'll make that change and include it in the GA change for discussion noting possible reversion.