[Change Proposal] Add support for explicitly defining dynamic_templates

[axw]

In elastic/beats#25422 we are extending fields.yml in libbeat to support explicitly defining dynamic_templates, for use with the dynamic_templates bulk request parameter introduced in Elasticsearch 7.13.0: elastic/elasticsearch#69948. The purpose of this is to support dynamically mapping complex metrics, such as histograms and summaries. Naturally we would like to support this in integration packages too.

The above mentioned Beats PR introduces a new boolean "dynamic_template" property to fields in fields.yml. If this property is true, then the field represents a dynamic template rather than a concrete field. For example:

- name: histogram_template
  type: histogram
  dynamic_template: true

This will result in an entry in the dynamic_templates section of the index template like this:

"dynamic_templates": {
  "histogram_template": {
    "mapping": {
      "type": "histogram"
    }
  }
}

Tasks:

• Update the Kibana Fleet plugin's template creation logic to understand the dynamic_template field property, creating dynamic templates ([Fleet] add support for the dynamic_template field property kibana#102289)
• Add dynamic_template to the field schema in package-spec (Add dynamic_template to field properties #187)

[ruflin]

++ on adding support for this. @exekias Do we already have packages where this could become useful?

[exekias]

Yes! Thanks for opening @axw, the obvious one will be Prometheus, where we can use this to get rid of the suffixes to match types. @ChrsMark FYI.

I'm +1 to getting this as explained, as it matches what we have in Beats after elastic/beats#25422

[axw]

Seems we have consensus so I've removed the discuss label, created implementation issues, and added checkboxes.

[simitt]

cc @jen-huang for visibility.

[axw]

I'm going to close this and revert the change to package-spec. It occurred to me that we an define the dynamic templates directly in the data stream manifest. See elastic/kibana#102289 (comment)