Dynamically generate template on startup

[ruflin]

So far the template was loaded from the beatname.template.json files. This now uses the fields.yml file to dynamically generate the template based on the ES version connected to. The name of the template is set dynamically based on template.name. By default it has the value {beatname}.

• Remove generated templates
• Package fields.yml
• Clean up config options as 2.x option is not needed anymore
• Update docs
• Update and test packaging

Part of #3654

[tsg]

I assume this approach (re-using the template.path setting) just for the prototype phase or the actual plan? I'd say we probably want to rethink the configuration options.

[ruflin]

Agree on rethinking the config options. I added it to #3654

[tsg]

My 2c on this.

Config option is called template.path which points to fields.yml. Is the name still ok?

I'd say it's better to use a different name, since this is quite different conceptually. Also, I'd suggest it is a good idea to let the user provide a fully custom template file (which they can generate/edit whatever name they like).

I wouldn't worry too much about backwards compatibility in this case, since I'd guess (and hope) most people didn't use a custom template path.

Should template config be moved to the top level (like dashboard)

Yeah, configuration is tricky here, perhaps we have a global fields section where the fields.yml is configured, and the rest of the things pull from that?

What is our recommendation on loading the template manually? Generate first with index_template binary and then load manually?

I agree there should be a way to write the template to a file. But I guess the primary way of adjusting things will be by editing the fields.yml file, which will be considered configuration?

Do we ship in index_template binary similar to import_dashboards? Can we integrate it into beats directly? -> setup

A binary has the disadvantage of adding to the package size. We could consider creating a beats-tools package that contains only these tools. This would be for things that you don't need for getting started, but are helpful in some situations.

[tsg]

Another, kind of random, thought: I'd like to make it fairly easy to change some index settings (that typically go in the template), like _source: on/off, number_of_shards, number_of_replicas, etc. We could put these in fields.yml or in the main Beat configuration file, but in both cases we should be careful about how much we want to duplicate the ES settings in the Beats config.

[ruflin]

@tsg Agreement on the points above. I kind of start to like the idea of a beats-tools package which is mainly for sysadmins etc for the setup phase but could also be used in the future to do some "checks".

I added all the points above to the meta issue to be solved #3654 I suggest moving forward with this PR as is and to further changes in follow up PR's.

[ruflin]

jenkins, retest it